Results 1 to 10 of 10

Thread: NanoZip huge efficiency issue

  1. #1
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts

    NanoZip huge efficiency issue

    Today, while browsing the net, I new I was gonna have a lot of spare CPU power for some time and decided to use it to do some testing. I didn't want to post the results, there was quite a lot of things running, also some accessing hard drive, so results are not very accurate, but give some insight.

    Test set: Chromium sources, almost 1.4GB and over 110000 files, almost entirely text.
    .tgz I got from google -
    437 MB.
    The first test, quick.exe -1.
    603.6 MB, 302.x seconds (I didn't write the number down, it's from my memory).
    Then FreeArc -m1.
    432.9 MB, 363.453 s. Too fast, something's wrong with Quick. Cache? Tested Quick again,
    603.6 MB, 228.547 s. this time.
    Another test: FreeArc -m2.
    331.9 MB / 628.156 s.
    Not bad.

    Now, FA was going to get beating...The King of efficiency in my another test preformed on application sources, NanoZip -cf.
    483.2 MB in 2000.882 s.
    Stats given by NZ:
    Compression: 3m 2.23s
    IO-in: 26m 30.73s
    IO-out 18.75s

    As you see, IO-in is very wrong...

    I decided to test it again, I didn't believe that other things I was doing had significant impact on this time, but who knows...Let's do it again. I didn't want to test in exactly the same mode, so I used the second fastest, -cF.
    359.4 MB in 3418.375s
    Compression: 9m 30.02s
    IO-in: 44m 47.16s
    IO-out 12.31s

    Bulat, do you like it?

    BTW, Igor's Timer's Process Time showed in the lat test was 122.906 s., ~3% of total time. Totally irrelevant.

  2. #2
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by m^2 View Post
    483.2 MB in 2000.882 s.
    Compression: 3m 2.23s
    IO-in: 26m 30.73s
    As you see, IO-in is very wrong...
    Could you try : NZ a -cF -m100m ... ?
    Didn't you notice disk trashing ??
    Could it be that your system is swapping out data to the disk all the time to free some memory... ?

    359.4 MB in 3418.375s
    Compression: 9m 30.02s
    IO-in: 44m 47.16s
    IO-out 12.31s
    BTW, Igor's Timer's Process Time showed in the lat test was 122.906 s., ~3% of total time. Totally irrelevant.
    Not sure...123 sec for ?1500MB, that's about 12Mb/s. It's still slow, but could be about right. I get about 20-25Mb/s in -cF mode for txt. (C2Q E6600 2.4Ghz)
    Last edited by pat357; 10th September 2008 at 01:00.

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by pat357 View Post
    Could you try : NZ a -cF -m100m ... ?
    Didn't you notice disk trashing ??
    Could it be that your system is swapping out data to the disk all the time to free some memory... ?
    NZ checks for amount of free memory and allocates 64 MBs less.
    My memory usage during the first test actually decreased, so I don't think that it's the case, I'll check though. I've seen that it was going pretty fast until it reached .svn-base files that take about half of total size. Then it was just terrible. I have no idea, why...


    Quote Originally Posted by pat357 View Post
    Not sure...123 sec for ?1500MB, that's about 12Mb/s. It's still slow, but could be about right. I get about 20-25Mb/s in -cF mode. (C2Q E6600 2.4Ghz)
    I have Pentium D@2.66. I would expect more than a half, but it's not impossible. BTW we can see that it can't use 4 cores, your CPU would score much better.

  4. #4
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by m^2 View Post
    going pretty fast until it reached .svn-base files that take about half of total size.
    You could try to compress this file in a separate archive and see whether the problem is related to this specific data or not.
    Quote Originally Posted by m^2 View Post
    I have Pentium D@2.66. I would expect more than a half, but it's not impossible. BTW we can see that it can't use 4 cores, your CPU would score much better.
    NZ with 4 cores might be for future versions I guess it is possible to implement BWT for 4 cores...(Winrk might be even doing this already, not sure though..).
    The speed of NZ -cF obviously depends of the kind of data : for large txt-files, I get up to 45 Mb/s, while for PDF something like < 15Mb/s.

  5. #5
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by pat357 View Post
    You could try to compress this file in a separate archive and see whether the problem is related to this specific data or not.

    NZ with 4 cores might be for future versions I guess it is possible to implement BWT for 4 cores...(Winrk might be even doing this already, not sure though..).
    The speed of NZ -cF obviously depends of the kind of data : for large txt-files, I get up to 45 Mb/s, while for PDF something like < 15Mb/s.
    100 MB didn't help at all.
    *.svn* doesn't work, I'll have to copy them to another place...tomorrow.
    Last edited by m^2; 10th September 2008 at 02:02.

  6. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    OK, I tried it with svn data only. 720MB in over 72000 files.
    Better result (of 2):

    1252.797 s.

    Performance per file slightly better, performance per byte lower.

    When I switched sorting off, the job was finished in 505.907 s. (svn files only)
    Though it's a big improvement, the result is still underwhelming.

  7. #7
    Member
    Join Date
    Jan 2007
    Location
    Moscow
    Posts
    239
    Thanks
    0
    Thanked 3 Times in 1 Post
    You can try to defragment source disk. It doesn't explain why other archivers don't blame this issue though...

  8. #8
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by nimdamsk View Post
    You can try to defragment source disk. It doesn't explain why other archivers don't blame this issue though...
    Fragmentation is not an issue.

  9. #9
    Member
    Join Date
    May 2008
    Location
    Antwerp , country:Belgium , W.Europe
    Posts
    487
    Thanks
    1
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by m^2 View Post
    OK, I tried it with svn data only. 720MB in over 72000 files.
    Better result (of 2):
    1252.797 s.
    Performance per file slightly better, performance per byte lower.
    When I switched sorting off, the job was finished in 505.907 s. (svn files only)
    Though it's a big improvement, the result is still underwhelming.
    Very strange.... do the other NZ compression modes(cd, cD,co..) also have this "issue" ?
    Could you also try with the -mn option ?

  10. #10
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Got something.
    It has over 126000 handles opened and with double digit commit charge~ 500 MB private bits.
    Additionally, my system cache hits 600 MB.
    Peak memory usage - 1.5 GB, usually ~1300 MB.
    I tried to spy it with Process Monitor and each CreateFile is paired with CloseFile.

Similar Threads

  1. NanoZip - a new archiver, using bwt, lz, cm, etc...
    By Sami in forum Data Compression
    Replies: 280
    Last Post: 29th November 2015, 10:46
  2. HFCB: Huge Files Compression Benchmark
    By Bulat Ziganshin in forum Data Compression
    Replies: 129
    Last Post: 6th January 2015, 14:31
  3. Idea for raising compression efficiency on disk images
    By Mexxi in forum Data Compression
    Replies: 10
    Last Post: 18th February 2010, 04:56
  4. Nanozip decompression data troubles
    By SvenBent in forum Data Compression
    Replies: 11
    Last Post: 12th January 2009, 22:25
  5. enwik9 benchmark nanozip, bliz, m99, dark
    By Sami in forum Data Compression
    Replies: 6
    Last Post: 31st July 2008, 20:24

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •