NanoZip huge efficiency issue

[m^2]

Today, while browsing the net, I new I was gonna have a lot of spare CPU power for some time and decided to use it to do some testing. I didn't want to post the results, there was quite a lot of things running, also some accessing hard drive, so results are not very accurate, but give some insight.

Test set: Chromium sources, almost 1.4GB and over 110000 files, almost entirely text.
.tgz I got from google -
437 MB.
The first test, quick.exe -1.
603.6 MB, 302.x seconds (I didn't write the number down, it's from my memory).
Then FreeArc -m1.
432.9 MB, 363.453 s. Too fast, something's wrong with Quick. Cache? Tested Quick again,
603.6 MB, 228.547 s. this time.
Another test: FreeArc -m2.
331.9 MB / 628.156 s.
Not bad.

Now, FA was going to get beating...The King of efficiency in my another test preformed on application sources, NanoZip -cf.
483.2 MB in 2000.882 s.
Stats given by NZ:
Compression: 3m 2.23s
IO-in: 26m 30.73s
IO-out 18.75s

As you see, IO-in is very wrong...

I decided to test it again, I didn't believe that other things I was doing had significant impact on this time, but who knows...Let's do it again. I didn't want to test in exactly the same mode, so I used the second fastest, -cF.
359.4 MB in 3418.375s
Compression: 9m 30.02s
IO-in: 44m 47.16s
IO-out 12.31s

Bulat, do you like it?

BTW, Igor's Timer's Process Time showed in the lat test was 122.906 s., ~3% of total time. Totally irrelevant.

[pat357]

483.2 MB in 2000.882 s.
Compression: 3m 2.23s
IO-in: 26m 30.73s
As you see, IO-in is very wrong...

Could you try : NZ a -cF -m100m ... ?
Didn't you notice disk trashing ??
Could it be that your system is swapping out data to the disk all the time to free some memory... ?

359.4 MB in 3418.375s
Compression: 9m 30.02s
IO-in: 44m 47.16s
IO-out 12.31s
BTW, Igor's Timer's Process Time showed in the lat test was 122.906 s., ~3% of total time. Totally irrelevant.

Not sure...123 sec for ?1500MB, that's about 12Mb/s. It's still slow, but could be about right. I get about 20-25Mb/s in -cF mode for txt. (C2Q E6600 2.4Ghz)

[m^2]

Could you try : NZ a -cF -m100m ... ?
Didn't you notice disk trashing ??
Could it be that your system is swapping out data to the disk all the time to free some memory... ?

NZ checks for amount of free memory and allocates 64 MBs less.
My memory usage during the first test actually decreased, so I don't think that it's the case, I'll check though. I've seen that it was going pretty fast until it reached .svn-base files that take about half of total size. Then it was just terrible. I have no idea, why...

Not sure...123 sec for ?1500MB, that's about 12Mb/s. It's still slow, but could be about right. I get about 20-25Mb/s in -cF mode. (C2Q E6600 2.4Ghz)

I have Pentium D@2.66. I would expect more than a half, but it's not impossible. BTW we can see that it can't use 4 cores, your CPU would score much better.

[pat357]

going pretty fast until it reached .svn-base files that take about half of total size.

You could try to compress this file in a separate archive and see whether the problem is related to this specific data or not.

I have Pentium D@2.66. I would expect more than a half, but it's not impossible. BTW we can see that it can't use 4 cores, your CPU would score much better.

NZ with 4 cores might be for future versions I guess it is possible to implement BWT for 4 cores...(Winrk might be even doing this already, not sure though..).
The speed of NZ -cF obviously depends of the kind of data : for large txt-files, I get up to 45 Mb/s, while for PDF something like < 15Mb/s.

[m^2]

You could try to compress this file in a separate archive and see whether the problem is related to this specific data or not.

NZ with 4 cores might be for future versions I guess it is possible to implement BWT for 4 cores...(Winrk might be even doing this already, not sure though..).
The speed of NZ -cF obviously depends of the kind of data : for large txt-files, I get up to 45 Mb/s, while for PDF something like < 15Mb/s.

100 MB didn't help at all.
*.svn* doesn't work, I'll have to copy them to another place...tomorrow.

[m^2]

OK, I tried it with svn data only. 720MB in over 72000 files.
Better result (of 2):

1252.797 s.

Performance per file slightly better, performance per byte lower.

When I switched sorting off, the job was finished in 505.907 s. (svn files only)
Though it's a big improvement, the result is still underwhelming.

[nimdamsk]

You can try to defragment source disk. It doesn't explain why other archivers don't blame this issue though...

[m^2]

You can try to defragment source disk. It doesn't explain why other archivers don't blame this issue though...

Fragmentation is not an issue.

[pat357]

OK, I tried it with svn data only. 720MB in over 72000 files.
Better result (of 2):
1252.797 s.
Performance per file slightly better, performance per byte lower.
When I switched sorting off, the job was finished in 505.907 s. (svn files only)
Though it's a big improvement, the result is still underwhelming.

Very strange.... do the other NZ compression modes(cd, cD,co..) also have this "issue" ?
Could you also try with the -mn option ?

[m^2]

Got something.
It has over 126000 handles opened and with double digit commit charge~ 500 MB private bits.
Additionally, my system cache hits 600 MB.
Peak memory usage - 1.5 GB, usually ~1300 MB.
I tried to spy it with Process Monitor and each CreateFile is paired with CloseFile.