FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

[Bulat Ziganshin]

thanks, fixed

[jo.henke]

May I also suggest this patch:

--- src/main.cpp
+++ src/main.cpp
@@ -100,4 +100,5 @@
       r.qoutsize += size;
     }
+#ifndef FREEARC_NO_TIMING
     double after = GetSomeTime();
     //r.start_time -= after-before;   // Don't take into account I/O times
@@ -139,4 +140,5 @@
     }
     //r.start_time -= GetSomeTime()-after;   // Don't take into account I/O times
+#endif
     return size;
 
@@ -151,5 +153,7 @@
       double speed     = (r.mode==COMPRESS? insizeMB : outsizeMB) / mymax(time,0.001);
       fprintf (stderr, "\r%s%s %.3lf -> %.3lf mb (%.1lf%%)", r.method_name, r.mode==COMPRESS? "compressed":"Unpacked", insizeMB, outsizeMB, ratio);
+#ifndef FREEARC_NO_TIMING
       if (time>0.001)  fprintf (stderr, ", time %.3lf secs, speed %.3lf mb/sec", time, speed);
+#endif
       fprintf (stderr, "\n");
     }

It's because I found that gcc doesn't optimize these parts away automatically.

64-bit versions are also welcome

Do you think, you could rewrite the match finders to use (32 bit) indexes instead of whole pointers (especially for the calculations in next_pair)? I think, the step to a 64 bit compile would be much closer then.

[m^2]

Bulat, why did you limit hash row length to 256?
4096 was very good.

[Bulat Ziganshin]

with lzma or tornado?

[m^2]

Tornado.

[Bulat Ziganshin]

can you show those great results with l4096?

[m^2]

can you show those great results with l4096?

i.e. metacompressor file 2.
tor 0.4 -12 -s2: 273,378,727
tor 0.4 -12 -s2 -l4096: 272,802,449

[Bulat Ziganshin]

try -5 -s2 -l4096 -u1

[m^2]

try -5 -s2 -l4096 -u1

Tor 0.4 or 0.5?

[Bulat Ziganshin]

0.5

[Bulat Ziganshin]

updated http://www.haskell.org/bz/tornado.zip
added 64-bit compilers support and tor64.exe

[m^2]

updated http://www.haskell.org/bz/tornado.zip
added 64-bit compilers support and tor64.exe

Intel compiler? Cool.
If you have it, why did you compile the 32 bit version with GCC?
BTW, there are no chances to mate it with GHC to and make a 64bit FreeArc, right?

[Bulat Ziganshin]

atm, 32-bit icl compile is slower than gcc one. even 64-bit is compile is only at par. may be it's problem of enabling more optimization, though

64-bit windows ghc version doesn't exist still. i will look eventually at 64-bit linux, just now i have one more show-stopper - ppmd compressor

[m^2]

atm, 32-bit icl compile is slower than gcc one. even 64-bit is compile is only at par. may be it's problem of enabling more optimization, though

64-bit windows ghc version doesn't exist still. i will look eventually at 64-bit linux, just now i have one more show-stopper - ppmd compressor

Maybe look at 7zip's sources? Igor managed to make a 64 bit ppmd.

[jo.henke]

added 64-bit compilers support and tor64.exe

Thanks a lot for that!

Unfortunately there are some issues with Linux and GCC 4.3: In compile64, change "-march=i486 -mtune=pentiumpro" to something like "-mtune=generic", but then:

In file included from Tornado.cpp:7,
                 from main.cpp:4:
DataTables.cpp: In function ?bool check_for_data_table(int, int&, int&, byte*, byte*, byte*&, byte*, uint64&, byte* (&)[64][64])?:
DataTables.cpp:50: error: cast from ?byte*? to ?int? loses precision
main.cpp: In function ?int main(int, char**)?:
main.cpp:273: warning: format ?%d? expects type ?int?, but argument 2 has type ?long unsigned int?

[Bulat Ziganshin]

>Maybe look at 7zip's sources? Igor managed to make a 64 bit ppmd.

yes, it's one of variants. actyually, i recalled that my dict and rep are probably 64-bit incompatible too and need the same conversion

jo.henke, thank you. i will look at all stuff you mentioned on this page

[Bulat Ziganshin]

updated http://www.haskell.org/bz/tornado.zip : just patches suggested by jo.henke. hope that now 64-bit & no-timer compiles will be ok

[jo.henke]

just patches suggested by jo.henke. hope that now 64-bit & no-timer compiles will be ok

Thanks! Just one small thing in compile-be: Remove "-march ...".

[Bulat Ziganshin]

ok

[Bulat Ziganshin]

updated http://www.haskell.org/bz/tornado.zip
-3 mode compression improved by another 0.5%

funny: Vista has the service that prereads frequently used files. on my box this means that it starts to read into memory my beloved testfiles immediately after every reboot

[m^2]

Thanks.

TCUP:
tor 0.5a1 3     133593478 ~9.5
tor 0.5a1 4     120445642 ~20
tor 0.5a(new) 3 129298794 ~12.5

XSOS (VM image, 442515968 B)
tor 0.5a1 3     74698766   6.656
tor 0.5a1 4     68172202  22.157
tor 0.5a(new) 3 71867214 ~21.5

I think that you start to use data tables way too early. I've seen many cases like XSOS when speed drops greatly because of it with the previous versions. IMO it should start from 6.

[Bulat Ziganshin]

are you sure that datatables are problem? can you test with -t0 (using tor-full)?

what's your cpu (especially cache sizes)?

[m^2]

are you sure that datatables are problem? can you test with -t0 (using tor-full)?

what's your cpu (especially cache sizes)?

Before the speed was dropping with -4. Now -3.
It's the only reason I see, other things change slightly between the modes.
You say that it shouldn't be this way?

Pentium D 805 2.66Ghz, FSB 533 mhz, 16+12 KB L1 (data+instr.), 1 MB L2.

[Bulat Ziganshin]

can you test it with -t0?

[m^2]

Strange thing. Huge variance. Results:
11.766
20.469
18.312
20.437
10.703
20.344
11.172

Practically only FireFox running during the test.
CPU time: 6.2-6.8s.

[Bulat Ziganshin]

so, case closed?

if cpu times is more stable, you may compare them

[m^2]

so, case closed?

if cpu times is more stable, you may compare them

Which end users care about CPU time? And global times are usually very stable too, on a clean machine I often get 3 results in a row <0.05% from each other.

1. Possibly you have some serious issue, probably IO related. I'll recheck on a clean machine though.

As we're at the topic, I have to report that I've seen such variation on a clean machine only once. When testing FreeArc on BMP, IIRC only higher LZMA modes. They usually run 120s or 130s (don't remember) and sometimes ~90s.

2. -t1 CPU usage: 21s.

[jo.henke]

Just a small hint for people, who want to build a small and fast tor binary with minimal library dependence in Linux (nice for embedded usage):

• start from the command line in the file "compile"
• add "-DFREEARC_NO_TIMING -fno-threadsafe-statics"
• remove "-lrt"
• replace "g++" by "gcc" and "-funroll-loops" by "-fprofile-generate"
• compile
• run the resulting binary with all the parameters you typical use (compression and decompression)
• compile again with "-fprofile-use" instead of "-fprofile-generate"

In my case the final binary is 116352 bytes large and depends only on libc:

$ ldd tor
	linux-gate.so.1 =>  (0xf7f78000)
	libc.so.6 => /lib/libc.so.6 (0xf7e07000)
	/lib/ld-linux.so.2 (0xf7f79000)

Who would expect it to be a C++ program?

[Bulat Ziganshin]

i don't changed i/o in tornado for a long time. overall, i don't think that unstable execution times is due to smth in tornado rather than your system

2. we need to compare -3 -t0 and -3 -t1 times to make any conclusions

jo.henke, why not make new compile script based on your suggestions? you can compress and decompress tor executable itself in -1..-11 modes (with reduced memory, of course) and make it easy for anyone to replace ./tor with any larger file they have

[m^2]

i don't changed i/o in tornado for a long time. overall, i don't think that unstable execution times is due to smth in tornado rather than your system

This is very possible, but still I'd like to check it. Sorry, can't tell you when.

2. we need to compare -3 -t0 and -3 -t1 times to make any conclusions

I meant "-3 -t1", I know there's no point comparing totally different modes.
3 t0: 6.2-6.8s.
3 t1: ~21s.