Page 2 of 19 FirstFirst 123412 ... LastLast
Results 31 to 60 of 555

Thread: FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)

  1. #31
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    thanks, fixed

  2. #32
    Member jo.henke's Avatar
    Join Date
    Dec 2008
    Location
    Europe
    Posts
    56
    Thanks
    0
    Thanked 4 Times in 1 Post
    May I also suggest this patch:
    Code:
    --- src/main.cpp
    +++ src/main.cpp
    @@ -100,4 +100,5 @@
           r.qoutsize += size;
         }
    +#ifndef FREEARC_NO_TIMING
         double after = GetSomeTime();
         //r.start_time -= after-before;   // Don't take into account I/O times
    @@ -139,4 +140,5 @@
         }
         //r.start_time -= GetSomeTime()-after;   // Don't take into account I/O times
    +#endif
         return size;
     
    @@ -151,5 +153,7 @@
           double speed     = (r.mode==COMPRESS? insizeMB : outsizeMB) / mymax(time,0.001);
           fprintf (stderr, "\r%s%s %.3lf -> %.3lf mb (%.1lf%%)", r.method_name, r.mode==COMPRESS? "compressed":"Unpacked", insizeMB, outsizeMB, ratio);
    +#ifndef FREEARC_NO_TIMING
           if (time>0.001)  fprintf (stderr, ", time %.3lf secs, speed %.3lf mb/sec", time, speed);
    +#endif
           fprintf (stderr, "\n");
         }
    It's because I found that gcc doesn't optimize these parts away automatically.

    Quote Originally Posted by Bulat Ziganshin View Post
    64-bit versions are also welcome
    Do you think, you could rewrite the match finders to use (32 bit) indexes instead of whole pointers (especially for the calculations in next_pair)? I think, the step to a 64 bit compile would be much closer then.

  3. #33
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Bulat, why did you limit hash row length to 256?
    4096 was very good.

  4. #34
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    with lzma or tornado?

  5. #35
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Tornado.

  6. #36
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    can you show those great results with l4096?

  7. #37
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    can you show those great results with l4096?
    i.e. metacompressor file 2.
    tor 0.4 -12 -s2: 273,378,727
    tor 0.4 -12 -s2 -l4096: 272,802,449

  8. #38
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    try -5 -s2 -l4096 -u1

  9. #39
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    try -5 -s2 -l4096 -u1
    Tor 0.4 or 0.5?

  10. #40
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    0.5

  11. #41
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    updated http://www.haskell.org/bz/tornado.zip
    added 64-bit compilers support and tor64.exe

  12. #42
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    updated http://www.haskell.org/bz/tornado.zip
    added 64-bit compilers support and tor64.exe
    Intel compiler? Cool.
    If you have it, why did you compile the 32 bit version with GCC?
    BTW, there are no chances to mate it with GHC to and make a 64bit FreeArc, right?

  13. #43
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    atm, 32-bit icl compile is slower than gcc one. even 64-bit is compile is only at par. may be it's problem of enabling more optimization, though

    64-bit windows ghc version doesn't exist still. i will look eventually at 64-bit linux, just now i have one more show-stopper - ppmd compressor

  14. #44
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    atm, 32-bit icl compile is slower than gcc one. even 64-bit is compile is only at par. may be it's problem of enabling more optimization, though

    64-bit windows ghc version doesn't exist still. i will look eventually at 64-bit linux, just now i have one more show-stopper - ppmd compressor
    Maybe look at 7zip's sources? Igor managed to make a 64 bit ppmd.

  15. #45
    Member jo.henke's Avatar
    Join Date
    Dec 2008
    Location
    Europe
    Posts
    56
    Thanks
    0
    Thanked 4 Times in 1 Post
    Quote Originally Posted by Bulat Ziganshin View Post
    added 64-bit compilers support and tor64.exe
    Thanks a lot for that!

    Unfortunately there are some issues with Linux and GCC 4.3: In compile64, change "-march=i486 -mtune=pentiumpro" to something like "-mtune=generic", but then:
    Code:
    In file included from Tornado.cpp:7,
                     from main.cpp:4:
    DataTables.cpp: In function ?bool check_for_data_table(int, int&, int&, byte*, byte*, byte*&, byte*, uint64&, byte* (&)[64][64])?:
    DataTables.cpp:50: error: cast from ?byte*? to ?int? loses precision
    main.cpp: In function ?int main(int, char**)?:
    main.cpp:273: warning: format ?%d? expects type ?int?, but argument 2 has type ?long unsigned int?

  16. #46
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    >Maybe look at 7zip's sources? Igor managed to make a 64 bit ppmd.

    yes, it's one of variants. actyually, i recalled that my dict and rep are probably 64-bit incompatible too and need the same conversion

    jo.henke, thank you. i will look at all stuff you mentioned on this page

  17. #47
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    updated http://www.haskell.org/bz/tornado.zip : just patches suggested by jo.henke. hope that now 64-bit & no-timer compiles will be ok

  18. #48
    Member jo.henke's Avatar
    Join Date
    Dec 2008
    Location
    Europe
    Posts
    56
    Thanks
    0
    Thanked 4 Times in 1 Post
    Quote Originally Posted by Bulat Ziganshin View Post
    just patches suggested by jo.henke. hope that now 64-bit & no-timer compiles will be ok
    Thanks! Just one small thing in compile-be: Remove "-march ...".

  19. #49
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    ok

  20. #50
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    updated http://www.haskell.org/bz/tornado.zip
    -3 mode compression improved by another 0.5%

    funny: Vista has the service that prereads frequently used files. on my box this means that it starts to read into memory my beloved testfiles immediately after every reboot

  21. #51
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Thanks.
    Code:
    TCUP:
    tor 0.5a1 3     133593478 ~9.5
    tor 0.5a1 4     120445642 ~20
    tor 0.5a(new) 3 129298794 ~12.5
    
    XSOS (VM image, 442515968 B)
    tor 0.5a1 3     74698766   6.656
    tor 0.5a1 4     68172202  22.157
    tor 0.5a(new) 3 71867214 ~21.5
    I think that you start to use data tables way too early. I've seen many cases like XSOS when speed drops greatly because of it with the previous versions. IMO it should start from 6.

  22. #52
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    are you sure that datatables are problem? can you test with -t0 (using tor-full)?

    what's your cpu (especially cache sizes)?

  23. #53
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    are you sure that datatables are problem? can you test with -t0 (using tor-full)?

    what's your cpu (especially cache sizes)?
    Before the speed was dropping with -4. Now -3.
    It's the only reason I see, other things change slightly between the modes.
    You say that it shouldn't be this way?

    Pentium D 805 2.66Ghz, FSB 533 mhz, 16+12 KB L1 (data+instr.), 1 MB L2.

  24. #54
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    can you test it with -t0?

  25. #55
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Strange thing. Huge variance. Results:
    11.766
    20.469
    18.312
    20.437
    10.703
    20.344
    11.172

    Practically only FireFox running during the test.
    CPU time: 6.2-6.8s.

  26. #56
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    so, case closed?

    if cpu times is more stable, you may compare them

  27. #57
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    so, case closed?

    if cpu times is more stable, you may compare them
    Which end users care about CPU time? And global times are usually very stable too, on a clean machine I often get 3 results in a row <0.05% from each other.

    1. Possibly you have some serious issue, probably IO related. I'll recheck on a clean machine though.

    As we're at the topic, I have to report that I've seen such variation on a clean machine only once. When testing FreeArc on BMP, IIRC only higher LZMA modes. They usually run 120s or 130s (don't remember) and sometimes ~90s.

    2. -t1 CPU usage: 21s.

  28. #58
    Member jo.henke's Avatar
    Join Date
    Dec 2008
    Location
    Europe
    Posts
    56
    Thanks
    0
    Thanked 4 Times in 1 Post
    Just a small hint for people, who want to build a small and fast tor binary with minimal library dependence in Linux (nice for embedded usage):
    • start from the command line in the file "compile"
    • add "-DFREEARC_NO_TIMING -fno-threadsafe-statics"
    • remove "-lrt"
    • replace "g++" by "gcc" and "-funroll-loops" by "-fprofile-generate"
    • compile
    • run the resulting binary with all the parameters you typical use (compression and decompression)
    • compile again with "-fprofile-use" instead of "-fprofile-generate"

    In my case the final binary is 116352 bytes large and depends only on libc:
    Code:
    $ ldd tor
    	linux-gate.so.1 =>  (0xf7f78000)
    	libc.so.6 => /lib/libc.so.6 (0xf7e07000)
    	/lib/ld-linux.so.2 (0xf7f79000)
    Who would expect it to be a C++ program?

  29. #59
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    734
    Thanked 660 Times in 354 Posts
    i don't changed i/o in tornado for a long time. overall, i don't think that unstable execution times is due to smth in tornado rather than your system

    2. we need to compare -3 -t0 and -3 -t1 times to make any conclusions

    jo.henke, why not make new compile script based on your suggestions? you can compress and decompress tor executable itself in -1..-11 modes (with reduced memory, of course) and make it easy for anyone to replace ./tor with any larger file they have

  30. #60
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    i don't changed i/o in tornado for a long time. overall, i don't think that unstable execution times is due to smth in tornado rather than your system
    This is very possible, but still I'd like to check it. Sorry, can't tell you when.

    Quote Originally Posted by Bulat Ziganshin View Post
    2. we need to compare -3 -t0 and -3 -t1 times to make any conclusions
    I meant "-3 -t1", I know there's no point comparing totally different modes.
    3 t0: 6.2-6.8s.
    3 t1: ~21s.

Page 2 of 19 FirstFirst 123412 ... LastLast

Similar Threads

  1. Dict preprocessor
    By pat357 in forum Data Compression
    Replies: 5
    Last Post: 2nd May 2014, 21:51
  2. Wav Compression in Freearc, from TTA to TAK ?
    By eleria in forum Data Compression
    Replies: 10
    Last Post: 4th February 2010, 21:14
  3. REP and Delta fails with big files
    By SvenBent in forum Data Compression
    Replies: 14
    Last Post: 23rd November 2008, 19:41
  4. 4x4 bug?
    By m^2 in forum Data Compression
    Replies: 11
    Last Post: 15th November 2008, 19:25
  5. 4x4 - multithreaded compressor
    By Bulat Ziganshin in forum Forum Archive
    Replies: 12
    Last Post: 19th April 2008, 17:25

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •