Results 1 to 15 of 15

Thread: Some comments on Tornado

  1. #1
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts

    Some comments on Tornado

    I'll start with a bug report.
    Quote Originally Posted by Tornado
    -l# -- length of hash row (1..9999), default 4
    With 9999 Tornado crashes.
    It seems that this value has to be a power of 2.

    Also, some modes could be tweaked a bit.
    From my tests:
    -c2 provides much better compression than -c1 with very low speed penalty. I suggest changing this in -1 mode.
    -c4 provides better efficiency than c3 from mode -3 up. I think that at least -4 should get this.
    -12 -s2 -l256 is always much faster and usually better than the default -s1 -l512.

  2. #2
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    >With 9999 Tornado crashes.

    thanks, will fix sometime

    -1 mode is optimized for speed. btw, note that speed difference may differ between cpus and data being compressed

    >-c4

    that you mean by eficiency? afair, on texts -c4 doesn't provide any substantial compression improvementr oover -c3 while makes program several percents slower even in -5

    >-12 -s2 -l256

    these settings are used in -11 mode -s2 is better than -s1 and i don't remember why -s1 used in -12 mode. probably i just forget to change this setting

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    >With 9999 Tornado crashes.

    thanks, will fix sometime

    -1 mode is optimized for speed. btw, note that speed difference may differ between cpus and data being compressed
    I use 1 CPU, but 6 different test sets. Also text and compressed ones. The biggest measured speed difference was 6.4%. The smallest size gain - 10.0%. This was HDD->HDD, but my HDD is fast and CPU rather slow so usually the speed difference will be even lower.

    Quote Originally Posted by Bulat Ziganshin View Post
    >-c4

    that you mean by eficiency? afair, on texts -c4 doesn't provide any substantial compression improvementr oover -c3 while makes program several percents slower even in -5
    I know 2 efficiency measures, one is maximumcompression.com and the other I developed myself, you can see it in my testing reports.
    In all tests with both measures c4 scored better in -3 and -4.
    Quote Originally Posted by Bulat Ziganshin View Post
    >-12 -s2 -l256

    these settings are used in -11 mode -s2 is better than -s1 and i don't remember why -s1 used in -12 mode. probably i just forget to change this setting
    OK, let me suggest also updating Tornado in FreeArc. You don't use s2 there at all.

  4. #4
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    1. well, i measured pure cpu time, without i/o. i will consider your suggestion when time permits

    2. may be. on my tests on old box (1ghz) it looked natural choice. again, i should recheck it again

    3. i think that tor:9..tor:11 modes use -s2, it's just silent i can't support this parameter explicitly since it will break backward compatibility (it's drawback of FA archive format)

  5. #5
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    1. well, i measured pure cpu time, without i/o. i will consider your suggestion when time permits

    2. may be. on my tests on old box (1ghz) it looked natural choice. again, i should recheck it again
    Nice, thanks

    Quote Originally Posted by Bulat Ziganshin View Post
    3. i think that tor:9..tor:11 modes use -s2, it's just silent i can't support this parameter explicitly since it will break backward compatibility (it's drawback of FA archive format)
    All I know is
    Code:
    // Preconfigured compression modes
    PackMethod std_Tornado_method[] =
        //                 row hashsize  caching buffer parser  hash3 shift update
        { {  0, STORING,     0,       0, 0,      1*mb,  0     , 0 ,     0,  999 }
        , {  1, BYTECODER,   1,   16*kb, 0,      1*mb,  GREEDY, 0 ,     0,  999 }
        , {  2, HUFCODER,    1,   16*kb, 0,      2*mb,  GREEDY, 0 ,     0,  999 }
        , {  3, HUFCODER,    2,   64*kb, 0,      4*mb,  GREEDY, 0 ,     0,  999 }
        , {  4, HUFCODER,    2,    4*mb, 1,      8*mb,  GREEDY, 0 ,     0,  999 }
        , {  5, ARICODER,    4,    8*mb, 1,     16*mb,  LAZY  , 12,     0,  999 }
        , {  6, ARICODER,    8,   16*mb, 1,     32*mb,  LAZY  , 13,     0,  999 }
        , {  7, ARICODER,   16,   32*mb, 1,     64*mb,  LAZY  , 14,     0,    4 }
        , {  8, ARICODER,   32,   64*mb, 1,    128*mb,  LAZY  , 15,     0,    3 }
        , {  9, ARICODER,   64,  128*mb, 1,    256*mb,  LAZY  , 16,     0,    2 }
        , { 10, ARICODER,  128,  256*mb, 1,    512*mb,  LAZY  , 17,     0,    2 }
        , { 11, ARICODER,  256,  512*mb, 1,   1024*mb,  LAZY  , 18,     0,    1 }
        , { 12, ARICODER,  512, 1024*mb, 1,    768*mb,  LAZY  , 19,     0,    1 }
        };

  6. #6
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    now it's

    Code:
        //                        row hashsize  caching buffer parser  hash3 shift update
        { {  0, STORING,   false,   0,       0, 0,      1*mb,  0     ,   0,    0,  999 }
        , {  1, BYTECODER, false,   1,   16*kb, 0,      1*mb,  GREEDY,   0,    0,  999 }
        , {  2, HUFCODER,  false,   1,   16*kb, 0,      2*mb,  GREEDY,   0,    0,  999 }
        , {  3, HUFCODER,  false,   2,   64*kb, 0,      4*mb,  GREEDY,   0,    0,  999 }
        , {  4, HUFCODER,  true,    2,    4*mb, 1,      8*mb,  GREEDY,   0,    0,  999 }
        , {  5, ARICODER,  true,    4,    8*mb, 1,     16*mb,  LAZY  ,   1,    0,  999 }
        , {  6, ARICODER,  true,    8,   16*mb, 1,     32*mb,  LAZY  ,   1,    0,  999 }
        , {  7, ARICODER,  true,   16,   32*mb, 1,     64*mb,  LAZY  ,   1,    0,    4 }
        , {  8, ARICODER,  true,   32,   64*mb, 1,    128*mb,  LAZY  ,   1,    0,    3 }
        , {  9, ARICODER,  true,   64,  128*mb, 2,    256*mb,  LAZY  ,   2,    0,    2 }
        , { 10, ARICODER,  true,  128,  256*mb, 2,    512*mb,  LAZY  ,   2,    0,    2 }
        , { 11, ARICODER,  true,  256,  512*mb, 2,   1024*mb,  LAZY  ,   2,    0,    1 }
        , { 12, ARICODER,  true,  512, 1024*mb, 1,    768*mb,  LAZY  ,   1,    0,    1 }

  7. #7
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    One more thing about -L parameter:
    It should be mentioned that maximum value depends on hash size and i.e. for 16KB (-1 mode) it's 4096.

    Also, I found a comment in the sources about HT5 matchfinder...I feel that it would be much better than current -12. I'm aware that size would grow, but IMO it's worth it.
    Last edited by m^2; 12th November 2008 at 19:48.

  8. #8
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    1. where should i mention it? is the small program's help?
    2. i actually implemented basic version of ht5..ht7 but it need more work to make them really useful. OTOH in lzma ht5..ht7 are really useful since ht4 provides arcghives ~2% larger than bt4 (in *max* mode)

  9. #9
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    1. where should i mention it? is the small program's help?
    "1..hashsize/4" doesn't take much space.
    Quote Originally Posted by Bulat Ziganshin View Post
    2. i actually implemented basic version of ht5..ht7 but it need more work to make them really useful. OTOH in lzma ht5..ht7 are really useful since ht4 provides arcghives ~2% larger than bt4 (in *max* mode)
    Very nice, thanks.

  10. #10
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    1. but harder to understand. his is not a real problem, overall (imho)

  11. #11
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Feature suggestion: automatic memory usage reduction. I just found that tor -12 was compressing a 1.5 GB file for ~30-35 hours. Didn't finish. For comparison, -10 needed 7 minutes.

  12. #12
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    may be. i consider tornado.exe as demo program, it lacks a lot of features for real compressor (data integrity checking, header)

  13. #13
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    may be. i consider tornado.exe as demo program, it lacks a lot of features for real compressor (data integrity checking, header)
    Yes, but this can really block other things...

  14. #14
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,497
    Thanks
    735
    Thanked 660 Times in 354 Posts
    but why you use compression modes which isn't supported by your computer??

  15. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Bulat Ziganshin View Post
    but why you use compression modes which isn't supported by your computer??
    1. No documentation mentions system requirements.
    2. It always worked. Even in a case when file size + hash size slightly exceeded my memory size (when you substract the part used by OS - exceeded significantly), I didn't notice slowdown bigger than expected ~4x compared to -10. This suggested that Tor limits itself.

Similar Threads

  1. FreeArc compression suite (4x4, Tornado, REP, Delta, Dict...)
    By Bulat Ziganshin in forum Data Compression
    Replies: 554
    Last Post: 26th September 2018, 02:41
  2. Tornado 0.4
    By Bulat Ziganshin in forum Forum Archive
    Replies: 83
    Last Post: 15th April 2008, 12:45
  3. Tornado 0.3
    By Bulat Ziganshin in forum Forum Archive
    Replies: 12
    Last Post: 10th March 2008, 06:16
  4. Tornado - fast lzari compressor
    By Bulat Ziganshin in forum Forum Archive
    Replies: 23
    Last Post: 27th July 2007, 13:26
  5. tornado 0.2 is not yet finished...
    By Bulat Ziganshin in forum Forum Archive
    Replies: 15
    Last Post: 11th July 2007, 23:06

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •