Page 8 of 8 FirstFirst ... 678
Results 211 to 223 of 223

Thread: MONSTER OF COMPRESSION - New Benchmark -

  1. #211
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    ive compiled tornado 0.5alpha with 128k and 256k input blocks. can you try them to compare?
    Ill check it out and post results later.

  2. #212
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    775
    Thanked 687 Times in 372 Posts
    for my box, 256k still much faster:
    <div class=""jscript""><pre>>timer tor-128k.exe -2 dll700.dll

    Kernel Time = 6.289 = 00:00:06.289 = 9%
    User Time = 32.767 = 00:00:32.767 = 50%
    Process Time = 39.056 = 00:00:39.056 = 59%
    Global Time = 65.204 = 00:01:05.204 = 100%


    >timer tor-256k.exe -2 dll700.dll

    Kernel Time = 5.848 = 00:00:05.848 = 9%
    User Time = 32.526 = 00:00:32.526 = 52%
    Process Time = 38.375 = 00:00:38.375 = 62%
    Global Time = 61.559 = 00:01:01.559 = 100%</pre>[/QUOTE]


    i think that this depends on HDD firmware

    I didnt receive a activation mail yet.
    nor me

    Quote Originally Posted by Christian
    Memory mapped does not perform well.
    for me too, ive tested it also (actually, once ive wrote rather universal i/o library)

    [QUOTE=Christian]<div class=""quoting"">Quoting: Christian</div>For HDDs simple IO is almost as fast as async I/O</div>
    well, ive compared it to tornado, may be its really just not optimized. i will experiment with slug if you claim that it uses simple i/o

  3. #213
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Here are the results:

    Code:
    ------------------ 
    tor-128k -3 
    ------------------ 
    User Time    =    12.343 
    Global Time  =    21.938 
     
    User Time    =    12.125 
    Global Time  =    20.828 
     
    User Time    =    12.750 
    Global Time  =    24.250 
     
    User Time    =    12.187 
    Global Time  =    22.437 
     
    -> 425.802.874 
    ------------------ 
    tor-256k -3 
    ------------------ 
    User Time    =    12.109 
    Global Time  =    22.094 
     
    User Time    =    12.140 
    Global Time  =    21.844 
     
    User Time    =    12.265 
    Global Time  =    23.750 
     
    User Time    =    12.218 
    Global Time  =    22.031 
     
    -> 425.802.874

    Hmmm, maybe Filemon doesn't work correctly, but it seems that tor-128k is reading 128k blocks the first 8M only. After that, it always reads 4k followed by 124k. It still outputs 8M blocks. It's the same with tor-256k - 256k on the first 8M, than 4k + 252k. Maybe this helps. Additionally, maybe the 8M are hurting, too.

  4. #214
    Member
    Join Date
    Dec 2006
    Posts
    611
    Thanks
    0
    Thanked 1 Time in 1 Post
    Quote Originally Posted by Christian
    I didnt receive a activation mail yet
    I got it instantly, it only landed into spam folder (gmail).

  5. #215
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by joey
    you wrote:

    6PACK -1 , 6pack reads 128K -> 574.618.151
    6PACK_OPT -1 , 6pack_opt seems to be reading 64K -> 577.201.364

    6PACK_OPT is faster but worse in compression?
    Its compression is slightly worse and speed is really close. But compression speed (process time) might be better on systems with smaller cache. Maybe LovePimple can shed some light on the improvements.

    Quote Originally Posted by joey
    what about to make it testable for us ?

    would it be possible to have here a commandline parameter ?
    If you look at my tools youll notice that I dont like to many commandline parameters. Maybe Ill add a switch, but please dont count on it. Additionally, I chose 64k because many very known tools use this size, too.

    Quote Originally Posted by joey
    and would it be possible (in a further version)
    to add a directory-support? - may be only a dream?
    No plans for that, sorry. Id really like to, but I dont have enough time. And the little spare time I have I spend with my girlfriend/friends/hobbies and sometimes developing compression algorithms.

  6. #216
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    i will experiment with slug if you claim that it uses simple i/o
    I even dont know if Slugs IO works good on other systems (except mine and my girlfriends laptop). But yes, it uses simple IO (e.g. fread and fwrite). The idea is to let the OS do all the async stuff. If we read/write only small blocks the OS will do the async work with its read-ahead and cache-behind systems. We just have to make it easy for the OS to guess - so, Slug always reads/writes exactly 64K. FYI, Thor always reads/writes exactly 32K. Maybe this is all bogus, but explorer, copy and some other tools work like this, too. Just use Filemon and look around a little bit.

  7. #217
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    775
    Thanked 687 Times in 372 Posts
    Quote Originally Posted by Christian
    Hmmm, maybe Filemon doesnt work correctly, but it seems that tor-128k is reading 128k blocks the first 8M only. After that, it always reads 4k followed by 124k. It still outputs 8M blocks. Its the same with tor-256k - 256k on the first 8M, than 4k + 252k. Maybe this helps. Additionally, maybe the 8M are hurting, too.
    thanks, ill check it

    Quote Originally Posted by Christian
    Maybe this is all bogus, but explorer, copy and some other tools work like this, too. Just use Filemon and look around a little bit.
    well, i mean that i thought that thor at least used some b/g thread or async calls, even with the same 32k blocks - i think its hard to check this with Filemon?

  8. #218
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    well, i mean that i thought that thor at least used some b/g thread or async calls, even with the same 32k blocks - i think its hard to check this with Filemon?
    Yes, you can not check this with Filemon. But Process Explorer shows only one thread.

    Im off to bed. Goodnight everyone!

  9. #219
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    775
    Thanked 687 Times in 372 Posts
    Quote Originally Posted by Christian
    But Process Explorer shows only one thread.
    there is also async i/o

  10. #220
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    there is also async i/o
    Where do you find this in PE?

    Btw., my own tests showed that overlapped IO is not worth it when reading/writing small blocks (e.g. 32k, 64k). The OS does the same thing anyway - at least when youre not using things like "FILE_FLAG_NO_BUFFERING". Well, you might loose a little bit of memory bandwidth from the in-memory-copying, but it does not make any difference here.

  11. #221
    Programmer Bulat Ziganshin's Avatar
    Join Date
    Mar 2007
    Location
    Uzbekistan
    Posts
    4,564
    Thanks
    775
    Thanked 687 Times in 372 Posts
    Quote Originally Posted by Christian
    Where do you find this in PE?
    i mean that such apis exist and they may be used even in 1-threaded program. but if this is can be checked by analyzing executable, its great

    Quote Originally Posted by Christian
    Well, you might loose a little bit of memory bandwidth from the in-memory-copying, but it does not make any difference here.
    this depends. memcpy is 230mb/sec and tor -1 is >50mb/sec on my box

  12. #222
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    i mean that such apis exist and they may be used even in 1-threaded program. but if this is can be checked by analyzing executable, its great
    I dont know a way to check if a program uses overlapped IO. But I can tell that Thor does not use threads and reads/writes 32k all of the time. Further, I checked overlapped IO and it didnt make a difference while being more complex.

    Quote Originally Posted by Bulat Ziganshin
    this depends. memcpy is 230mb/sec and tor -1 is >50mb/sec on my box
    Right, but L2-cache is coming into play here. And cached copying is much much faster.

    Nonetheless, imo, the best solution is to use threading. Its platform independent and you dont have to rely on the OS. But since simple IO works surprisinly good, I prefer it - well, because its simple.

  13. #223
    Programmer
    Join Date
    Feb 2007
    Location
    Germany
    Posts
    420
    Thanks
    28
    Thanked 160 Times in 18 Posts
    Quote Originally Posted by Bulat Ziganshin
    but if this is can be checked by analyzing executable, its great
    Actually you can use a disassembler and check the fopen/CreateFile/whatever calls and look at the used flags. But I distaste such practice strongly.
    Btw., Im hoping that Metacompressor goes online again. Id love to have some more detailed results for Slug. Testing on my system all the time is useless - because Slug was tailored to work good on it.

Page 8 of 8 FirstFirst ... 678

Similar Threads

  1. HFCB: Huge Files Compression Benchmark
    By Bulat Ziganshin in forum Data Compression
    Replies: 129
    Last Post: 6th January 2015, 14:31
  2. MONSTER OF COMPRESSION - New Benchmark
    By LovePimple in forum Data Compression
    Replies: 225
    Last Post: 23rd December 2009, 10:57
  3. New benchmark for generic compression
    By Matt Mahoney in forum Data Compression
    Replies: 20
    Last Post: 29th December 2008, 08:20
  4. Compression speed benchmark
    By Sportman in forum Forum Archive
    Replies: 104
    Last Post: 23rd April 2008, 16:38
  5. Synthetic compression benchmark
    By giorgiotani in forum Forum Archive
    Replies: 6
    Last Post: 3rd March 2008, 11:14

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •