Page 3 of 5 FirstFirst 12345 LastLast
Results 61 to 90 of 139

Thread: DARK - a new BWT-based command-line archiver

  1. #61
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Sometimes a little 'fallback' can be very useful

  2. #62
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    I'm sorry to hear about the HDD crash!

    Did you back-up the source code?

  3. #63
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Version 0.38 if i'm not mistaken was the last backuped, I will get this backup only tomorrow, so I don't know exactly. Right now I have only 0.36 But don't worry! New code will be more beautiful, fast and stable. It's time to give 'dark' a new birth

    Theoreticaly I have an aweful improvement (up to 0.50), the one and only problem is my time...

  4. #64
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Version 0.38 if i'm not mistaken was the last backuped, I will get this backup only tomorrow, so I don't know exactly. Right now I have only 0.36 But don't worry! New code will be more beautiful, fast and stable. It's time to give 'dark' a new birth

    As this is a going to be a rebirth; are you going to add the dual (FAST and Super-High) compression levels we talked about earlier?

    Maybe you could have three compression levels like THOR?


    Theoreticaly I have an aweful improvement (up to 0.50), the one and only problem is my time...

    I understand!

  5. #65
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Currently (If not count that project is freezed for a while ) I'm working on the compression ratio AND compression fast/slow modes

    Very possibly for high speed the ST6 (shindler Transform) will be used, but It's only one variant (decompression is slower).

  6. #66
    Guest
    It's Stephan Busch here (Squeeze Chart 2006)

    On one of my large testsets (Encyclopedia) your Dark 0.41
    fails; it keeps cpu load at 100% and works, but filesize of the
    archive does not grow. I closed commandline window after 2 hours
    and tried lower mem settings, but it does not finnish compression, neither the tar file nor the normal files in directory..
    Dark's compression is amazing and better than your competitor YBS
    in my benchmarks; and I would like to also see a wav and bmp filter, and maybe the jpg lossless recompressor of PAQ7 / PAQ8;
    and some tricks for text preprocessing would also be nice.
    I apreciate your work and if I finnish the last testset, I will
    release my new Squeeze Chart with your programs results.


    Yours,

    Stephan Busch
    squeezechart@gmx.de

  7. #67
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Thanks, Stephan!

    >On one of my large testsets (Encyclopedia) your Dark 0.41
    >fails; it keeps cpu load at 100% and works, but filesize of the
    >archive does not grow. I closed commandline window after 2
    >hours and tried lower mem settings, but it does not finnish
    > compression, neither the tar file nor the normal files in directory..

    It has some bugs still... Thank you for testing!
    As I sad before, 'dark' will reborn, soon. (Of course, without this bug)

    > Dark's compression is amazing and better than your competitor
    > YBS in my benchmarks;

    Dark had never been better than ybs on texts, but can be in near future (without filters). May be it's the perfect fragmentor module's work.

    > and I would like to also see a wav and bmp filter, and maybe the > jpg lossless recompressor of PAQ7 / PAQ8. and some tricks for
    > text preprocessing would also be nice

    Roger that

    Thanks again!
    Dmitry Malyshev

  8. #68
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Dark's compression is amazing

    Never underestimate the POWER of the DARK side!

  9. #69
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    I would like to also see a wav and bmp filter, and maybe the jpg lossless recompressor of PAQ7 / PAQ8;
    and some tricks for text preprocessing would also be nice.
    I apreciate your work


    I agree!

  10. #70
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    To tell the truth, I don't satisfied with dark's compression engine at all... If you know what I know, you'd say it must be much better
    Fragmentator is a powerful thing, especially in solid mode

    BTW, Dark-0.50 will sort files before compressing and pack the path list, It will increase solid compression.
    And what is amazing for me - I'm using dark myself! I've compressed some old games with dark-0.41 better then 7z and WinRAR in PPMD mode

  11. #71
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    To tell the truth, I don't satisfied with dark's compression engine at all... If you know what I know, you'd say it must be much better

    I understand this but I have faith that you will eventually make it *MUCH* better.


    BTW, Dark-0.50 will sort files before compressing and pack the path list, It will increase solid compression.
    And what is amazing for me - I'm using dark myself! I've compressed some old games with dark-0.41 better then 7z and WinRAR in PPMD mode


    Yes; it could be a very useful archiver if it had both FAST and Super-HIGH compression modes. It would be much better if it had selectable FAST, MEDIUM, and Super-HIGH compression though!

  12. #72
    Guest
    New super-archiver with GUI? Easier said than done... Anyway, lets wait and see!

  13. #73
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    New super-archiver with GUI? Easier said than done... Anyway, lets wait and see!

    Don't really need the GUI myself. I'm quite happy with the command line version.

  14. #74
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Dark0.45 - revision 9 (from 0.36)

    1) Memory requiments - 5N (!!!!) (pack/unpack)
    2) Archon lost 'Anchors module' (*)
    3) Archon has got 'Tandem Repeats module' (*)
    4) Reverse and Infolev flags
    5) Solid archive: filesorting (+compression?)
    6) Interface changed (c/x, %, ...)
    7) OpenWatcom 1.0 ( was 1.4)

    But! It hadn't tested carefully yet. Some sort of beta...
    http://kvark.at.tut.by/dark-0.45r9.zip

    (*) see Archon3 final structure,
    http://darchiver.narod.ru/dark/Archon3fs.doc

  15. #75
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks for the update!

    Compression is not as good on some files but its noticably faster. Using the reverse sorting order (-r) has given some interesting results with some files compressing better than with any previous version of Dark.

  16. #76
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Faster?!

    On my machine it's 1.5 times slower than 0.41b (I thought because of Watcom ver 1.0).
    Also thanks for '-r' testing! I hadn't found such files myself

  17. #77
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    dark-0.45, the 'true' first version of new-born dark
    http://kvark.at.tut.by/dark-0.45.zip
    Homepage will be updated later.

    Changes(since 0.45r9):
    - Solved solid decompression problem. No more big temp files, increased speed
    - Several fixes. This version seems to be stable (much more than dark-0.41b).
    - compiled with OpenWatcomC 1.5

  18. #78
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Do you ready for the next dark?
    Look at it's homepage and welcome 'dark-0.46'!

    book1: 768771 -> 215000 bytes )

  19. #79
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Thanks for the updates!

    I'll test them as soon as possible and let you know my results.

  20. #80
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    I've tested v0.46 pretty thoroughly and so far no problems found.

    I'll continue to test it while we wait for Werner's benchmark results to be published.


    Faster?!

    On my machine it's 1.5 times slower than 0.41b (I thought because of Watcom ver 1.0).


    I can't understand why its so much slower on your machine. Even when using the default settings it shows good speed on my tests.

    Here are some timings using only the default settings for all versions of Dark.

    Test files: 2,201 files: txt, bmp, exe, dll, html etc
    Total uncompressed size: 397 MB (416,954,653 bytes)

    Dark v0.41b

    Compressed size: 169 MB (178,237,336 bytes)

    Compression time:
    Kernel Time = 6.562 = 00:00:06.562 = 2%
    User Time = 271.078 = 00:04:31.078 = 93%
    Process Time = 277.640 = 00:04:37.640 = 95%
    Global Time = 289.234 = 00:04:49.234 = 100%


    Dark v0.45r9

    Compressed size: 169 MB (178,237,336 bytes)

    Compression time:
    Kernel Time = 6.750 = 00:00:06.750 = 2%
    User Time = 271.046 = 00:04:31.046 = 93%
    Process Time = 277.796 = 00:04:37.796 = 96%
    Global Time = 288.843 = 00:04:48.843 = 100%


    Dark v0.46

    Compressed size: 169 MB (178,237,336 bytes)

    Compression time:
    Kernel Time = 6.203 = 00:00:06.203 = 2%
    User Time = 272.812 = 00:04:32.812 = 91%
    Process Time = 279.015 = 00:04:39.015 = 93%
    Global Time = 298.672 = 00:04:58.672 = 100%

  21. #81
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Here are my timings for compressing ENWIK8 with 20m block size.

    Dark v0.46
    Block: 20m, Frag: No, Solid: No
    Compressed size: 21.9 MB (23,007,621 bytes)

    Compression time:
    Kernel Time = 0.984 = 00:00:00.984 = 1%
    User Time = 88.750 = 00:01:28.750 = 93%
    Process Time = 89.734 = 00:01:29.734 = 94%
    Global Time = 94.828 = 00:01:34.828 = 100%

    Good compression considering the 20m block size!


    Compressed size with 30m block size (beats WinRAR 3.6).
    21.5 MB (22,604,036 bytes)

    Compressed size with 128m block size.
    20.2 MB (21,231,325 bytes)

  22. #82
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Are you sure posted results are correct? They cannot match byte to byte!

    Anyway, thanks a lot, LovePimple!!!

    You've done a great work! I'm satisfied with dark-0.46 stability
    Will wait for Werner and Matt

    Upd. I see the improvement is easy to miss here... I was tuning model for little files, book1 especially. May be I must check big files too...
    BTW, what about 160Mb block?

  23. #83
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Are you sure posted results are correct? They cannot match byte to byte!

    Yes, posted results are correct. I still have the original compressed files and have double checked the results.

    All timings were obtained by using this program.
    URL


    Upd. I see the improvement is easy to miss here... I was tuning model for little files, book1 especially. May be I must check big files too...
    BTW, what about 160Mb block?


    I don't have enough RAM to test with 160Mb block size.

  24. #84
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    It's impossible!
    Can you send me at least 1 file were the compression matches byte-to-byte? I have never seen such situation myself (I mean dark-0.45r9 - dark-0.46).

  25. #85
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Apologies, you are of course correct!

    Due to an error in my batch script the incorrect version of Dark was being used during the test. The correct results are posted below.


    Test files: 2,201 files: txt, bmp, exe, dll, html etc
    Total uncompressed size: 397 MB (416,954,653 bytes)

    Dark v0.41b

    Compressed size: 170 MB (178,743,358 bytes)

    Compression time:
    Kernel Time = 6.640 = 00:00:06.640 = 2%
    User Time = 266.296 = 00:04:26.296 = 92%
    Process Time = 272.937 = 00:04:32.937 = 94%
    Global Time = 288.766 = 00:04:48.766 = 100%


    Dark v0.45r9

    Compressed size: 170 MB (178,972,955 bytes)

    Compression time:
    Kernel Time = 6.562 = 00:00:06.562 = 2%
    User Time = 286.921 = 00:04:46.921 = 92%
    Process Time = 293.484 = 00:04:53.484 = 94%
    Global Time = 309.500 = 00:05:09.500 = 100%


    Dark v0.46

    Compressed size: 169 MB (178,237,336 bytes)

    Compression time:
    Kernel Time = 6.390 = 00:00:06.390 = 2%
    User Time = 270.828 = 00:04:30.828 = 92%
    Process Time = 277.218 = 00:04:37.218 = 94%
    Global Time = 293.500 = 00:04:53.500 = 100%

  26. #86
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Test Files: Calgary Corpus (14 files)
    Total uncompressed size: 2.99 MB (3,141,622 bytes)

    Dark v0.41 (C)kvark, 2006
    Block: 4m; Frag: auto; Solid: no
    Allocated memory: 24mb
    Total time: 1547ms

    Compressed size: 766 KB (785,226 bytes)


    Dark v0.45r9 (C)kvark, 2006
    Block: 4m; Frag: Auto; Solid: No;
    Memory usage: 20 Mb
    Total time: 1594ms

    Compressed size: 767 KB (785,658 bytes)


    Dark v0.46 (C)kvark, 2006
    Block: 4m, Frag: Auto, Solid: No
    Memory usage: 20Mb
    Total time: 1531ms

    Compressed size: 765 KB (783,710 bytes)

  27. #87
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Test Files: All 10 SFC files from maximumcompression.com
    Total uncompressed size: 50.6 MB (53,134,726 bytes)

    Dark v0.41 (C)kvark, 2006
    Block: 4m; Frag: auto; Solid: no
    Allocated memory: 24mb
    Total time: 41219ms

    Kernel Time = 0.406 = 00:00:00.406 = 0%
    User Time = 40.312 = 00:00:40.312 = 97%
    Process Time = 40.718 = 00:00:40.718 = 98%
    Global Time = 41.250 = 00:00:41.250 = 100%

    Compressed size: 12.1 MB (12,756,916 bytes)


    Dark v0.45r9 (C)kvark, 2006
    Block: 4m; Frag: Auto; Solid: No;
    Memory usage: 20 Mb
    Total time: 50219ms

    Kernel Time = 0.437 = 00:00:00.437 = 0%
    User Time = 49.171 = 00:00:49.171 = 97%
    Process Time = 49.609 = 00:00:49.609 = 98%
    Global Time = 50.218 = 00:00:50.218 = 100%

    Compressed size: 12.1 MB (12,771,462 bytes)


    Dark v0.46 (C)kvark, 2006
    Block: 4m, Frag: Auto, Solid: No
    Memory usage: 20Mb
    Total time: 43922ms

    Kernel Time = 0.359 = 00:00:00.359 = 0%
    User Time = 43.078 = 00:00:43.078 = 98%
    Process Time = 43.437 = 00:00:43.437 = 98%
    Global Time = 43.938 = 00:00:43.938 = 100%

    Compressed size: 12.1 MB (12,736,798 bytes)

  28. #88
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Test files: 9,234 Files: txt, bmp, exe, dll, html etc, 459 Folders
    Total uncompressed size: 1.62 GB (1,740,468,114 bytes)

    Dark v0.46 (C)kvark, 2006
    Block: 4m, Frag: Auto, Solid: No
    Memory usage: 20Mb
    Total time: 1417734ms

    Kernel Time = 29.500 = 00:00:29.500 = 2%
    User Time = 1319.453 = 00:21:59.453 = 93%
    Process Time = 1348.953 = 00:22:28.953 = 95%
    Global Time = 1417.813 = 00:23:37.813 = 100%

    Compressed size: 739 MB (775,501,285 bytes)

  29. #89
    Programmer kvark's Avatar
    Join Date
    Aug 2006
    Location
    Toronto, Canada
    Posts
    74
    Thanks
    1
    Thanked 1 Time in 1 Post
    Thanx a lot, again!
    Now I see clearly 0.3% improvement and it's very good!

    Werner updated his results. Now dark-0.46 is 48th... poor
    I hope the enwik tests will show better results, especially with 160Mb block

  30. #90
    Moderator

    Join Date
    May 2008
    Location
    Tristan da Cunha
    Posts
    2,034
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Also thanks for '-r' testing! I hadn't found such files myself

    It seems to work best on certain multimedia files. I've had good results using -r on .wav, .bmp, and even some .gif files.

    One example is Werner's rafale.bmp which can be improved from 787197 to 785096 bytes by using the "r" (-b3600kr) switch.

Page 3 of 5 FirstFirst 12345 LastLast

Similar Threads

  1. UCLC (Ultimate Command Line Compressors) Benchmarks
    By osmanturan in forum Data Compression
    Replies: 2
    Last Post: 4th September 2015, 11:30
  2. Replies: 39
    Last Post: 10th April 2014, 22:26
  3. DARK archiver
    By encode in forum Data Compression
    Replies: 7
    Last Post: 30th November 2011, 23:31
  4. BCM v0.01 - New BWT+CM-based compressor
    By encode in forum Data Compression
    Replies: 81
    Last Post: 9th February 2009, 15:47
  5. a small plea for the command line compression developers
    By SvenBent in forum Data Compression
    Replies: 2
    Last Post: 14th June 2008, 02:51

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •