Page 12 of 13 FirstFirst ... 210111213 LastLast
Results 331 to 360 of 361

Thread: EMMA - Context Mixing Compressor

  1. #331
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Quote Originally Posted by mpais View Post
    @Shelwien
    Yes, that was the post I was talking about. I've done a few quick tests with mod_ppmd_v4, compression seems slightly worse than with v3.
    I'll run a few more tests and I'll update the archives with this new version.
    From my test I've got the same insights - for both - my testbed and enwik8 - ppmd_v4 have slightly worse scores for almost every file. For bigger files difference is generally relatively bigger. Same as for text files.

  2. #332
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,267
    Thanks
    200
    Thanked 984 Times in 510 Posts
    Well, v4 includes additional SSE stage - I guess its not compatible with the one already in Emma.
    Ok, here I made a patched v3 - http://nishi.dreamhosters.com/u/mod_ppmd_v3a_dll.rar

  3. The Following 2 Users Say Thank You to Shelwien For This Useful Post:

    Darek (2nd May 2017),mpais (2nd May 2017)

  4. #333
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    As far for now I've got ppmd_mod_v3 scores:

    enwik8 - 16'523'517 - in 7181,2s
    enwik9 -
    134'166'083 - in 76593,1s - quite good improvement - another 1MB of gain - EMMA is slowly approaching to paq8hp12any...

    v4 and v3a tests in progress.

    @Shelwien - question - should v3a give the same results as v3?

  5. #334
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,267
    Thanks
    200
    Thanked 984 Times in 510 Posts
    I backported v3a from v4 sources, but in theory yes.

  6. The Following User Says Thank You to Shelwien For This Useful Post:

    Darek (2nd May 2017)

  7. #335
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    Shelwien, I've updated the archives with your patched v3a version, thank you.

  8. #336
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Quote Originally Posted by Shelwien View Post
    I backported v3a from v4 sources, but in theory yes.
    I've tested all three revisions of ppmd_mod and scores for enwik files for 64 bit EMMA are as follows:

    ppmd_mod v3: enwik8 = 16'523'517 in 7181,2s; enwik9 = 134'166'083 - in 76593,1s
    ppmd_mod v3a: enwik8 = 16'523'517 in 6168,7s; enwik9 = 134'164'521 - in 67097,2s
    ppmd_mod v4: enwik8 = 16'535'653 in 7553,0s; enwik9 = 134'250'780 - in 80558,6s

    Strange thing - mod v3 and mod v3a have the same scores for all my testbed files and enwik8 (both for x86 and 64 versions) but ENWIK9 scores are different...

    Another info: mod v3a looks like the fastest but it was running in best conditions. Generaly timings are similar.

    Darek



  9. #337
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    @Matt - again, could I ask you to put this new score into LTCB table?

    EMMA V1.23 with ppmd_mod v3a by Shelwien:

    ENWIK8: 16'523'517 bytes, time 6168,7s
    ENWIK9: 134'164'521 bytes, time 67097,2s - again 1MB beter than previous best score!

    Decompressor batch file size (attached in post) = 1'079'026 bytes compressed by 7zip. I've added ppmd_mod64.dll v3a file.

    Then the final score for LTCB with my zip is: 135'243'547 bytes.

    Other information for LTCB input:
    System - Core i7 4900MQ 2.8GHz ovwerclocked to 3.8GHz, 32GB, Win7Pro 64
    Memory used: 3800MB
    EMMA 1.23 settings: all settings = MAX, eceept: image and audio models = off, use fast mode on long matches = off, xml=on, x86model=off, x86 exe code = off, delta coding = off, dictionary = off, ppmd memory = 1024, ppmd order = 14
    Regards,
    Darek
    Attached Files Attached Files

  10. #338
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    EMMA v0.1.24, available at goo.gl/EDd1sS

    Code:
    Changes:
    - Ported some of the parsing improvements from PackRAW
    - Improved 2 data structures
    - Bug fixes
    Just a small update, to incorporate what I could from PackRAW into EMMA.
    The data structures I changed are used by the JPEG, GIF, ARW, RW2 and RAW models, so those should get some small gains.

    On a side note, I've upgraded to the latest version of Delphi (10.2), and I've found what appears to be a bug in the x64 compiler.
    I think I've managed to workaround it, but should you experience weird behaviour (crashes, sub-par compression), try the x86
    version and let me know.
    The bug seems to be related to the compiler's dead code removal optimization, since it only happens when I turn on optimizations
    for the x64 build, without it everything is fine, but slower. It doesn't happen with the legacy x86 compiler. And the same code worked
    fine with optimization turned on in XE7, so it's either a regression or a new bug. I've found it in code like this:


    a := ..;
    ...
    while (a<b) do
    begin
    ...
    inc(a);
    end


    The while loop is simply ignored when optimization is turned on, it seems the compiler thinks the value attributed to a is never used.
    Solved by changing the code to:


    a := ..;
    ...
    if (a<b) then
    repeat
    ...
    inc(a);
    until a=b;


    I don't know if there are more Delphi coders here, but in any case maybe this will help.

  11. The Following 5 Users Say Thank You to mpais For This Useful Post:

    Darek (28th July 2017),Hacker (28th July 2017),load (28th July 2017),necros (27th January 2019),Stephan Busch (28th July 2017)

  12. #339
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Scores for my testbed. Small changes in some files Q.WK3, C.TIF, K.WAD and L.PAK, but quite big change in A.TIF file - 6%. Did you change something in record model?
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	emma24.jpg 
Views:	96 
Size:	281.1 KB 
ID:	5066  

  13. #340
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    Thanks for testing Darek. I just tweaked the detection heuristic, in my tests the difference was minimal (a few bytes on most files), apparently it's doing something right for that file.
    In any case, EMMA can't detect the pixel data in that file, so it'll always set it back on your testset.

  14. #341
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Quote Originally Posted by mpais View Post
    Thanks for testing Darek. I just tweaked the detection heuristic, in my tests the difference was minimal (a few bytes on most files), apparently it's doing something right for that file.
    In any case, EMMA can't detect the pixel data in that file, so it'll always set it back on your testset.
    I understand - every program have it's own advantages and disadvantages. Mix of them gives a total result . As you wrote - sometime in future nonstream emma version could cover it.

  15. #342
    Member
    Join Date
    May 2008
    Location
    Kuwait
    Posts
    332
    Thanks
    35
    Thanked 36 Times in 21 Posts
    Today I just noticed that EMMA does not recognize jpeg encoded in base64.. (sample EPS file attached). Can you add such support.
    I think this bash may help..http://www.tcm.phy.cam.ac.uk/sw/psimages
    Attached Files Attached Files

  16. #343
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    EMMA doesn't do preprocessing, so it can't decode it first and then detect the JPEG image, sorry.
    You can try to use precomp first, to see if it can decode it, and then use paq8px(d) on the .pcf file, the improved JPEG model they have is now better than the one in EMMA.

  17. #344
    Member
    Join Date
    Oct 2014
    Location
    South Africa
    Posts
    38
    Thanks
    23
    Thanked 7 Times in 5 Posts
    Bug report:

    I was comparing paq8px with Emma, so I compressed paq8px_v101.exe with Emma preset (Executables MAX); but Emma failed to decompress the file correctly. The decompressed EXE file could not be run. Binary comparison showed lots of differences.

    Please check it.

    paq8px_v101 is available here:

    https://encode.su/threads/342-paq8px...ll=1#post53944

  18. The Following User Says Thank You to msat59 For This Useful Post:

    mpais (29th August 2017)

  19. #345
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    Thanks, I'll check it out. I'm currently working on paq8px and cmix, so it might take a while.

  20. #346
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    EMMA v0.1.24.1 is available at goo.gl/EDd1sS

    This is mostly a bug fix release, so don't expect much difference in terms of compression.
    I've tweaked the record model again and made the JPEG parser accept Extended Sequential DCT mode JPEG's (like those produced by guetzli).

  21. The Following 2 Users Say Thank You to mpais For This Useful Post:

    Darek (30th August 2017),necros (27th January 2019)

  22. #347
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    v24.1 scores for my testbed.
    Record model tweaks give good results for image files without parsers (TIFF) and also other files, especially biggest got some KB of gain.
    Very, very tiny (but always it's something) gains for TEXT files.
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	EMMA_v24_1.jpg 
Views:	69 
Size:	412.3 KB 
ID:	5189  

  23. The Following User Says Thank You to Darek For This Useful Post:

    mpais (30th August 2017)

  24. #348
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts

    EMMA 0.1.25

    Available at goo.gl/EDd1sS

    Code:
    Changes:
    - Improved some data structures
    - Improved 8bpp grayscale image model
    - Improved 24bpp image model
    - Improved record model
    Just a few changes, I haven't had the time to do a complete revamp of the image models in EMMA, but these changes should make them more competitive with the new models in paq8px(d) and cmix.

    I also tweaked the heuristic for record length detection to make it more stable on pixel data, it was oscillating a lot.

  25. The Following 5 Users Say Thank You to mpais For This Useful Post:

    Darek (24th September 2017),Hacker (24th September 2017),Mike (24th September 2017),necros (27th January 2019),Stephan Busch (25th September 2017)

  26. #349
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Scores for my testset. 51KB of gain! Very nice results for image files but also some gains for WAV and the two biggest files.

    Two records:
    1 - for 0.WAV file - maybe It's still not overall record but for nonspecialized compressors it's the best score!
    2 - for B.TGA - emma v25 beat recently topped cmix v13l score!
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	EMMA_v25.jpg 
Views:	58 
Size:	416.9 KB 
ID:	5347  

  27. #350
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    I get 555.144 bytes for A.TIF with preset "Best" on the x64 version.

    It was one of the images I used to test the heuristic for detecting 24bpp images with the record model, along with C.TIF, a few .PPM and some modified .BMP images without any header.

    I still have to run a few more tests, but if it's stable, I'll port it to paq8/cmix.

  28. #351
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Quote Originally Posted by mpais View Post
    I get 555.144 bytes for A.TIF with preset "Best" on the x64 version.
    I'ts great result! Which emma settings are you use for "Best" preset?

    In attached file are my "Best" presets tested by one of the recent versions, but for A.TIF I've got much worse score...
    Attached Thumbnails Attached Thumbnails Click image for larger version. 

Name:	emma_presets.jpg 
Views:	60 
Size:	641.3 KB 
ID:	5355  

  29. #352
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    You use the Delta transform Darek, that will now hurt compression when compared to using just the record model.

  30. #353
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Ok, I've changed it and I've got 570'230 bytes - much better but still worse than your score... hmmm. Maybe there is other set which hurts compression?
    Could you tell me your full settings for Best mode for A.TIF?

  31. #354
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    Your settings are actually better Darek, just turn off the delta transform, the adaptive learning rate and the english dictionary. I get 553.910 bytes using those settings.

  32. The Following User Says Thank You to mpais For This Useful Post:

    Darek (25th September 2017)

  33. #355
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    Thanks. That mean I'll need to retest some settings for emma again.
    According to A.TIF score - emma without TIFF parser is approaching to scores of compressors with TIFF parsers!

  34. #356
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    Well, if the record model can consistently detect the width of 8 and 24bpp images, it's just a matter of using some contexts from those models and compression will improve a lot.

    It's something that may be interesting for other compressors too, since there are a lot of image formats and we can't have parsers for everything.

  35. #357
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    940
    Thanks
    558
    Thanked 380 Times in 284 Posts
    I think it's very good idea to compress nonspecified files with audio or images inside. Especially some game files have lots of imagaes inside without any headers.

  36. #358
    Member
    Join Date
    Jul 2014
    Location
    Mars
    Posts
    173
    Thanks
    121
    Thanked 11 Times in 10 Posts
    Sorry if I missed smthing but my feat. requests:
    1. Drag n drop folder support
    2. Add several files to archive not 1 file = 1 archive, even TARring will be OK
    3. Upgrade to RAD Studio 10.3?
    4. Automatic selection of best options / presets based on user hardware and on input files scanning (by header or stream?) Another step to practical compression)
    5. Github page?

  37. #359
    Member
    Join Date
    Feb 2016
    Location
    Luxembourg
    Posts
    521
    Thanks
    198
    Thanked 745 Times in 302 Posts
    Quote Originally Posted by necros View Post
    Sorry if I missed smthing but my feat. requests:
    1. Drag n drop folder support
    2. Add several files to archive not 1 file = 1 archive, even TARring will be OK
    3. Upgrade to RAD Studio 10.3?
    4. Automatic selection of best options / presets based on user hardware and on input files scanning (by header or stream?) Another step to practical compression)
    5. Github page?
    I've stopped working on this project, sorry.
    I've ported a lot of it to paq8px and cmix, so you can check those out. As it stands, the only advantages of EMMA compared to those are the models for raw images, for which I made PackRAW which is a much more practical tool.
    As for automating the codec selections based on input scanning, that was what Fairytale was intended to do.

  38. #360
    Member
    Join Date
    Oct 2016
    Location
    Slovakia
    Posts
    22
    Thanks
    37
    Thanked 3 Times in 3 Posts
    Quote Originally Posted by mpais View Post
    I've stopped working on this project, sorry.

Page 12 of 13 FirstFirst ... 210111213 LastLast

Similar Threads

  1. Context mixing file compressor for MenuetOS (x86-64 asm)
    By x3k30c in forum Data Compression
    Replies: 0
    Last Post: 12th December 2015, 06:19
  2. Context Mixing
    By Cyan in forum Data Compression
    Replies: 9
    Last Post: 23rd December 2010, 20:45
  3. Simple bytewise context mixing demo
    By Shelwien in forum Data Compression
    Replies: 11
    Last Post: 27th January 2010, 03:12
  4. Context mixing
    By Cyan in forum Data Compression
    Replies: 7
    Last Post: 4th December 2009, 18:12
  5. CMM fast context mixing compressor
    By toffer in forum Forum Archive
    Replies: 171
    Last Post: 24th April 2008, 13:57

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •