Page 71 of 71 FirstFirst ... 2161697071
Results 2,101 to 2,112 of 2112

Thread: paq8px

  1. #2101
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    94
    Thanks
    102
    Thanked 39 Times in 25 Posts
    Hi Darek, really top much time. I don't know if the next time
    I will test the lstm version.
    I would ask to the "guru" if they think
    That code may be tuned..
    Luca

  2. #2102
    Member
    Join Date
    Sep 2014
    Location
    Italy
    Posts
    94
    Thanks
    102
    Thanked 39 Times in 25 Posts
    My big testset: V192 -9 vs. v193 -9

    Click image for larger version. 

Name:	image.png 
Views:	16 
Size:	51.2 KB 
ID:	7898
    Click image for larger version. 

Name:	image.png 
Views:	17 
Size:	57.7 KB 
ID:	7899
    Good gain on JPG!

    JPEG: -23297,00 KB
    PDF: -2404,00 KB
    DOC: -719,00 KB
    MP3: 0,00 KB
    MP4: 0,00 KB
    XML: 0,00 KB
    TXT: 0,00 KB
    BAK: -145,00 KB
    ZIP: 0,00 KB
    EXE: -803,00 KB
    ISO: -2172,00 KB
    TOTAL: -29540,00

    Compression time +5%

    Great works guys!

  3. Thanks:

    Gotty (14th September 2020)

  4. #2103
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    340
    Thanks
    50
    Thanked 62 Times in 50 Posts
    Quote Originally Posted by LucaBiondi View Post
    My big testset: V192 -9 vs. v193 -9

    Click image for larger version. 

Name:	image.png 
Views:	16 
Size:	51.2 KB 
ID:	7898
    Click image for larger version. 

Name:	image.png 
Views:	17 
Size:	57.7 KB 
ID:	7899
    Good gain on JPG!

    JPEG: -23297,00 KB
    PDF: -2404,00 KB
    DOC: -719,00 KB
    MP3: 0,00 KB
    MP4: 0,00 KB
    XML: 0,00 KB
    TXT: 0,00 KB
    BAK: -145,00 KB
    ZIP: 0,00 KB
    EXE: -803,00 KB
    ISO: -2172,00 KB
    TOTAL: -29540,00

    Compression time +5%

    Great works guys!
    How about decompression ? Can it decompress successfully without corruption ,?

  5. #2104
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Quote Originally Posted by Gotty View Post
    Code:
    - Fixed memory leaks (TextParserStateInfo, Ilog) - cosmetic issue only
    - Unmatched new[] vs delete issue in OLS (cosmetic issue only) is fixed as a side effect of converting all arrays to Array in OLS
    - Small parameter tweak for 8-bit audio, 8-bit image, 24-bit image and generic data (mixer scaling parameters in ContextModel.cpp)
    - LSTM: got rid of most of the valarray-specific operations (GCC seems to implement some valarray operations erroneously, and compression results were different when compiled by GCC and MSVC)
    - LSTM: Activations.hpp: faster sigmoid using tanh instead of exp
    - LSTM: Approximations for tanh and exp (faster than the tanh and exp functions provided by stdlib)
    - LSTM: Probability output of the LSTM model could get negative. Fixed by clipping.
    Remarks:
    Compressing using AVX2 and non-AVX2 mode still produce incompatible results, but GCC and MSVC builds already produce the same output in non-AVX2 mode at least.
    Some speedup with the LSTM model. It's due to the faster exp() and tanh() approximations and also to getting rid of valarray-specific operations. But partly also because I switched from GCC 8.1 to GCC 9.3

    paq8px obj1 -8L

    Pentium Gold G5600 SSE2 mode
    v188 (GCC 7.1): 31.60s
    v189 (GCC 9.3): 28.02s

    Xeon E2286M AVX2 mode
    v188 (GCC 7.1): 11.19s
    v189 (GCC 9.3): 9.72s

    @Gotty, I'll try to submit paq8px_v189 version to LTCB (It's 3'rd place) but to reduce unpacking file I'll try to use source code instead ob exe build.
    Could you post paq8px v189 scource code or point me where it is?

    I wrote "I'll try" because I've tried to submit 2 scores for Silesia (paq8px v190 and paq8px v192) and wrote mail to Matt but there no answer
    However Byron find a way to do it with tensorflow-compress recently - maybe he can help.

  6. #2105
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    551
    Thanks
    356
    Thanked 355 Times in 192 Posts
    Here is the source code of v189.
    Feel free to delete the following files (not strictly part of the paq8px "source code"): README, DOC, CHANGELOG, and the following folders: zlib/ and build/
    However if you used text pretraining (you did of course) than the dictionaries in build/ should probably stay.
    I remember: with enwik8 the
    source+result with -9a (i.e. with no pretraining)was better then
    source+dictionaries+result with -9at (i.e. with text-pretraining). I didn't test the same lately but I suspect that it still stands: using no pretraining and excluding the dictionaries from the "package" would give slightly better result.
    Attached Files Attached Files

  7. Thanks (2):

    Darek (14th September 2020),Mike (15th September 2020)

  8. #2106
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    551
    Thanks
    356
    Thanked 355 Times in 192 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    How about decompression ? Can it decompress successfully without corruption ,?
    I know that Darek tests decompression, with some complex files I too test decompression. As far as I can tell, yes - it can decompress properly. Anyway, if anyone experiences any problem with decompression, they usually report it here. It happens, it's rare, and we fix it immediately.

  9. #2107
    Member
    Join Date
    Mar 2011
    Location
    USA
    Posts
    249
    Thanks
    113
    Thanked 123 Times in 72 Posts
    Quote Originally Posted by Darek View Post
    I wrote "I'll try" because I've tried to submit 2 scores for Silesia (paq8px v190 and paq8px v192) and wrote mail to Matt but there no answer
    However Byron find a way to do it with tensorflow-compress recently - maybe he can help.
    Sure, if you have trouble making a submission, feel free to email me the submission and I can forward it to Matt. I have used email to make my previous submissions.

  10. Thanks:

    Darek (14th September 2020)

  11. #2108
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Quote Originally Posted by suryakandau@yahoo.co.id View Post
    How about decompression ? Can it decompress successfully without corruption ,?
    Did you have any specific version and/or file which have decompression issue?

  12. #2109
    Member
    Join Date
    Aug 2015
    Location
    indonesia
    Posts
    340
    Thanks
    50
    Thanked 62 Times in 50 Posts
    Quote Originally Posted by Darek View Post
    Did you have any specific version and/or file which have decompression issue?
    I just wonder with lucabiondi test set on jpeg. The result is about 23mb. It is not a small compression....

  13. Thanks:

    LucaBiondi (21st September 2020)

  14. #2110
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Quote Originally Posted by Gotty View Post
    Here is the source code of v189.
    Feel free to delete the following files (not strictly part of the paq8px "source code"): README, DOC, CHANGELOG, and the following folders: zlib/ and build/
    However if you used text pretraining (you did of course) than the dictionaries in build/ should probably stay.
    I remember: with enwik8 the
    source+result with -9a (i.e. with no pretraining)was better then
    source+dictionaries+result with -9at (i.e. with text-pretraining). I didn't test the same lately but I suspect that it still stands: using no pretraining and excluding the dictionaries from the "package" would give slightly better result.
    I'll put to unpacking package all text preprocessing and using files (dictionary - fortunatelly (I need to check it deeper) it's the same used in DRT, plus .emb and .ext files) and of course DRT pllus resplit_1423 executable and batch. Version v189 didn't use rnn files yet.

  15. Thanks:

    LucaBiondi (21st September 2020)

  16. #2111
    Member
    Join Date
    Dec 2008
    Location
    Poland, Warsaw
    Posts
    1,217
    Thanks
    743
    Thanked 495 Times in 383 Posts
    Some enwik scores (mostly enwik8 ) for the latest paq8px versions:

    16'190'519 - enwik8 -12 by Paq8px_v187fix2
    16'080'588 - enwik8 -12eta by Paq8px_v187fix2
    15'889'931 - enwik8.drt -12eta by Paq8px_v187fix2
    127'626'051 - enwik9_1423 -12eta by Paq8px_v187fix2
    124'786'260 - enwik9_1423.drt -12eta by Paq8px_v187fix2

    15'900'206 - enwik8 -12leta by Paq8px_v188, change: -1,12%
    15'503'221 - enwik8.drt -12leta by Paq8px_v188, change: -2,43%

    15'907'081 - enwik8 -12leta by Paq8px_v188b, change: 0,04%
    15'505'761 - enwik8.drt -12leta by Paq8px_v188b, change: 0,02%

    15'896'588 - enwik8 -12leta by Paq8px_v189, change: -0,07%
    15'490'302 - enwik8.drt -12leta by Paq8px_v189, change: -0,10%
    121'056'858 - enwik9_1423.drt -12leta by Paq8px_v189, change: -2,99% - best score measured

    15'896'196 - enwik8 -12leta by Paq8px_v190, change: 0,00%
    15'896'126 - enwik8 -12lreta by Paq8px_v190, change: 0,00%
    15'490'045 - enwik8.drt -12leta by Paq8px_v190, change: 0,00%
    15'490'045 - enwik8.drt -12lreta by Paq8px_v190, change: 0,00%

    15'888'954 - enwik8 -12lreta by Paq8px_v191, change: -0,05%
    15'638'056 - enwik8.drt -12lreta by Paq8px_v191, change: 0,96%

    15'898'305 - enwik8 -12lreta by Paq8px_v191a, change: 0,01%
    15'638'254 - enwik8.drt -12lreta by Paq8px_v191a, change: 0,96%

    15'898'544 - enwik8 -12lreta by Paq8px_v192, change: 0,00%
    15'888'545 - enwik8 -12leta by Paq8px_v192, change: -0,06%
    15'638'126 - enwik8.drt -12lreta by Paq8px_v192, change: 0,00%
    15'479'471 - enwik8.drt -12leta by Paq8px_v192, change: -1,02%

    15'884'947 - enwik8 -12lreta by Paq8px_v193, change: -0,02% - best score measured
    15'476'230 - enwik8.drt -12lreta by Paq8px_v193, change: -0,02%- best score measured
    126'150'760 - estimated enwik9_1423 -12lreta by Paq8px_v193, change: -0,02% - estimated best score - time to compress about 6 days
    120'946'885 - estimated enwik9_1423.drt -12lreta by Paq8px_v193, change: -0,02% - estimated best score - time to compress about 4 days

    15'899'080 - enwik8 -12lreta by Paq8px_v193fix2, change: 0,09%
    15'477'066 - enwik8.drt -12lreta by Paq8px_v193fix2, change: 0,01%

    Looks that paq8px v193 got the best scores for enwik8 and probably on enwik9, however until the test, the best score of enwik9 belongs to paq8px v189.
    It's close to break 120'000'000 barrier but there is about 1MB to gain

  17. Thanks:

    Gotty (20th September 2020)

  18. #2112
    Member Gotty's Avatar
    Join Date
    Oct 2017
    Location
    Switzerland
    Posts
    551
    Thanks
    356
    Thanked 355 Times in 192 Posts
    Paq8px (and cmix) is mentioned (and used as an inspiration and baseline) in this paper:
    Performance and Implementation Modeling of Gated Linear Networks on FPGA for Lossless Image Compression

  19. Thanks (5):

    Darek (21st September 2020),LucaBiondi (21st September 2020),Mike (21st September 2020),schnaader (21st September 2020),Shelwien (21st September 2020)

Page 71 of 71 FirstFirst ... 2161697071

Similar Threads

  1. FrontPAQ - GUI frontend for PAQ8PF and PAQ8PX
    By LovePimple in forum Download Area
    Replies: 26
    Last Post: 17th January 2019, 13:36
  2. Alternative paq8px builds
    By M4ST3R in forum Download Area
    Replies: 20
    Last Post: 25th June 2010, 16:19
  3. Optimized paq7asm.asm code not compatible with paq8px?
    By M4ST3R in forum Data Compression
    Replies: 7
    Last Post: 3rd June 2009, 15:34

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •