Activity Stream

Filter
Sort By Time Show
Recent Recent Popular Popular Anytime Anytime Last 24 Hours Last 24 Hours Last 7 Days Last 7 Days Last 30 Days Last 30 Days All All Photos Photos Forum Forums
  • A_Better_Vice's Avatar
    Today, 11:40
    I have not attempted to re-compress the output files as I do not expect any type of serious gains. I can give it a try for for amusement and maybe it will hit 1% smaller if not have negative gains. I did however compress encrypted files from AxCrypt at again 30% smaller. Only more recently in the last few weeks or so have I had a serious gain in speeds where it went from originally weeks to days to hours to seconds using various techniques. In looking through my code I see that I should easily still be able to chop off some overhead and make it run even faster from removing calls that do not contribute to do anything. Lots of calls all over the place since I was trying so many different techniques until I landed the MUCH more productive technique in terms of speed. Within a few days I should get it to ~1kB per second vs 1kb per second in a non parallel non Linux non Cuda environment not from genius re tweaking or gen two but just cleaning the crap up and gutting calls that have no positive impact to this technique that works. Off to bed soon. My best work is from 11pm to 4am but at around 5am I can start making serious mistakes ... I will be posting some specific decompression speeds as well soon and official benchmarks I guess more than likely on my website. Sent out some feelers to some companies and a few only accept emails from private domains not coldmail or warmmail or even hotmail... So time to fireup my website. So much to do, but happy times are here again :) Hope all is well with everyone. How is Ukraine ? My mother in law is from there.
    8 replies | 447 view(s)
  • xinix's Avatar
    Today, 09:30
    xinix replied to a thread Paq8pxd dict in Data Compression
    i7-3820 4GHz Mem 64gb ____ 125.997.424 bytes - enwik9 pxd_v79_SSE4 -s15 (128167 sec) 125.532.596 bytes - enwik9 pxd_v79_SSE4 -x15 (160497 sec)
    763 replies | 288858 view(s)
  • snowcat's Avatar
    Today, 06:58
    snowcat replied to a thread 2019-nCoV in The Off-Topic Lounge
    I'm a Vietnamese and currently my country have only 204 cases, with 55 cases are fully recovered and 0 death so far... Not sure if I should panick that much but I am panicking af right now, especially when I know how bad it could be. Look at what it did to Italy and USA...
    33 replies | 1904 view(s)
  • uhost's Avatar
    Today, 06:30
    count = how many divisions are taken eg: 3055 Fist Div Result = 1041 After 3 or 4 steps we have 17 or 15 (0) == == (1) ==== (2) === (3) == (4) = (5) 3055 => 1041 => 1007 => 17 => 15 => 1 1 have different angles & generate different master number For example : 01,001,0001 This value is equal but 1 position is different. this method complete success but i try to more effective new method it encoder is completed but decode some complicated that can 3055 =>7 with angles (position )7 ,281474970863668=>32751=>14 with angles (position )7,281474970863667=>32767=>15 with angles (position )7 it take 1 or max 3 step if you do not understand my explanation please forgive me when i complete this method i will explain how to work
    37 replies | 1550 view(s)
  • uhost's Avatar
    Today, 05:56
    :)
    37 replies | 1550 view(s)
  • Self_Recursive_Data's Avatar
    Today, 04:00
    So SSE, how does it work... if I have 17 models to mix, do I use a short model ex. order3 and use it as a goat for the rest? How does this improve compression. Can you explain it clearly? Why "secondary"....where do I pull it from, when, and what do I merge it to... I'm lost...can you elaborate enough context to walk me through this?
    172 replies | 8356 view(s)
  • Shelwien's Avatar
    Today, 03:36
    > Green VS enwiki8. If I remember correctly, Green achieved approx. 21,800,000 bytes. Well, existing implementations of bytewise CM/PPM with SSE (ppmonstr,durilca,ash) seem to reach 18,0xx,xxx only with preprocessing (DRT or similar) - their results are around 19,0xx,xxx normally. > And I'm wondering if you can add *both* SSE & SEE in the same algorithm? Yes, its exactly what ppmonstr/durilca do. But you don't need to encode escapes in a CM, green doesn't keep even escape statistics. > Using the 21.8MB mark, how much would SEE bring it down by? 1MB improvement? You'd need a PPM for that, but yes. > And what about SSE? 0.34MB improvement? This question will let me know how useful SSE vs SEE is. I repeat, there's no one specific SSE implementation. PAQ has a bitwise SSE, while in theory its also possible to precompute the whole byte probability distribution, and with it implement unary SSE like in ppmonstr. SSE is just a secondary model. Its unknown what kind of model it would be, so its hard to predict its performance. Wrong SSE implementation can make compression worse quite easily too.
    172 replies | 8356 view(s)
  • Self_Recursive_Data's Avatar
    Today, 03:17
    > How much is "21.8MB" in bytes? 22,858,956? Green VS enwiki8. If I remember correctly, Green achieved approx. 21,800,000 bytes. And I'm wondering if you can add *both* SSE & SEE in the same algorithm? Using the 21.8MB mark, how much would SEE bring it down by? 1MB improvement? And what about SSE? 0.34MB improvement? This question will let me know how useful SSE vs SEE is.
    172 replies | 8356 view(s)
  • Shelwien's Avatar
    Today, 02:53
    > If we add SEE to Green, and Green already compresses enwiki8 to 21.8MB, How much is "21.8MB" in bytes? 22,858,956? > how much would it improve this? 19.6MB? SEE only makes sense in PPM - it implies a completely different weighting method. http://mattmahoney.net/dc/text.html#1839 ppmd J -m256 -o10 -r1 21,388,296 // Uses SEE ppmd_sh9 16 1863 20,784,856 ppmonstr J -m1863 -o16 19,040,451 // Uses SSE > Now, say we add SSE, too. How much would it improve this "19.6MB"? 18MB? Maybe, depends on specific implementation. SSE is not a fixed function - its any statistical model that uses primary prediction as context. > But isn't this the same thing as Local Text for giving more weight > to Models that have been predicting great recently? > I.e. adaptive Global Model weight favoring models based on error. > Or, is SSE similar but meant for specific context? Normally SSE is orthogonal to weights and primary contexts. It takes already mixed primary predictions and refines them using statistics in context of these predictions (and possibly some other).
    172 replies | 8356 view(s)
  • Self_Recursive_Data's Avatar
    Today, 01:51
    If we add SEE to Green, and Green already compresses enwiki8 to 21.8MB, how much would it improve this? 19.6MB? Now, say we add SSE, too. How much would it improve this "19.6MB"? 18MB? I have another question about SSE. It updates a table based on prediction error, to refine the prediction, basically. But isn't this the same thing as Local Text for giving more weight to Models that have been predicting great recently? I.e. adaptive Global Model weight favoring models based on error. Or, is SSE similar but meant for specific context?
    172 replies | 8356 view(s)
  • Darek's Avatar
    Today, 00:32
    Darek replied to a thread Paq8pxd dict in Data Compression
    enwik8/9 update: 16'339'122 - enwik8 -s8 by Paq8pxd_v74_AVX2 15'993'409 - enwik8 -s15 by Paq8pxd_v74_AVX2 16'279'540 - enwik8 -x8 by Paq8pxd_v74_AVX2 15'928'916 - enwik8 -x15 by Paq8pxd_v74_AVX2 15'880'133 - enwik8.drt -x15 by Paq8pxd_v74_AVX2 125'752'479 - enwik9_1423 -x15 by Paq8pxd_v74_AVX2 - best overall score for paq8pxd serie 16'291'281 - enwik8 -s8 by Paq8pxd_v78_AVX2 15'941'450 - enwik8 -s15 by Paq8pxd_v78_AVX2 16'231'687 - enwik8 -x8 by Paq8pxd_v78_AVX2 15'877'659 - enwik8 -x15 by Paq8pxd_v78_AVX2 15'852'312 - enwik8.drt -x15 by Paq8pxd_v78_AVX2 125'942'438 - enwik9 -x15 by Paq8pxd_v78_AVX2 - plain enwik9 file tested by Kaitz 125'797'519 - enwik9_1423 -x15 by Paq8pxd_v78_AVX2 16'272'537 - enwik8 -s8 by Paq8pxd_v79_AVX2 15'925'621 - enwik8 -s15 by Paq8pxd_v79_AVX2 - tested by Sportman 16'214'034 - enwik8 -x8 by Paq8pxd_v79_AVX2 15'862'122 - enwik8 -x15 by Paq8pxd_v79_AVX2 - tested by Sportman - best score w/o DRT preprocessor for paq8pxd serie 15'843'925 - enwik8.drt -x15 by Paq8pxd_v79_AVX2 - best overall score for paq8pxd serie 125'67x'xxx - (estimated) - enwik9_1423 -x15 by Paq8pxd_v79_AVX2 - we'll see...
    763 replies | 288858 view(s)
  • schnaader's Avatar
    Today, 00:25
    I don't understand this "count" step. What do I have to count? If I count zeros and ones for example, I get 2 (zeros) and 10 (ones), not 3 and 17. Also, first you wrote 3055 => 1041 => 1007 => 17, the second one looks like 3055 => 113 => 15, which one is correct? Last question: Why can't 17 and 19 be reduced further?
    37 replies | 1550 view(s)
  • uhost's Avatar
    Yesterday, 23:08
    After 3 conversion 3055 = >1041=>1007=>17,3053=>1043=>1005=>19 this method can provide decimal free conversion 3055 =101111101111 count=11(3)+10001(17)=1110001 if you want reduce again 113=1110001 113=>15==1111
    37 replies | 1550 view(s)
  • uhost's Avatar
    Yesterday, 22:41
    :)
    37 replies | 1550 view(s)
  • Shelwien's Avatar
    Yesterday, 22:19
    Please read my previous post. In short, yes I implemented various versions of AC tail cutting, it saves up to 8 or so bytes per file. And no, unless you work with small random-access blocks, its usually not worth the trouble.
    172 replies | 8356 view(s)
  • schnaader's Avatar
    Yesterday, 22:05
    Have you tried compressing the output files of your program again? Perhaps you can make them 30% smaller, too.
    8 replies | 447 view(s)
  • schnaader's Avatar
    Yesterday, 21:52
    Out of curiosity, what are the results of reducing 3055 and 3053?
    37 replies | 1550 view(s)
  • uhost's Avatar
    Yesterday, 21:07
    well, please complete your Dream in real world :_good2:. you have well knowledge about math. use it and catch it i will be finish my algorithm, i am generated some math equation for reduced huge value to small this main advantage is no decimal data this 5 step can reduce 3054 =>2
    37 replies | 1550 view(s)
  • suryakandau@yahoo.co.id's Avatar
    Yesterday, 18:19
    Are you sure about compression time ? Because it is forked from cmix v17 with reduced memory usage …..in cmix thread you said that cmix v18 take only 8.76 days for enwik9...is it caused by compiled source code ???:_think:
    15 replies | 1138 view(s)
  • Self_Recursive_Data's Avatar
    Yesterday, 18:18
    That doesn't answer my question. Arithmetic Coding produces a very long decimal number: 0.3483856347597978987972181615551535345534625637078345232143287399295393590997145 The tail end of this number can be cut back/ shaved off, because it is unneeded: 0.3483856347597978987972181615551535345534625637078345232143287399295393 But it only improves compression by a extremely small amount. Programming this may cost more than it saves! Is it worth it? How much does chopping the tail save? And how much does coding it cost? The answer I expect from you is either "No, it'd be stupid to implement it" or "yes, it actually it actually improves compression because..."
    172 replies | 8356 view(s)
  • Shelwien's Avatar
    Yesterday, 16:51
    Precomp is commonly used with disabled internal compression for exactly this reason - its integrated compression is not very good.
    8 replies | 447 view(s)
  • Fallon's Avatar
    Yesterday, 15:03
    Fallon replied to a thread WinRAR in Data Compression
    WinRAR - What's new in the latest version https://www.rarlab.com/download.htm
    179 replies | 125764 view(s)
  • Darek's Avatar
    Yesterday, 10:45
    Darek replied to a thread Paq8pxd dict in Data Compression
    Scores of 4 corpuses for paq8pxd_v79. Slightly improvement but for all corpuses.
    763 replies | 288858 view(s)
  • A_Better_Vice's Avatar
    Yesterday, 09:03
    Have created a C++ version and continued to tweak. Installed Linux as I expect my code to run faster in Linux as well. Is there any software out there that does compress precomp04.exe generated files ~30% smaller or more ? I have looked and have not found any. Please do share if anyone is familiar with anything else. I am not sure if at 51 I have another 40 years in my life to improve this immensely more.
    8 replies | 447 view(s)
  • User's Avatar
    Yesterday, 08:19
    User replied to a thread Paq8pxd dict in Data Compression
    Dark green - absolutely the best result light green - paq8pxd_v24x64, paq8pxd_v44, paq8pxd_v46_SPEED, paq8pxd_v53 - Good ratio speed/ size (Total)
    763 replies | 288858 view(s)
  • Darek's Avatar
    Yesterday, 00:37
    Darek replied to a thread Paq8pxd dict in Data Compression
    Thanks, but there are two green colors - light green and dark green (I called it khaki), what does mean light green?
    763 replies | 288858 view(s)
  • User's Avatar
    Yesterday, 00:03
    User replied to a thread Paq8pxd dict in Data Compression
    Green - good yellow - bad red - critical error https://encode.su/threads/342-paq8px?p=64151&viewfull=1#post64151 https://encode.su/threads/342-paq8px?p=63987&viewfull=1#post63987 --- paq8pxd_v24x64, paq8pxd_v46_SPEED, paq8pxd_v53 - Good ratio speed/ size
    763 replies | 288858 view(s)
  • suryakandau@yahoo.co.id's Avatar
    Yesterday, 00:01
    How much memory usage using -x15?
    763 replies | 288858 view(s)
  • Romul's Avatar
    29th March 2020, 21:15
    No. According to the equations should work. But how in practice is unknown. Everything is only in the form of equations, graphs and formulas. But, now I will try to write a program. And it will be seen how wrong I am. ))) ​ PS: I write through an online translator, so my text may not look very correct.
    37 replies | 1550 view(s)
  • uhost's Avatar
    29th March 2020, 21:06
    is your algorithm implemented practical level ?
    37 replies | 1550 view(s)
  • Romul's Avatar
    29th March 2020, 20:41
    Although the ratio F/F and tends to the golden ratio, with n->∞ . (Where F is the fibonacci numbers). But, my algorithm has some relationship with the golden ratio, not with the Fibonacci numbers. I will also try to implement my algorithm. So, the competition? Where are you from ? Besides that Earth is the third planet from the sun? )) PS: I write through an online translator, so my text may not look very correct.
    37 replies | 1550 view(s)
  • compgt's Avatar
    29th March 2020, 20:33
    compgt replied to a thread 2019-nCoV in The Off-Topic Lounge
    No country is winning the coronavirus war. The death toll is catastrophic. Especially if this coronavirus is designed for biological warfare. The virus designers must learn now the impact to humanity is severe, even to global economies, must not be repeated in the future. The spread of the virus is hard to control. The best we could do is to impose lockdowns across countries, and cities. Biological warfare is unacceptable. Gene-sequencing laboratories should not be re-creating these viruses, even if they think they can contain them.
    33 replies | 1904 view(s)
  • uhost's Avatar
    29th March 2020, 20:01
    you are Brilliant !:D So i say you Fibonacci numbers related to my trick
    37 replies | 1550 view(s)
  • Darek's Avatar
    29th March 2020, 19:18
    Darek replied to a thread Paq8pxd dict in Data Compression
    @User, at first - very good tests. Thanks. Questions - did you test the latest paq8px versions? v182, 183...? What the red color meaning for 7zip.dll? Why pa8pxd v75 isn't colored as the best score (khaki clolor, isn't it?) for 7zip.dll file? In general - what the different colors meaning?
    763 replies | 288858 view(s)
  • uhost's Avatar
    29th March 2020, 19:11
    when i would complete The theory part, i will send sample program for your advice and testing
    37 replies | 1550 view(s)
  • uhost's Avatar
    29th March 2020, 18:55
    hi, i am followed mixed mod with My own calculation i had Complete 50% but now still some problem. i am trying to fix it and i can found all positive values equivalent to another positive value like as -2 +2 <= when i finished my theorem i will explain
    37 replies | 1550 view(s)
  • User's Avatar
    29th March 2020, 18:38
    User replied to a thread Paq8pxd dict in Data Compression
    Good ratio speed/ size - paq8pxd_v44 Request, kaitz, fix errors in paq8pxd_v44 and create paq8pxd_v44_2 (AVX2 and SSE4)? --- low anchor - fp8 middle anchor - paq8pxd_v44 high anchor - paq8pxd_v77 (-79)
    763 replies | 288858 view(s)
  • suryakandau@yahoo.co.id's Avatar
    29th March 2020, 16:22
    @sportman could you test this version on your computer ? Thank you
    15 replies | 1138 view(s)
  • Sportman's Avatar
    29th March 2020, 15:05
    Sportman replied to a thread Paq8pxd dict in Data Compression
    Only enwik8 -x15: 15,862,122 bytes, 7,566.827 sec., paq8pxd v79 -x15
    763 replies | 288858 view(s)
  • Darek's Avatar
    29th March 2020, 14:17
    Darek replied to a thread Paq8pxd dict in Data Compression
    nice score = 0,1% of improvement! @Sportman => do you plan to test some other enwik8/enwik9 options with paq8pxd_v79? I'am asking to not to duble the work.
    763 replies | 288858 view(s)
  • Sportman's Avatar
    29th March 2020, 12:47
    Sportman replied to a thread Paq8pxd dict in Data Compression
    enwik8: 15,941,450 bytes, 5,899.784 sec., paq8pxd v78 -s15 15,925,621 bytes, 5,879.782 sec., paq8pxd v79 -s15
    763 replies | 288858 view(s)
  • Darek's Avatar
    29th March 2020, 12:11
    Darek replied to a thread Paq8pxd dict in Data Compression
    And here some enwik scores for the latest versions: 16'339'122 - enwik8 -s8 by Paq8pxd_v74_AVX2 15'993'409 - enwik8 -s15 by Paq8pxd_v74_AVX2 16'279'540 - enwik8 -x8 by Paq8pxd_v74_AVX2 15'928'916 - enwik8 -x15 by Paq8pxd_v74_AVX2 15'880'133 - enwik8.drt -x15 by Paq8pxd_v74_AVX2 125'752'479 - enwik9_1423 -x15 by Paq8pxd_v74_AVX2 - best overall score for paq8pxd serie 16'319'686 - enwik8 -s8 by Paq8pxd_v75_AVX2 15'976'838 - enwik8 -s15 by Paq8pxd_v75_AVX2 16'260'265 - enwik8 -x8 by Paq8pxd_v75_AVX2 15'912'509 - enwik8 -x15 by Paq8pxd_v75_AVX2 15'859'187 - enwik8.drt -x15 by Paq8pxd_v75_AVX2 125'761'484 - enwik9_1423 -x15 by Paq8pxd_v75_AVX2 16'314'392 - enwik8 -s8 by Paq8pxd_v76_AVX2 15'965'102 - enwik8 -s15 by Paq8pxd_v76_AVX2 16'253'017 - enwik8 -x8 by Paq8pxd_v76_AVX2 15'899'380 - enwik8 -x15 by Paq8pxd_v76_AVX2 15'856'800 - enwik8.drt -x15 by Paq8pxd_v76_AVX2 125'974'773 - enwik9_1423 -x15 by Paq8pxd_v76_AVX2 16'316'789 - enwik8 -s8 by Paq8pxd_v77_AVX2 15'967'512 - enwik8 -s15 by Paq8pxd_v77_AVX2 16'255'214 - enwik8 -x8 by Paq8pxd_v77_AVX2 15'901'484 - enwik8 -x15 by Paq8pxd_v77_AVX2 15'856'824 - enwik8.drt -x15 by Paq8pxd_v77_AVX2 125'998'738 - enwik9_1423 -x15 by Paq8pxd_v77_AVX2 16'291'281 - enwik8 -s8 by Paq8pxd_v78_AVX2 15'941'450 - enwik8 -s15 by Paq8pxd_v78_AVX2 16'231'687 - enwik8 -x8 by Paq8pxd_v78_AVX2 15'877'659 - enwik8 -x15 by Paq8pxd_v78_AVX2 - best score w/o DRT preprocessor for paq8pxd serie 15'852'312 - enwik8.drt -x15 by Paq8pxd_v78_AVX2 - best overall score for paq8pxd serie 125'797'519 - enwik9_1423 -x15 by Paq8pxd_v78_AVX2
    763 replies | 288858 view(s)
  • Darek's Avatar
    29th March 2020, 12:09
    Darek replied to a thread Paq8pxd dict in Data Compression
    Heer are scores of my testset for paq8pxd v79. Slightly worse score for K.WAD... ;(
    763 replies | 288858 view(s)
  • kaitz's Avatar
    29th March 2020, 01:44
    kaitz replied to a thread Paq8pxd dict in Data Compression
    Yes.
    763 replies | 288858 view(s)
  • Darek's Avatar
    29th March 2020, 01:14
    Darek replied to a thread Paq8pxd dict in Data Compression
    @Kaitz - did you change the buffer limit in paq8pxd v79 version?
    763 replies | 288858 view(s)
  • kaitz's Avatar
    29th March 2020, 00:23
    kaitz replied to a thread Paq8pxd dict in Data Compression
    paq8pxd_v79 - Change wordModel1 some html entities rollback - Some fixes ​enwik8 -s8 is about 18kb smaller then v78.
    763 replies | 288858 view(s)
  • RichSelian's Avatar
    28th March 2020, 20:41
    RichSelian replied to a thread 2019-nCoV in The Off-Topic Lounge
    i brought hundreds of face masks (rather cheap in china now, about $0.2 per one) and a lot of alcohol spray home from carrefour today. all goods are supplied as normal, no looting. i thinks china is winning the coronavirus war. no more native people get infected for almost a week. all new cases come from immigration.
    33 replies | 1904 view(s)
  • kaitz's Avatar
    28th March 2020, 12:07
    kaitz replied to a thread Paq8pxd dict in Data Compression
    True, i did not realize it myself. Again was some long time ago set as to reduce memory usage. Also made quick test. Only matchmodels active: 174008558 250mb (buf) 172560820 1gb (buf)
    763 replies | 288858 view(s)
  • suryakandau@yahoo.co.id's Avatar
    28th March 2020, 03:15
    it is fork from cmix v7 because from ltcb site the fastest version of cmix is cmix v7....the result of xml file for cmixHP3 is: 5345280 bytes -> 267754 bytes in 2038.38 s. cross entropy: 0.401 enwik9 on running
    15 replies | 1138 view(s)
  • Shelwien's Avatar
    28th March 2020, 02:27
    Shelwien replied to a thread Paq8pxd dict in Data Compression
    I found a suspicious thing: bufn.setsize(0x10000); if (level>=9) buf.setsize(0x10000000); //limit 256mb else buf.setsize(MEM()*8); Do I read it right and paq8pxd uses 256mb buffer for enwik9 here?
    763 replies | 288858 view(s)
  • Shelwien's Avatar
    28th March 2020, 02:18
    > But how many bits can this shaving the end save you really? ~128 bits at most, no? Well, if you're using a rangecoder with 128 bit precision, then it could be 128 bits I guess. Common rc overhead is like this: - pad at start to avoid extra condition on carry flush: 1 byte - decoder lookahead bytes flushed to avoid tail checks: 2-4 bytes depending on rc renorm threshold - EOF/filesize coding: 3-4 bytes But its also possible to find some use for the value in the final interval - filesize (can be used instead of EOF) or crc % range maybe. Its also possible to save any (variable) number of tail bytes, when its likely for the model to output all 0s or all 1s at the end (which happens for some data types, eg. executables tend to have long zero padding at the end). > Is coding it worth it? For slow coders (like paq or cmix or nncp) it makes sense to have entropy coder with lowest possible redundancy - its kinda dumb when you save 5 bytes by adding a whole new submodel, while your rangecoder keeps encoding a megabyte of zeroes to 400 bytes because of low precision: https://encode.su/threads/2515-mod_ppmd?p=51672&viewfull=1#post51672 But for speed-optimized coders it may be better to keep the overhead, because extra checks are not worth it. > After Arithmetic Coding is finished the complete file, > you end up between the final high and low bounds. > You can shave off a few bits at the end. Ex. 0.451-0.372 becomes 0.38. The volume of encoded information is usually not bit-aligned, so its usually impossible to completely avoid any redundancy. I have some low-overhead rangecoders here: https://encode.su/threads/3084-Simple-rangecoder-with-low-redundancy-on-short-strings?p=59643&viewfull=1#post59643
    172 replies | 8356 view(s)
  • Self_Recursive_Data's Avatar
    28th March 2020, 01:00
    Question. After Arithmetic Coding is finished the complete file, you end up between the final high and low bounds. You can shave off a few bits at the end. Ex. 0.451-0.372 becomes 0.38. But how many bits can this shaving the end save you really? ~128 bits at most, no? ​Is coding it worth it?
    172 replies | 8356 view(s)
  • Darek's Avatar
    28th March 2020, 00:30
    Darek replied to a thread Paq8pxd dict in Data Compression
    125'752'479 - enwik9_1423 -x15 by Paq8pxd_v74_AVX2 125'797'519 - enwik9_1423 -x15 by Paq8pxd_v78_AVX2 - slightly worse than paqpxd v74
    763 replies | 288858 view(s)
  • Shelwien's Avatar
    27th March 2020, 03:57
    Shelwien replied to a thread cmixHP in Data Compression
    here, gcc 8.2. Seems like a patch of cmix_v7.
    15 replies | 1138 view(s)
  • suryakandau@yahoo.co.id's Avatar
    27th March 2020, 03:46
    cmixHP2 in this version i don't use lstm model.. the result is worse but for enwik9 benchmark it needs <10 gb RAM and the time is <= 100 hours 1000000000 bytes -> 149086410 bytes in 321779.79 s. this is the source and the binary compiled with dev c++. maybe someone could compiled it using mingw please ? thank you
    15 replies | 1138 view(s)
  • Darek's Avatar
    26th March 2020, 22:28
    Darek replied to a thread Paq8pxd dict in Data Compression
    Other enwik8 scores: 16'316'789 - enwik8 -s8 by Paq8pxd_v77_AVX2 15'967'512 - enwik8 -s15 by Paq8pxd_v77_AVX2 16'255'214 - enwik8 -x8 by Paq8pxd_v77_AVX2 15'901'484 - enwik8 -x15 by Paq8pxd_v77_AVX2 15'856'824 - enwik8.drt -x15 by Paq8pxd_v77_AVX2 16'291'281 - enwik8 -s8 by Paq8pxd_v78_AVX2 15'941'450 - enwik8 -s15 by Paq8pxd_v78_AVX2 16'231'687 - enwik8 -x8 by Paq8pxd_v78_AVX2 15'877'659 - enwik8 -x15 by Paq8pxd_v78_AVX2 15'852'312 - enwik8.drt -x15 by Paq8pxd_v78_AVX2 - drt got smaller improvement than pure file however it still provides to best score ever for paq8pxd series! enwik9 estimate = 125'802'xxx - very close to paq8pxd v74!
    763 replies | 288858 view(s)
  • Romul's Avatar
    26th March 2020, 15:13
    Is your algorithm somehow related to the golden ratio? I have an algorithm that can theoretically compress any input stream. And this algorithm has some relation with the so-called god number (1,618...). https://en.wikipedia.org/wiki/Golden_ratio But, this requires verification. So far, everything is only in the form of equations. I got these equations about five years ago, and tried to better understand their properties. PS. I write through an online translator, so my text may not look very correct.
    37 replies | 1550 view(s)
  • lz77's Avatar
    26th March 2020, 13:20
    blzpack seems only LZ-type compressor (at least I see in enwik8.blz literals like 'mediawiki xmlns=' and '"http://www.'). blzpack -2 -b1024m enwik8 gives ~ the same ratio as my program (while this is a prototype program for debugging and improving my algorithm), but my program is much faster than blzpack -1... I want to get time/ratio similar to zstd v1.4.4 for win64: timer.exe zstd.exe -1 -f --no-progress enwik8 Both ratio & compression time will be the same but decompression time at this time 15% worse. :) I want to find a buyer(s) for my algorithms/sources. Maybe I will sell it as shareware...
    8 replies | 531 view(s)
  • Darek's Avatar
    26th March 2020, 12:04
    Darek replied to a thread Paq8pxd dict in Data Compression
    First enwik scores: 16'319'686 - enwik8 -s8 by Paq8pxd_v75_AVX2 16'314'392 - enwik8 -s8 by Paq8pxd_v76_AVX2 = -6'300 bytes 16'316'789 - enwik8 -s8 by Paq8pxd_v77_AVX2 = +2'400 bytes 16'291'281 - enwik8 -s8 by Paq8pxd_v78_AVX2 = -25'500 bytes -> good improvement!
    763 replies | 288858 view(s)
  • Jyrki Alakuijala's Avatar
    26th March 2020, 05:55
    I tried reaching out to his relatives. I didn't get a response back.
    305 replies | 315907 view(s)
  • Darek's Avatar
    26th March 2020, 02:19
    Darek replied to a thread Paq8pxd dict in Data Compression
    paq8pxd_v78 scores for 4 corpuses => another version with all 4 records for paq8pxd serie!
    763 replies | 288858 view(s)
  • birdie's Avatar
    26th March 2020, 01:10
    Any updates?
    305 replies | 315907 view(s)
  • Jyrki Alakuijala's Avatar
    25th March 2020, 21:09
    Single-threaded brunsli is 1 % weaker and 2.5x faster. Brunsli covers more use cases in the exotic jpeg land. Brunsli is single-threaded, lepton multi-threaded. Brunsli variation in jpeg xl may allow multi-threaded coding in 256x256 tiles.
    34 replies | 9856 view(s)
  • Shelwien's Avatar
    25th March 2020, 17:41
    You can see an example here: https://encode.su/threads/3128-brunsli-jpeg-recompressor?p=60448&viewfull=1#post60448 Lepton sometimes has faster decoding (due to MT) and better compression (not at the same time as MT).
    34 replies | 9856 view(s)
  • Jyrki Alakuijala's Avatar
    25th March 2020, 17:35
    gipfeli does something that is in between of Huffman and not doing it. It might qualify to your requirements (or not)...
    8 replies | 531 view(s)
  • Darek's Avatar
    25th March 2020, 15:14
    Darek replied to a thread Paq8pxd dict in Data Compression
    paq8pxd_v78 scores on my testset. In general no big changes. Some improvements for textual files. Some loses for bigger files.
    763 replies | 288858 view(s)
  • CompressMaster's Avatar
    25th March 2020, 15:14
    Here.
    8 replies | 531 view(s)
  • CompressMaster's Avatar
    25th March 2020, 15:11
    Here.
    8 replies | 531 view(s)
  • Darek's Avatar
    25th March 2020, 15:08
    Darek replied to a thread Paq8pxd dict in Data Compression
    enwik8/9 scores for paq8pxd_v76: 15'928'916 - enwik8 -x15 by Paq8pxd_v74_AVX2 125'752'479 - enwik9_1423 -x15 by Paq8pxd_v74_AVX2 15'912'509 - enwik8 -x15 by Paq8pxd_v75_AVX2 125'761'484 - enwik9_1423 -x15 by Paq8pxd_v75_AVX2 15'899'380 - enwik8 -x15 by Paq8pxd_v76_AVX2 125'974'773 - enwik9_1423 -x15 by Paq8pxd_v76_AVX2 - hmmm, there is an 0,17% loss to v75 version, 0.18% to v74 version. The v74 is still the best! paq8pxd v77 and v78 tests ongoing.
    763 replies | 288858 view(s)
  • well's Avatar
    25th March 2020, 11:11
    i have been testing phda9 $./phda9 C9 enwik9 out is it right? C9 - is it right option for enwik9?as you can see behind phda9 is not universal compressor, so i ask for right using instructions! $./phda9 C enwik9 out 91% Segmentation fault $ so C9 - is strictly for enwik9?
    87 replies | 30786 view(s)
More Activity