Activity Stream

Filter
Sort By Time Show
Recent Recent Popular Popular Anytime Anytime Last 24 Hours Last 24 Hours Last 7 Days Last 7 Days Last 30 Days Last 30 Days All All Photos Photos Forum Forums
  • Bulat Ziganshin's Avatar
    Today, 13:27
    NUVIA Phoenix Targets +40-50% ST Performance Over Zen 2 for Only 33% the Power When tested by GeekBench 5, at every point, ARM’s results are more power efficient/higher performant than anything available on x86, even though at the high end Apple and Intel are almost equal on performance (for 4x the power on Intel). Note that Intel cores run up to 5 Ghz, while Apple cores run only up to 3 GHz.
    14 replies | 889 view(s)
  • Darek's Avatar
    Today, 10:34
    Darek replied to a thread paq8px in Data Compression
    @mpais - yes - there are the same scores, maybe verified. Yesterday I couldn't find my previous post with scores - now I see it was already on place. "Oficial Tester" - hmmm, sounds proud :). Maybe just "Tester". According to text option - yes, I've started to test paq8px v191a and v191 with -t option reverse.
    2064 replies | 559450 view(s)
  • Jarek's Avatar
    Today, 10:24
    There were some discussions in which, if I properly remember, there was nearly no performance improvement ... but I cannot find it. If you need details maybe write to Jean-Marc Valin or Timothy Terriberry. I haven't seen them here, but Daala mailing list is still active: http://lists.xiph.org/mailman/listinfo/daala I am returning to PVQ from time to time and have mixed feelings. For me the main advantage is smaller MSE of pyramidal lattice than of square lattice (bottom left below), but practically exploiting it is quite difficult. Performing rate-distortion optimization surprisingly leads to uniform quantization (e.g. section IV in https://arxiv.org/pdf/2007.12055 ), so PVQ sounds a good choice ... however, it needs to be combined with proper probability distribution of the lattice, which e.g. for 8x8 DCT is quite complex.
    44 replies | 4486 view(s)
  • ivan2k2's Avatar
    Today, 04:49
    Interesting. It looks like a mix of zram, garbage collector, defragmentation and delta. Want to see real numbers...
    1 replies | 98 view(s)
  • mpais's Avatar
    Today, 00:57
    mpais replied to a thread paq8px in Data Compression
    @moisesmcardona Thanks, that's good to know. If native builds are that much faster, there's still a lot to be gained. And then there's AVX512 and bfloat16.. @Darek Those are the same scores you posted, are they not? Anyways, you can just skip to v191a, it's just that your old "best" settings don't apply on v191. For instance, I got 191.761 bytes with "-9l" on I.EXE with v191, I'm guessing you used text pre-training on that file before v191 since it gave better results. And as always, thank you for testing, we should petition the mods to get you a special "Official Tester" badge on your profile.
    2064 replies | 559450 view(s)
  • Darek's Avatar
    Today, 00:15
    Darek replied to a thread paq8px in Data Compression
    Ok, there are scores for pure pax8px_v191 - super scores for textual files (paq beats best cmix veriosns!!!!) but very bad scores for the biggest files -similar to non LSTM scores.
    2064 replies | 559450 view(s)
  • JamesWasil's Avatar
    Yesterday, 23:44
    Compression method/approach for hardware memory systems: http://news.mit.edhttp://news.mit.edu/2019/hardware-data-compression-0416u/2019/hardware-data-compression-0416
    1 replies | 98 view(s)
  • CompressMaster's Avatar
    Yesterday, 23:14
    I have searched for some (built-in, Footej Camera2, ManualCam, SnapCamera), but I´m still unsatisfied with results. Sure, benefits of durable phones aren´t cameras, but still, what´s the best possible app for highest image quality? I have BlackView BV-5800.
    0 replies | 25 view(s)
  • fabiorug's Avatar
    Yesterday, 21:16
    I'm asking if they find Daala at that setting any good, or is only an oversharpener that inflates file size adding fake details and not preserving the naturalness of image. If this type of filtering would be good to add in a codec that isn't daala (even not jpeg) like with a toggle for example sharpness. So people who likes this filtering need to export png and recompress? you are saying that Jarek? as there is zero chance it would get adopted in image codecs. Talking about what you said, I know AV1 discarded PVQ for computational complexity, as it was complex enough. But I don't want to talk about video. Even if i think you're right, Daala is video, is not simply PVQ filtering for image, is a complex video codec like others even if you add an image. Anyway in the demo you can submit file and there is information if you are interested. It's an old demo I know.
    44 replies | 4486 view(s)
  • Jarek's Avatar
    Yesterday, 21:03
    PVQ is in Opus, from Daala was considered for AV1 but didn't get there ... I haven't seen it in connection with JPEG XL (?)
    44 replies | 4486 view(s)
  • fabiorug's Avatar
    Yesterday, 20:56
    http://ba.rr-dav.id.au/daala-demo/ https://imgur.com/a/VbsfEiO Is there a way to use PVQ in Jpeg XL? That's one setting I find has better quality than JpegXL at lower quantization. As far as I know in the description they wrote it uses only 8x8 dct and Lapping, so the structure is very different to jpeg xl that is a simple block based codec. So maybe jpeg xl with PVQ and similar settings would be ruin or would require a total rewriting of the software. But the thing is jpeg xl wants the image to be very similar to the original The main goal of JPEG XL is to achieve a minimum size with an almost imperceptible difference. Source: https://cloud.tencent.com/developer/article/1667847 PVQ I guess is still an experimental technology and especially with the setting I mentioned even if give the illusion the image is cleaner with PVQ the file sizes get bigger, but the thing is the people who want to use PVQ should export in Png, then jpeg xl, then png. And is time consuming and it doesn't give the best results, as it would be integrated in a future codec with similar quality. And to me the quality at the setting I posted on imgur for the daala demo by barrbrain is good enough. I don't know if is the case for JPEG XL developers and if they find efficient or interesting to integrate similar type of filtering in their codec, so I'm asking in this forum.
    44 replies | 4486 view(s)
  • moisesmcardona's Avatar
    Yesterday, 16:38
    moisesmcardona replied to a thread paq8px in Data Compression
    Hi @mpais, Your suggestion worked. The files now have the same size and the checksums matched between the Intel and AMD CPU. Same goes for the native build. The size and checksums matched between the CPUs, but the files are incompatible between the native and non-native build. The native build was about 20 seconds faster on the AMD CPU and about 42 seconds faster on the Intel CPU. tested with -1l -simd avx2.
    2064 replies | 559450 view(s)
  • mpais's Avatar
    Yesterday, 16:29
    mpais replied to a thread paq8px in Data Compression
    Changes: - Repurposed option switch "r" to handle loading of pre-trained LSTM models, decoupling it from other switches The next model is still going to take a few days to finish training, and since I was already planning on doing this, I've removed retraining (too little benefit) and repurposed the "r" swtich to handle the loading of LSTM models. For now that just means using the english model. @Darek So for this interim version, instead of "-9lta" for your text files, you'd use "-9rta". "t" will just use the old text pre-training, "r" will complement it by loading the LSTM model (and no need for "l" since "r" implies it).
    2064 replies | 559450 view(s)
  • mpais's Avatar
    Yesterday, 15:40
    mpais replied to a thread paq8px in Data Compression
    @Darek: I see, you used text pre-training even on non-textual files to squeeze a bit more compression. But now doing so means the LSTM is right from the start highly trained to predict english text, and it will take a very long time to re-adapt. You'll get much lower losses by skipping pre-training on those files. In the next version I'll separate "regular" text pre-training from LSTM model loading, so you can use them separately, for files where regular pre-training helped a bit, but LSTM model loading hurts a lot.
    2064 replies | 559450 view(s)
  • Darek's Avatar
    Yesterday, 15:27
    Darek replied to a thread paq8px in Data Compression
    Scores for my tetsset on paq8px v191 - there are some differrent effescts: a) for rextual files and 24bpp files the scores are awesome! Socres are better than cmix v18 scores! All textual files are the best number at all! b) for some biger files (G,EXE, H,EXE, I.EXE, K.WAD, L.PAK) scores are highly worse - compared to non LSTM scores...
    2064 replies | 559450 view(s)
  • Bulat Ziganshin's Avatar
    Yesterday, 15:22
    I need any dynamic multi-symbol arithmetic or ANS codec for a quick experiment. Can you recommend any ready-to-use C/C++ sources?
    0 replies | 85 view(s)
  • Bulat Ziganshin's Avatar
    Yesterday, 13:55
    The list of delta codecs: xdelta3 (Apache license), 2016 Google open-vcdiff (Apache license), 2008? bsdiff (BSD license), 2015 ddelta (BSD license), 2017 zdelta (BSD license), 2004 Are you know any other delta engines? Which ones are more efficient from your experience?
    0 replies | 115 view(s)
  • Jarek's Avatar
    Yesterday, 06:18
    I have just updated https://arxiv.org/pdf/2007.12055 by exploiting statistical dependencies inside 8x8 DCT block - turns out it allows for huge savings: ~0.5 bits/value (per pixel for grayscale, up to x3 for RGB). So DCT decorrelates well - additional linear(!) predictions between its coefficients make nearly no sense. However, there are also different statistical dependencies between them, like width dependence (between variances): the higher absolute value of one coefficients, the larger width of Laplace should we use for its neighbors. It it is shown in the left plots below: the higher already decoded |DCT_12|, the larger width of Laplace should we use for DCT_13 - exploiting of which gives ~0.4 bit/value savings. Bottom right contains weighs for finding optimal width from already decoded in zigazg order - it becomes a bit costly, but inexpensive approximations should be close. If e.g. JPEG recompressors don't use it, expect finally breaking their 20-22% bound by a lot ...
    1 replies | 437 view(s)
  • MegaByte's Avatar
    Yesterday, 04:31
    I'm not sure why they didn't test the JPEG XT Part 8 residual coder since that's what their design was based on (though I don't know if it would be appropriate for the residuals from JPEG XS). The poor overall results might be mostly from the opposing design goal of JPEG XS (low latency, low complexity), but the less it was compressed by the JPEG XS layer, the more the other codecs were able to compress.
    1 replies | 155 view(s)
  • mpais's Avatar
    13th August 2020, 18:11
    mpais replied to a thread paq8px in Data Compression
    I don't have an AMD machine available to test, but I suspect it may be the AVX2 RSQRT instruction used in the Adam optimizer. @moisesmcardona: Could you try commenting out the line #define USE_RSQRT in Adam.hpp, recompiling and trying it out?
    2064 replies | 559450 view(s)
  • moisesmcardona's Avatar
    13th August 2020, 17:50
    moisesmcardona replied to a thread paq8px in Data Compression
    Yes, but also look that the AMD and Intel CPU produced different result, using the same executable. Intel AVX2 LSTM (non-native build): paq8px_v190.exe -1l -simd avx2 "C:\temp\test rav1e 0.1.0 993950d q175-s0.log" "C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm avx2 intel.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Creating archive C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm avx2 intel.log in single file mode... Filename: C:\temp\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18391 Time 293.94 sec, used 572 MB (600230782 bytes) of memory AMD AVX2 LSTM (non-native build): paq8px_v190.exe -1l -simd avx2 "H:\test rav1e 0.1.0 993950d q175-s0.log" "H:\test rav1e 0.1.0 993950d q175-s0 lstm avx2.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Creating archive H:\test rav1e 0.1.0 993950d q175-s0 lstm avx2.log in single file mode... Filename: H:\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18378 Time 151.14 sec, used 572 MB (600230729 bytes) of memory 13 bytes of difference. Same flags, same SIMD, same executable, different CPU, different results.
    2064 replies | 559450 view(s)
  • Gotty's Avatar
    13th August 2020, 17:06
    Gotty replied to a thread paq8px in Data Compression
    As far as I know AVX2 should produce the same results regardless of the CPU. As I see on your results, just a recompilation produces different results, not just targeting different CPUs. That's no good. I'll look into it.
    2064 replies | 559450 view(s)
  • Stefan Atev's Avatar
    13th August 2020, 17:06
    Hi, I just stumbled across this in my arxiv feed (https://arxiv.org/abs/2008.04558) and was wondering about the approach. From what I understand, the idea is to use the visually lossless mode of JPEG XS to produce a base image, and then use another JPEG variant to encode the residual image (suitably rescaled). My question: Does it make sense to use an algorithm designed for images on a residual image? It seems like the residual will not match the assumptions of most algorithms, indeed, it is almost uniquely designed not to match any assumptions about natural or computer-generated images. Am I missing something obvious? ​
    1 replies | 155 view(s)
  • moisesmcardona's Avatar
    13th August 2020, 15:32
    moisesmcardona replied to a thread paq8px in Data Compression
    It's normal SSE2 and AVX2 produce different result. But for AVX2, it's different on different CPUs, whereas for SSE2, it's the same result regardless of the CPU.
    2064 replies | 559450 view(s)
  • suryakandau@yahoo.co.id's Avatar
    13th August 2020, 15:14
    ​it is not normal.
    2064 replies | 559450 view(s)
  • moisesmcardona's Avatar
    13th August 2020, 15:05
    moisesmcardona replied to a thread paq8px in Data Compression
    @Gotty Thanks for the explanation. This makes sense, given that the system sometimes gets laggy even when the CPU is not at 100%. Usually, I'd just pause the process if I'm doing something important (BOINC offers a suspend function that suspends every task and resumes them only when told). As far as the LSTM question, is it normal that AVX2 produces different results on different CPUs? I was planning on using that since it saved around 2-5MB, even if it took more days, but given the differences between the machine, I'm not sure if it would be a good approach, given the different results. The issue didn't happen with SSE2 (See the output size from the above post).
    2064 replies | 559450 view(s)
  • compgt's Avatar
    13th August 2020, 14:35
    There's no need to state that, Dresdenboy. I've hinted that already in another thread: I suppose you're not one of them, paid to attack me? They made huge efforts to put me in a mental institution in the 1980s, trying to dispose of me quickly, but i was a known child genius or honor student that i was supported and protected by teachers in grade school and high school. So they didn't succeed; i continued my schooling, in between years this grade school time managed to sing and record the official Bread and America songs etc. you're now used to listen to. I was my own psychiatrist. I was very good psychiatrist assuaging and appeasing people to forget the Cold War. I influenced the field of modern psychiatry too because i was wizard. I designed pharmaceutical drugs and medicine, owning pharma tech companies but taken from me too. They succeeded in early 2010s to set me up for the mental institution. They set me up by "threatening" me. If i retaliate they will tag me as "violent" fit for a mental institution. These people are so hardcore corrupt that they will really live near me, to disturb me, to partake of my Hollywood billion$ they are corrupting. I recently remembered he was messing with me in college too, in the 1990s. He was a rapist maniac and hardcore killer up to 1980s, i now recall. That's what corrupt powerful politicians do to their enemies or those in their way, sending them to jail or mental institution. These are the corrupt people we should be afraid of. So now, Dresdenboy, i actually see my psychiatrist regularly just for the meds. Just like any other rock star, huh? ;) If i may add, how secure are today's singers and bands? P. S. I am now living peacefully here in the Philippines. It would be more peaceful and serene and i would be glad if i get my Hollywood million$ first, in the meantime, then my tech billion$.
    25 replies | 2579 view(s)
  • bwt's Avatar
    13th August 2020, 09:49
    Maybe 50% for the author and 50% for the participants that winning the competition or 70% for the author and 30% for the participant. It stiil encourage more participant to enter gdcc I think.
    71 replies | 5118 view(s)
  • compgt's Avatar
    13th August 2020, 09:46
    @Dresdenboy, i am telling the truth. I planned the timeline of data compression and encryption history, to a moderate and steady pace of development. It's classified top secret, now being declassified by me. I was the one who classified it in the first place anyway, since i owned the Hollywood and tech companies. I was on top anyway. I can live with my name not in songs as composer or co-composer, for example, because i own the Hollywood media (music and movie) companies anyway. Do you understand? But they took advantage of this fact, that my name is not there in the music bands albums and songs (and movies), they ousted me in the corporate. My voice is in the "official" songs of the bands Bread and America, and some songs of Nirvana, Scorpions, Queen and many other songs. We calculated and forecasted a billion$ revenue before for one high-selling Beatles music album, maybe with Bread music album/s too etc, i think. I made the Hollywood blockbuster movies Star Wars, Star Trek, Transformers, and the Marvel movies etc. to document how advanced we were during the Cold War. It was war, yet i was also this lovely boy singing to mellow everything. The same in tech, i was supposed to be co-founder of Yahoo, Google and Facebook (already owning Apple, Microsoft, IBM, Intel, AMD), though i chose and made popular Jerry Yang, Sergey Brin and Larry Page, and Mark Zuckerberg. It's the power and popularity of American globalism i understood early on but it was me, a Filipino Asian, orchestrating everything during the 1970s to the 80s, this timetable of science and technologies. Do you understand? @SCOTUS should just order these Hollywood and tech companies to pay me. It's that simple.
    25 replies | 2579 view(s)
  • Dresdenboy's Avatar
    13th August 2020, 07:27
    Dear Gerald, once I landed on your blog because of RLLZ and was quite a bit confused by your other postings there. A new check already brought up some dangerous stuff, like: (Source: https://grtamayoblog.blogspot.com/2020/02/paq-compression-programs.html) I've seen claims like that before. And this is serious stuff. Be very careful with such claims. There are two possible truths behind this. One is highly unlikely (p ~ 1), while the other is quite the opposite (1-p). In both ways there is just one way to solve this and show all those people, who stole your ideas, that you are right: Ask a psychiatrist to attest you full mental health, setting p to 1, so that billionaires, companies and countries have to accept this. Otherwise you'll continue to live a life full of such thoughts probably making you unhappy. Clear this mess up, find the truth!
    25 replies | 2579 view(s)
  • compgt's Avatar
    13th August 2020, 05:55
    I made modern Hollywood! I made the Hollywood blockbuster movies! The actors and actresses are just my people. I just made them. But they're popular more credible, they figured that out so they betrayed me and denied me. Same in tech. I pioneered modern computing! I'm here for the money, for my Hollywood and tech billion$ which they stole from me. I don't like lawsuits; i can't stand being interrogated. I just want to get paid my due billion$. I don't want to be the evil sorcerer next time to destroy the world to avenge myself!! I can't annihilate all those people who steal from me. Somehow i must stop. They must understand in their hearts that they owe me. Somethings gotta give. And it must be them, paying me. It's useless and senseless interacting with you here, @JamesWasil!!
    25 replies | 2579 view(s)
  • JamesWasil's Avatar
    13th August 2020, 04:59
    Well maybe you need to get used to it, because you've only posted crap and spammed people with nonsense and your delusional rants without provocation, and quite frankly, I've had enough of your bs and how you turn every single thing others have made or done into "ah yes, I remember doing/making that during the Cold War in the 1960's, 70's, and 80's!" Enough. I'm not helpless, I know exactly what I'm saying, and I don't go around spamming people or writing bullcrap like you have. It makes the forum look bad, and it detracts from both the conversation and the progress of things when you do it. I tried to ignore it when it was contained to the off-topic threads, but then you go and put it all here, too. None of this crap was around in 2007-2017 until you got here. I didn't make an account until a decade later when I felt like chiming in here and there and just read the forum between then and now. But even with agreements and disagreements back then that members have had, NO ONE was forced to smile and nod at your delusional behavior as they are now. It's ridiculous. Yeah, I'm one of the "uncontrolled", because I don't need medication and I have enough basic respect for 99% of the members here not to do the crap you're doing whether I know them or not. If you're some other user on here and you think this is funny, you just need to stop. This is not what this forum was about, and it shouldn't be about that now, either. This is not Facebook or Twitter. You've mentioned things about compression here and there and that's great. I support that and appreciate it. But what is not great is how you interject your delusions about having made everything that other people have done and claim it as your own from here to Hollywood. It's demeaning and undermining to the intelligence of everyone else here and elsewhere when you do that, even if you have some sort of psychological problem that makes you think that is ok and acceptable to do it. It is not. You have 0 proof of your claims in the off-topic thread, and that remains a constant here as well. You just need to get help and stop disrespecting people here by saying you did everything they ask about or discuss, because it's only going to irritate people and make them angry and not come back, rather than fostering a decent discussion of data compression techniques, information theory, or anything else. This entire thread should probably be moved to the Off-Topic area to not clutter up the main thread since it's more about your fantasies than it is about the subject itself.
    25 replies | 2579 view(s)
  • compgt's Avatar
    13th August 2020, 04:51
    Original text, original implementations. GPL was new that time. And I have to make my programs different from others because they were intended to be sample code programs for The Data Compression Guide.
    25 replies | 2579 view(s)
  • compgt's Avatar
    13th August 2020, 04:29
    I am not used to this, people attacking me, my person, in online forums. I thought people are civil enough to be polite here. I have not attacked you, JamesWasil. You're helpless, you don't know what you're saying. You're one of those uncontrolled. Me, i was stating the truth. You're the one who started insulting my name! Your apology is also an insult, undermining my intellect. I was telling the truth, of how modern Hollywood and modern computing industry came to be, by me, with me. Hollywood and the tech industry owes me bigtime. Their mistake was to totally betray me, left me with nothing to own. They were overkill, blinded by the billion$, fueled by their greed and envy. They're corrupting my Hollywood billion$. I made modern Hollywood. I pioneered modern computing. That was classified top secret, now declassified by me.
    25 replies | 2579 view(s)
  • Gotty's Avatar
    13th August 2020, 03:40
    Gotty replied to a thread paq8px in Data Compression
    @moisesmcardona Based on your results I have the feeling that the cause is indeed the thread count + memory. 16 threads are too much. Paq8px does something like that during compression: reading from memory... waiting for result ... performing some calculations, ... reading from memory... waiting for result ... performing some calculations, ... A significant amount of time is spent for just... waiting... for RAM. Level -1 is the fastest. Does it do less operations? No. It "only" uses less memory. So why is it faster then? When using less memory a larger percent of it can fit in the caches so there is a higher chance that the desired location is cached. Imagine yourself being the RAM. What you do is working hard to fetch some data and immediately the next request comes - you don't have much time to rest. You are the slowest component (from the CPU+cache+RAM trio). Everybody waits for you ;-) So let's do multi-threading. How many operations can your RAM modules do at any moment? (It depends on how many RAM modules you have and if they run in dual channel for example.) So how many threads can access your RAM simultaneously/concurrently? If you are running 16 threads but you have like 4 RAM modules in dual channel and let's suppose the rare (optimal) case when all threads want to access a very different memory location, still maximum 8 will be lucky, and 8 will wait for their chance to get data. Your maximum memory bandwidth with DDR4-2133 RAM is somewhere around 17 GiB/s. Latency plays mainly, but for the sake of simplicity let's ignore it. Let's suppose a paq8px thread will utilize it to 10%. That means (roughly speaking) that 10 such threads will utilize your memory fully and if you start a 11th one then these 11 threads will start to block each other. So 11 would run around the same speed as the 10 did. (Simplifying again.) All of them will wait for the same "slow" memory subsystem. It could be the case that your peak thread count is not 16 but 12 or 8 for paq8px. But let's not guess. Let's measure. What is the combined throughput for all the threads when we do compression simultaneously with 1..32 threads? A little experiment (Xeon E2286M, 4x16GB DDR4-2666, Dual channel, 19-19-19-43): paq8px -8 obj1 1 thread: 3.8s 2 threads: 4.1s x2 3 threads: 4.3s x3 4 threads: 4.5s x4 5 threads: 4.9s x5 6 threads: 5.4s x6 7 threads: 6.0s x7 8 threads: 6.6s x8 9 threads: 7.1s x9 10 threads: 8.1s x10 ... 16 threads: 14.3s x16 ... 32 threads: 26.5s x32 As you can see the more threads are running the slower each thread will be. A metric for speed would be: how many obj1 files can we compress per second in each case? (i.e. divide the number of threads by the elapsed seconds). Since obj1 is a very small file, memory will not yet be exhausted at all when compression suddenly finishes. The best thread count is 9 in this case (with 9 threads it can crunch 9/7.1 = 1.26 obj1 files per second - that is the maximum). paq8px -8 obj2 1 thread: 40.3s 2 threads: 44.3s x2 3 threads: 48.4s x3 4 threads: 52.7s x4 5 threads: 58.6s x5 6 threads: 65.5s x6 7 threads: 73.2s x7 8 threads: 83.0s x8 9 threads: 93.3s x9 10 threads: 105.4s x10 ... 16 threads: 192.9s x16 ... 32 threads: 355.9s x32 Obj2 is somewhat larger, compression takes more time, memory is filled up more (but still not be exhausted). The best thread count is similar: 10 (with 10 threads it can crunch 10/105.4 = 0.09487 obj2 files per second. When I use 16 threads for example its "speed" is just 16/192.9 = 0.08294 obj2 files per second). So on this system 9-10 is the optimal thread count for paq8px. Using 16 threads produces a 13% slower throughput than running with 10 threads.
    2064 replies | 559450 view(s)
  • LucaBiondi's Avatar
    13th August 2020, 00:16
    LucaBiondi replied to a thread paq8px in Data Compression
    Ok thank you Mpais. It was just to know your opinion Luca
    2064 replies | 559450 view(s)
  • JamesWasil's Avatar
    12th August 2020, 22:08
    I agree with you and bwt on that. It's great to have a prize, but if the people who win it will pay their programmers (or programmer) more than they paid you to create it in the first place...that's not going to encourage people and might be considered taking advantage of them instead. I'm not sure if it's too late to change it, but perhaps they could say that in addition to the prize, the creator with the winning submission gets a guaranteed percentage of revenue generated from the use of their algorithm, even if only up to a certain amount. Even with the bar raised up to 3% improvement over 0.25%, there would still be stronger encouragement for competition and developers if there were residual incentives to any amount. The large corporation(s) who stand to benefit from the results will no doubt make use of it to profit from it well, and it shouldn't be unreasonable for the winner of the contest (or if there are several winners, residuals / x submissions per category) to be able to enjoy some of those same benefits. That might be the only way to really guarantee that it will be worthwhile to all competitors, and it doesn't require the companies and hosts for this contest to put up any additional money out of pocket beyond the prize amount, since the additional percentages will be provided by the amount saved over profits for storage and data transmission later on.
    71 replies | 5118 view(s)
  • JamesWasil's Avatar
    12th August 2020, 21:04
    Nope. You're an idiot to not know anything but how to make false claims, you need mental help and a one-armed straight-jacket. You're a piss poor troll. Stop polluting the compression forums with your bullshit. It doesn't amount to anything and only takes away from the value of what people come here for. And yeah buddy, I'm just going to say it: you're absolutely insane. You invented nothing. Hollywood owes you nothing, because you didn't do anything. You just make bogus claims all over the place and need medication. It's not my fault or anyone else's here that you're an idiot and the world failed you, and we're really tired of hearing about it on every single thread you interject and spam message you've sent to people unsolicited. There were no insults until now. If you want to make insults, I'll give you some tomato. I was being nice to you but no more. That's the consequences of your ad-hominem after I apologized for getting your name wrong. I take back my apology. There are no consequences for telling it like it is, so there you go.
    25 replies | 2579 view(s)
  • JamesWasil's Avatar
    12th August 2020, 20:58
    Yeah, he copied things from other sites. Not original. He has big problems.
    25 replies | 2579 view(s)
  • Alexander Rhatushnyak's Avatar
    12th August 2020, 20:23
    No, it's a publicly available compressor, a pre-competition baseline.
    71 replies | 5118 view(s)
  • compgt's Avatar
    12th August 2020, 20:01
    You're an idiot to not know the consequences of your insults. Karma knows the truth.
    25 replies | 2579 view(s)
  • algorithm's Avatar
    12th August 2020, 19:26
    Filter and then LZ is a poor choice for compression of photographic images(maybe they gain some speed, but compression ratio is hurt). There are simply no patterns in photographic images. You try to compress noisy data. Also QLIC2 is very good. Either Alexander uses very good filters before compressing with FSE, or some kind of context modeling is used. BTW Is it considered a submission?
    71 replies | 5118 view(s)
  • mpais's Avatar
    12th August 2020, 17:07
    mpais replied to a thread paq8px in Data Compression
    As usual your options are better than mine and you actually managed to take #2 spot. Are you going to submit it? Also, don't forget that, if cmix gets updated with these improvements, it will again significantly increase the distance to #1 spot (well, with a huge help from precomp, now with Preflate integrated).
    2064 replies | 559450 view(s)
  • mpais's Avatar
    12th August 2020, 17:05
    mpais replied to a thread paq8px in Data Compression
    No, it would be a lot of work for not much gain, so, for me personally, it's low priority. Detection itself, if memory serves from when working on precomp, is finicky, since we'd need to rely on just a few bits to detect possible frames, and then need a way to validate them, which incurs decompressing them. Otherwise we'd be in same situation as now, where we don't check deflate or gif detections for proper validity, and then get transform fails. Honestly, the whole pre-processing stage in paq8 is a mess, one that I'm not looking forward to meddling in.
    2064 replies | 559450 view(s)
  • mpais's Avatar
    12th August 2020, 17:02
    mpais replied to a thread paq8px in Data Compression
    Changes: - Pre-training now available for the LSTM, with a heavily-quantized model trained on english texts - LSTM prediction is now promoted to the second layer of the paq mixer - Activations functions are now available in AVX2
    2064 replies | 559450 view(s)
  • moisesmcardona's Avatar
    12th August 2020, 16:15
    moisesmcardona replied to a thread paq8px in Data Compression
    Hmm, so now the AMD machine was faster! Of course, I tested now v190 while the previous results were from v189. Maybe the slowdown had to do with the amount of simultaneous tasks being run? Keep in mind I only had 50% of processes running (16 threads out of 32). However, now I noticed that the LSTM is producing different results on AMD and Intel (for the AVX2 SIMD). I'm not sure if this is intended or if it may be a bug. ------------------------------ AMD machine: SSE2 no LSTM: paq8px_v190.exe -1 -simd sse2 "H:\test rav1e 0.1.0 993950d q175-s0.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Highest SIMD vectorization support on this system: AVX2. Using SSE2 neural network and hashtable functions. Creating archive test rav1e 0.1.0 993950d q175-s0.log.paq8px190 in single file mode... Filename: H:\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18415 Time 45.42 sec, used 552 MB (579223768 bytes) of memory AMD SSE2 LSTM: paq8px_v190.exe -1l -simd sse2 "H:\test rav1e 0.1.0 993950d q175-s0.log" "H:\test rav1e 0.1.0 993950d q175-s0 lstm sse2.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Highest SIMD vectorization support on this system: AVX2. Using SSE2 neural network and hashtable functions. Creating archive H:\test rav1e 0.1.0 993950d q175-s0 lstm sse2.log in single file mode... Filename: H:\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18504 Time 302.81 sec, used 570 MB (598303097 bytes) of memory AMD AVX2 LSTM (non-native build): paq8px_v190.exe -1l -simd avx2 "H:\test rav1e 0.1.0 993950d q175-s0.log" "H:\test rav1e 0.1.0 993950d q175-s0 lstm avx2.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Creating archive H:\test rav1e 0.1.0 993950d q175-s0 lstm avx2.log in single file mode... Filename: H:\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18378 Time 151.14 sec, used 572 MB (600230729 bytes) of memory AMD AVX2 LSTM (native build) paq8px_v190_nativecpu.exe -1l -simd avx2 "H:\test rav1e 0.1.0 993950d q175-s0.log" "H:\test rav1e 0.1.0 993950d q175-s0 lstm avx2 native.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Creating archive H:\test rav1e 0.1.0 993950d q175-s0 lstm avx2 native.log in single file mode... Filename: H:\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18389 Time 128.81 sec, used 572 MB (600230750 bytes) of memory Intel SSE2 no LSTM: paq8px_v190.exe -1 -simd sse2 "C:\temp\test rav1e 0.1.0 993950d q175-s0.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Highest SIMD vectorization support on this system: AVX2. Using SSE2 neural network and hashtable functions. Creating archive test rav1e 0.1.0 993950d q175-s0.log.paq8px190 in single file mode... Filename: C:\temp\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18415 Time 72.22 sec, used 552 MB (579223788 bytes) of memory Intel SSE2 LSTM: paq8px_v190.exe -1l -simd sse2 "C:\temp\test rav1e 0.1.0 993950d q175-s0.log" "C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm sse2 intel.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Highest SIMD vectorization support on this system: AVX2. Using SSE2 neural network and hashtable functions. Creating archive C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm sse2 intel.log in single file mode... Filename: C:\temp\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18504 Time 556.58 sec, used 570 MB (598303150 bytes) of memory Intel AVX2 LSTM (non-native build): paq8px_v190.exe -1l -simd avx2 "C:\temp\test rav1e 0.1.0 993950d q175-s0.log" "C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm avx2 intel.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Creating archive C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm avx2 intel.log in single file mode... Filename: C:\temp\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18391 Time 293.94 sec, used 572 MB (600230782 bytes) of memory Intel AVX2 LSTM (native build): paq8px_v190_nativecpu.exe -1l -simd avx2 "C:\temp\test rav1e 0.1.0 993950d q175-s0.log" "C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm avx2 intel native.log" paq8px archiver v190 (c) 2020, Matt Mahoney et al. Creating archive C:\temp\test rav1e 0.1.0 993950d q175-s0 lstm avx2 intel native.log in single file mode... Filename: C:\temp\test rav1e 0.1.0 993950d q175-s0.log (320157 bytes) Block segmentation: 0 | text | 320157 bytes ----------------------- Total input size : 320157 Total archive size : 18385 Time 253.36 sec, used 572 MB (600230803 bytes) of memory Note that the SSE2 LSTM result matches for both CPUs but not the AVX2 version, both for Native and Non-native architecture build.
    2064 replies | 559450 view(s)
  • suryakandau@yahoo.co.id's Avatar
    12th August 2020, 16:10
    what about if the price give to the author but the certificate is for me ?? is it fair enough ??
    71 replies | 5118 view(s)
  • lz77's Avatar
    12th August 2020, 12:05
    Do not forget that you must ask the author of the open source program for permission to use it. But why does the author need competitors? The author also wants to get a prize! :)
    71 replies | 5118 view(s)
  • Gotty's Avatar
    12th August 2020, 08:58
    Gotty replied to a thread paq8px in Data Compression
    @moisesmcardona Could you please try to run a short test with a small file (like bib or book1) on your systems with memory setting -1 and simd setting (-simd sse2) where the systems are idle? Could you turn off any antivirus software in the meantime?
    2064 replies | 559450 view(s)
  • Sportman's Avatar
    12th August 2020, 08:32
    Sportman replied to a thread zpaq updates in Data Compression
    Surprising! https://www.biorxiv.org/content/10.1101/2020.07.22.216242v2.full.pdf
    2533 replies | 1101165 view(s)
  • suryakandau@yahoo.co.id's Avatar
    12th August 2020, 06:46
    I only doing research about hash function implementation on data compression and I think the bwt comment is right. The prize is to small compared to Google programmer salary. Btw Could you please make some tweaking on MCM and let me learn something about your tweaking ? Thank you
    71 replies | 5118 view(s)
  • hexagone's Avatar
    12th August 2020, 05:53
    @suryakandau@yahoo.co.id There is no value in a tweaking contest of MCM for specific data. Why don't you try to write something by yourself ? @MS1 The API for test 4 has no parameter to initialize the (de)compressor for a specific category: int32_t CDECL encodeInit( void **cmprContext ); Also, optionally a function like int32_t CDECL dispose(void *cmprContext ); would allow the (de)compressor to free memory allocated during init.
    71 replies | 5118 view(s)
  • suryakandau@yahoo.co.id's Avatar
    12th August 2020, 03:12
    Or maybe 0.25% better than original to get more participants ?
    71 replies | 5118 view(s)
  • suryakandau@yahoo.co.id's Avatar
    12th August 2020, 03:06
    3% better compression than original is too hard for the participants and the prize too small compared to 3% better than original.. How about 0.5%-1% ? Maybe the participant can grow up to enter gdcc.
    71 replies | 5118 view(s)
  • Shelwien's Avatar
    12th August 2020, 02:49
    > from gdcc notices thread, ms1 said do own work it means we can not combine open source. > it means we must to code from scratch. and it is appreciated only 3000 euros. No, Maxim (Ms1) simply doesn't want to keep testing MCM with minor changes, so now you need 3% better compression than original to get it accepted. Its really a troublesome point, because we can't really ask for fully independent implementations - we'd just not get any participants then. But accepting minor changes of open-source software also doesn't seem fair, so some compromise has to be made. > The problem is that there are too many categories. > If there were half as many, the prize would be 6000. I'd rather remove the 2nd place prize... speed categories are reasonable, and we had even more data types during discussion. In any case, this can't be changed at this point already. > An unique LZ77+Huffman can claim price only in Rapid Compression of english text... There's also blockwise test and "mixed" (kinda executables). Image test also can be made compatible by adding some filter, like http://www.radgametools.com/oodlelimage.htm
    71 replies | 5118 view(s)
  • hexagone's Avatar
    12th August 2020, 01:56
    hexagone replied to a thread paq8px in Data Compression
    Looking at memory frequency and amount is not sufficient to draw a conclusion. The sizes of the intermediate L1, L2 & L3 caches can make a big difference. Has anyone ever measured what amount of time is spent waiting for memory during compression ?
    2064 replies | 559450 view(s)
  • Darek's Avatar
    12th August 2020, 00:55
    Darek replied to a thread paq8px in Data Compression
    I think the same - memory isn't an issue. It's something in not AMD prper using AVX2 or other Intel favourise instructions. Of course if Ryzen got 3200MHz or 3600MHz memory it wil help but not a lot - maybe 5-10%. The major issue is in other place.
    2064 replies | 559450 view(s)
  • Gotty's Avatar
    12th August 2020, 00:12
    Gotty replied to a thread zpaq updates in Data Compression
    Zpaq is used as a Kolmogorov complexity approximation tool in the following article: ​ Phylogeny of the COVID-19 Virus SARS-CoV-2 by Compression Data compression is used to analyze SARS-CoV-2. Who would have thought?
    2533 replies | 1101165 view(s)
  • moisesmcardona's Avatar
    12th August 2020, 00:03
    moisesmcardona replied to a thread paq8px in Data Compression
    The laptop with the Intel i7-4700MQ runs 16GB DDR3 at 1666Mhz. The other laptop with the Intel i7-7700HQ runs 64GB DDR4 at 2400Mhz. The AMD machine is running the 128GB RAM at 2133Mhz. I personally don't think RAM is the issue, as the numbers on both Intel machines are very similar given their architecture and speed differences.
    2064 replies | 559450 view(s)
  • hexagone's Avatar
    11th August 2020, 23:59
    hexagone replied to a thread paq8px in Data Compression
    I would look at the memory specs. Typically memory is the bottleneck with this kind of software.
    2064 replies | 559450 view(s)
  • Darek's Avatar
    11th August 2020, 22:42
    Darek replied to a thread paq8px in Data Compression
    Theroretically Ryzen 9 3950x shuold be about 30-33% faster than i7-4770HQ... Looks like there are 2 times slowdown from hidden reason. Of course different builds runs different on different architectures but it's eally strange. I've checked other benchmarks for these CPUs (CPU-Z, Cinebench r11.5, Cinebench r15, Cinebench r20, Geekbench 4.0, Geekbench 5.0, Passmark, SisoftSandra Arithmetic, Userbench) and all these tests shows in average about 33% of single thread dominance of 3950x vs. 4770HQ... These must be: a) worst (really worst) case scenario for Ryzen or b) some compile/build implications
    2064 replies | 559450 view(s)
  • Dresdenboy's Avatar
    11th August 2020, 22:41
    He created that data compression site with sources and interesting articles. For the other thing - not everything is in one's own hands. I know a similar case.
    25 replies | 2579 view(s)
  • moisesmcardona's Avatar
    11th August 2020, 20:12
    moisesmcardona replied to a thread paq8px in Data Compression
    Ok, here's the results of the AMD 3950x CPU along with the previous ones from Intel. Took aprox. 200,000 seconds more on AMD than on Intel. NOTE: The real number is the CPU time, since that logs the time the process was using the CPU. Run time is the overall time it took for the process to run.
    2064 replies | 559450 view(s)
  • JamesWasil's Avatar
    11th August 2020, 20:07
    Oops, I thought your last name was Tomato not Tamayo. I must have misread. I apologize for getting your name wrong. As for the rest of it, professional help of some sort may be needed, but I doubt that a lawyer to sue the tech firms is what's required for that. I sure hope that compgt is a troll of sorts. Otherwise, this is a very sad thing.
    25 replies | 2579 view(s)
  • Sportman's Avatar
    11th August 2020, 20:01
    I see some forums hacked today, don't know if this forum need an update: Base Score: 9.8 CRITICAL https://nvd.nist.gov/vuln/detail/CVE-2019-16759 vBulletin 5.6.0, 5.6.1, 5.6.2 Security Patch: https://forum.vbulletin.com/forum/vbulletin-announcements/vbulletin-announcements_aa/4445227-vbulletin-5-6-0-5-6-1-5-6-2-security-patch
    55 replies | 8693 view(s)
  • bwt's Avatar
    11th August 2020, 18:07
    I think if we develop from the existing source code it is more efficient and participant can grow up
    71 replies | 5118 view(s)
  • bwt's Avatar
    11th August 2020, 18:02
    The main important thing You must have fresh idea to write it from scratch n improve for several months..
    71 replies | 5118 view(s)
  • lz77's Avatar
    11th August 2020, 17:31
    An unique LZ77+Huffman can claim price only in Rapid Compression of english text...
    71 replies | 5118 view(s)
  • algorithm's Avatar
    11th August 2020, 17:24
    The problem is that there are too many categories. If there were half as many, the prize would be 6000.
    71 replies | 5118 view(s)
  • bwt's Avatar
    11th August 2020, 15:27
    from gdcc notices thread, ms1 said do own work it means we can not combine open source. it means we must to code from scratch. and it is appreciated only 3000 euros.
    71 replies | 5118 view(s)
More Activity