paq8px

[Gotty]

@Darek, you can use your own usual dictionary file - the difference is only formatting. See my earlier post here.

@Surya, as Darek is getting seriously interested in your improvement, I looked at it. Unfortunately buggy. The very same problem as you used to have in most of your releases. It even hits an assert. I'm going to fix it, and then it will be a v201fix1 indeed.

@Darek:
Give me a moment, I will post a fix and the desired exe.

Edit: it will take more time. I verified only the lstm changes, and they didn't bring good results (just small fluctuations: some testfiles get slightly better some got slightly worse). I'll need more tests.

[Gotty]

Let's have a list of bugs/issues first.

So it's about Surya's v201fix1 posted here.

- You don't have a new changelog entry in the contribution. You should add one. Luckily we have a "list of changes" it in your post. It says: "i have make a little improvement on lstm model by adding 1 mixercontextsets and 2 mixerinputs". When comparing the posted source code with the hxim repo it turns out (after some hacks to fix the line endings) that actually there are modifications in textmodel, too. Indeed in a later post we have got the info. Please include all the changes in the changelog file and try to include it in your post.
- paq8px.cpp was not updated with the new version number.
- In TextModel::mix - We see a new variable "uint64_t ii = State * 64;". The original variable (i) is the context counter. Using the new variable (ii) you have just restarted it. A context counter must not be restarted. Just simply continue using "i".
- In TextModel::mix - Don't forget to remove your comments from the code that is not intended for publishing: "//185796"
- In SimdLstmModel.hpp - there are 3 new lines. All of them have issues. The usual ones: bits of the different context constituents are overlapped (apm3 and apm4) or overflown (m.set). The first two are just "issues" but the latter is a bug.

It's quite easy to fix the problems but verifying which brings the actual improvement or which is just noise I'll need to run many tests. Especially with the lstm model improvements it will take some time, so please be patient.

[CompressMaster]

@Gotty
well, suryakandau REALLY SHOULD follow your advices and if he have an improvement, he really should post these in paq8sk thread and not here.

[mpais]

That explained it. So fixing my detection routine jumped to no1 spot on my to do list. My next version will be about detections and tranforms anyway - as requested. It fits perfectly.

Sorry, I just saw this reply now. If you're planning on refactoring the detection, I have an unreleased Fairytale prototype that might be of interest. It implements a hybrid pool of memory and physical storage to use as a scratch buffer for recompression using a single allocated memory block and a single temporary file, so it fixes the main problem from the first prototype. Writing parsers for it is also much easier than for paq8px since it solves the contention problems we get, so no more of this:

if (!gifi && !bmpi && !tgai && ...)

[kaitz]

You can also look at paq8pxv detection. Maybe it helps. There is no negative test for somethig detecting at the same time. Not sure if its same as above.

[Gotty]

I have an unreleased Fairytale prototype that might be of interest.

I would be very glad! I haven't started integrating any pxd transforms yet, also didn't fix any bugs yet. So feel free to twist it and turn it. It will be a big change I believe.

[mpais]

It might be overkill, since it would likely require a big rewrite.

After calling it, you're left with the full block segmentation (including deduped blocks).

Since it uses a user definable scratch buffer, some content may not be recompressible within those limitations, so it would be skipped (e.g., a DEFLATE stream that expands to 3GB but we only allow usage of 2GB for temporary storage).

It also requires using its "hybrid" streams, which hold the block content in memory and/or physical storage, and might have been evicted from the scratch buffer to make way for other streams, so they need to be "revived".

If you're interested I can go into more detail, though a lot of it was already discussed here and on Gitter at the time.

My GDCC entries were prototypes of ideas I had for some codecs for it, the main goal for me was to have a framework that handled detection and clustering so that with specialized codecs one could get compression ratios close to paq8px but at 2-3 MB/s on contemporary hardware.

[Gonzalo]

@mpais: Any chance of releasing it?

[moisesmcardona]

Or maybe start a Fairytale fork for PAQ8PX and slowly merge it with the main branch?

[mpais]

@mpais: Any chance of releasing it?

Sure, I just need to clean it up a bit. Do you guys want a little example of how to use it, like we did with the previous prototypes, so you can test it?

Or maybe start a Fairytale fork for PAQ8PX and slowly merge it with the main branch?

That's something that has crossed my mind - just skip all the extra complexity of the Fairytale project and simply use this to build a new, no-frills CM compression engine.
I'd call it "paqx" as a way to continue the legacy of the name, and it seems like the logical next step, since paq9 already exists.

Any thoughts?

[Gonzalo]

Sure, I just need to clean it up a bit. Do you guys want a little example of how to use it, like we did with the previous prototypes, so you can test it?

That'd be great, yes! I don't think it'll be too difficult to figure out how to test it, but the author's recommendations are always appreciated.

[mpais]

The code is now live on GitHub.

That'd be great, yes! I don't think it'll be too difficult to figure out how to test it, but the author's recommendations are always appreciated.

It's pretty similar in usage to previous prototypes, but the repo doesn't include a usage sample, it's just the library code.
I also didn't port all the parsers, and haven't really looked at the code in a really long time, so it may not be up to snuff.

I'll see if I can cook up something when I have some time so it can be tested.

Let me know what you guys think.

[mpais]

I made a really simple example of how to use the analysis stage, for those who tested the previous prototypes it should be very familiar.

[Gotty]

no-frills CM compression engine.

I'm not sure - what does it mean? Could you describe the idea a bit deeper?

[mpais]

I'm not sure - what does it mean? Could you describe the idea a bit deeper?

Fairytale was supposed to be an extremely complex and powerful archiver, with a lot of features that don't make much sense if you just want to make an experimental CM compressor ala paq8.

After the initial analysis and content-based deduplication stage, it'd optionally run an additional classic deduplication stage on default blocks, followed by an optional clustering and similarity sorting stage to determine the order in which the blocks would actually get solidly compressed. All this was designed to improve compression ratio even when using simpler codecs, like zstd, brotli, lzma, etc.

You could choose the codec sequence to apply to each block type, e.g., for 24bpp images you could chain a fast whole image decorrelation filter with zstd, which would give you better compression than even the most optimized PNGs while retaining the very fast decompression provided by zstd. And anyone here with an idea for such a filter could quickly implement it as a codec in a Fairytale fork and see the results, without having to write any parsers, entropy stages or code for archiving functionality.

By spending more time when compressing on performing better data segmentation, you can then use highly specialized methods for each type of data to get much better compression ratios with minimal or even no extra cost in terms of decompression performance.

Now, if your interest is just in pushing the envelope in terms of compression ratio, you might not really care for all of this, since you'll probably be using so much memory and such complex models that the gains from all this extra complexity may be residual at best.