PPMX v0.05 - new PPM-based compressor

**encode** · 19th January 2009, 19:41

OK, new PPMX v0.05 is here!

Too many improvements since v0.04, including newly invented SEE, heavy code and parameter optimizations. All in all, it's the first actual release of my PPMX!

Enjoy!

**encode** · 19th January 2009, 19:45

SFC -> 12,314,165 bytes (with no filters)

calgary.tar -> 775,647 bytes
canterbury.tar -> 516,063 bytes

**Simon Berger** · 19th January 2009, 20:00

Hehe, your compressed size for calgary.tar grow more and more in your previews in the 0.04 thread and is now with the release even higher..

Is 0.05 now at least faster then 0.0.4 or was it because of problems in decompressing.

**osmanturan** · 19th January 2009, 20:23

Could you give a short briefing about modeling details?

**encode** · 19th January 2009, 20:35

Originally Posted by osmanturan

Could you give a short briefing about modeling details?

Technically, PPMX v0.05=PPMX v0.04+SEE. i.e. same model set, etc. However, the model is larger, ALL parameters was optimized with my new automated optimizer. SEE is very important with PPM. SEE adjusts the escape count/probability based on some additional info - SEE context. For example, SEE context may contain various fields and flags such as - Do we have masked symbols? Model order, Quantized Total Count, etc.

**osmanturan** · 19th January 2009, 20:54

Is your context for SEE aligned? Say 16x, 32x, 64x etc? I'm asking this because it has a good effect on BIT.

**encode** · 19th January 2009, 20:56

Originally Posted by osmanturan

Is your context for SEE aligned? Say 16x, 32x, 64x etc? I'm asking this because it has a good effect on BIT.

Yep!

**Vacon** · 19th January 2009, 21:11

Hello everyone,

great news!

IIRC you thought about opening sourcecode. Are you ready for this step...?

Best regards!

**encode** · 19th January 2009, 22:39

Originally Posted by Vacon

IIRC you thought about opening sourcecode. Are you ready for this step...?

Not yet. As with all of my programs I'll release it as an Open Source project at SourceForge.net only after I'll finish working on it. Furthermore, I do have some plans to do a PPMX compression library. Anyway, first of all I should beat PPMd...

**LovePimple** · 19th January 2009, 23:30

Originally Posted by encode

OK, new PPMX v0.05 is here!

Too many improvements since v0.04, including newly invented SEE, heavy code and parameter optimizations. All in all, it's the first actual release of my PPMX!

Enjoy!

Thanks Ilia!

**m^2** · 20th January 2009, 01:40

Again I didn't manage to finish the next update before you released it.

Will do quick tests tomorrow.

**pat357** · 20th January 2009, 17:31

PPMX might as well become a replacement for the old PPMd

I wonder how much you'l be able to improve ratio & speed further...
What other improvements can we expect concerning PPMX ?

**encode** · 20th January 2009, 18:16

Originally Posted by pat357

What other improvements can we expect concerning PPMX ?

Taking into account that I just started... you can expect anything!

**m^2** · 21st January 2009, 19:50

Originally Posted by encode

Not yet. As with all of my programs I'll release it as an Open Source project at SourceForge.net only after I'll finish working on it. Furthermore, I do have some plans to do a PPMX compression library. Anyway, first of all I should beat PPMd...

Do you intend to release LZSS sources too?

**encode** · 21st January 2009, 19:52

Originally Posted by m^2

Do you intend to release LZSS sources too?

If people will have interest. Also, I have a few ideas for further LZSS improvement, including a tiny ASM-driven decoder.

**m^2** · 21st January 2009, 19:54

Originally Posted by encode

If people will have interest. Also, I have a few ideas for further LZSS improvement, including a tiny ASM-driven decoder.

I am interested.

**encode** · 21st January 2009, 19:55

It would be cool if optimize the NTFS compression. NTFS file system has the simplest LZSS encoder/decoder. Currently, the encoder is oriented for fastest decompression, keeping lots of "air" in the compressed stream. Making an optimized varsion will make a compressed files smaller at the cost of compression time. Anyway, the decompression speed will be the same or even faster...

**Mihai Cartoaje** · 21st January 2009, 20:38

If you have an algorithm for file-system compression, you will be able to submit it for ZFS.

ZFS is the shit!

**lunaris** · 21st January 2009, 20:46

Originally Posted by encode

It would be cool if optimize the NTFS compression. NTFS file system has the simplest LZSS encoder/decoder. Currently, the encoder is oriented for fastest decompression, keeping lots of "air" in the compressed stream. Making an optimized varsion will make a compressed files smaller at the cost of compression time. Anyway, the decompression speed will be the same or even faster...

You can post some ideas at ntfs-3g - a free/opensource software implementation of ntfs.

http://www.ntfs-3g.org/

**m^2** · 21st January 2009, 20:53

Originally Posted by lunaris

You can post some ideas at ntfs-3g - a free/opensource software implementation of ntfs.

http://www.ntfs-3g.org/

MS would have to support it. With different compression it wouldn't be NTFS anymore.

**encode** · 22nd January 2009, 20:37

PPMX's homepage:
http://encode.su/ppmx/

**encode** · 11th February 2010, 19:15

For a last few months I'm working heavy for a new PPMX 0.06. Many times new PPMX was rewritten from scratch. I tried many techniques from simple trees to a hashed linked lists. Experimented with SSE2 (Streaming SIMD Extensions 2). Got an extreme speedup with a low order models. All in all, writing a good PPM that may compete with the PPMd is rather complex task. Additionally I explored a new SEE technique, far more superior to that previously used by me. So, new PPMX will be oriented on speed and efficiency. It's already four times faster than a previous release...

**encode** · 14th February 2010, 16:09

Well, currently, an order-5 PPMX compresses book1 to 214439 bytes. Adding more aggressive model update we may loose some compression on text files, but will have a serious compression gain on binary files. Anyway, I do plan to achieve at least 215xxx bytes on book1 and at the same time be cool with binaries...

**Matt Mahoney** · 20th February 2010, 05:03

Finally tested ppmx 0.05. http://mattmahoney.net/dc/text.html#1936

**encode** · 20th February 2010, 14:45

Thanks a lot!

**encode** · 11th July 2010, 22:51

Continue working on PPMX. This time I make PPMX small and fast and it MUST be released since current version (0.05) is such unoptimized compared to what I've got now. PPMX 0.05 has many redundant computations, lots of inefficient and dummy code... New version has an extremely simple and flexible code, it's not overloaded with extra stuff, but it uses some tricks to gain a little bit compression, it's a (relatively) low order PPM (an order-4) and this fact makes it quite specific - on some files like english.dic and rafale.bmp it's really efficient on others it's not that efficient. As an option I may add a small LZP preprocessor. Anyway, the goal is a new PPM-based compressor that have different properties than PPMd. New PPMX uses an optimized hashing, so it's memory usage is fixed and no need to flush or rebuild any tree like in PPMd...

**Shelwien** · 11th July 2010, 23:52

Its already good if there's some progress

But I'd suggest to avoid tuning stuff to specific formats like wordlist or uncompressed images -
its too easy to make better specific models for these.
Also I still think that the tree is the main feature of PPM. With hashtables it would be probably
better to do CM over o1 huffman code or something.

**encode** · 12th July 2010, 00:12

Originally Posted by Shelwien

Its already good if there's some progress

I spent far much more time on PPMX than on BCM... PPMX is MUCH more complex. And it's much interesting to work on real context coder that have no BWT laces.

Originally Posted by Shelwien

But I'd suggest to avoid tuning stuff to specific formats like wordlist or uncompressed images -
its too easy to make better specific models for these.

It's not really about the tuning. PPM encodes a symbol via just one context, sometimes a higher order contexts (usually order-5 and above) may provide completely wrong predictions. A good example is a simple high-order PPM's performance on already pointed rafale.bmp and english.dic. To avoid that, we should add some stuff like II, that not really helps, much more computationally expensive LOE and/or SSE, making PPM too heavy and these days better write CM. Today PPM should be fast, memory efficient and simple.

**Shelwien** · 12th July 2010, 00:26

Yeah, so as I said... there's no sense to care about compression of raw text (not preprocessed), wavs, bmps, or exes -
and when you consider writing a fast codec for any of these, there's no need for context switching (PPM) or mixing (CM) -
just plain structured symbol coding gives good enough compression and is fastest by definition.

**encode** · 12th July 2010, 00:28

Originally Posted by Shelwien

...and when you consider writing a fast codec for any of these, there's no need for context switching (PPM) or mixing (CM) -
just plain structured symbol coding gives good enough compression and is fastest by definition.

What do you mean, structured symbol coding?

Thread: PPMX v0.05 - new PPM-based compressor

Thread Tools

Search Thread

Display

PPMX v0.05 - new PPM-based compressor

Similar Threads

BCM v0.09 - The ultimate BWT-based file compressor!

BCM v0.08 - The ultimate BWT-based file compressor!

BCM v0.01 - New BWT+CM-based compressor

PPMX - a new PPM encoder

TURTLE incoming... Fast PPM file compressor.

Tags for this Thread

Posting Permissions