CCM 1.25 is here!

[Christian]

Hi everyone!

The changelog is very short - as always:
1) in this release CCMx got a ~5-10% speed boost over version 1.23
2) that's it

I'm on an X2 here - I hope that the additional speed is perceivable on other platforms, too.

http://www.mytempdir.com/2034717

I have not tested this version much. If there are any problems please let me know.


Happy compressing,

Christian

[LovePimple]

Thanks Chris!

Mirror: Download

[Nania Francesco]

Intel Core duo 2 6600 2GB RAM
Tested CCMX :
Enwik8 c 7 -> 20.819.733 speed: 1203.81 KiB/s
SFC Test Maximum compression c 7->10.734.204
UCLC Wave test c 7 ->33.351.652
UCLC PGM image test->5.531.088

[Black_Fox1]

Thanks Chris, it's over 10% improvement for CCMx in my test

[SvenBent]

is it compatible with odler versions ?
Can i decompress my old file with this new release?

its seem athat CCM and CCMx are not compatible with each other.

and now back to the testing

[SvenBent]

Quick feedbakc

WinXP Pro SP2 Dan OEM.iso
606.128.128 bytes


CCMx v1.23 (6)
549.835.520 bytes
392.766 seconds

CCMx v1.25 (6)
549.813.493 bytes
383.070 seconds

2.5% faster
and just slightly smaller.
buts is progress and I'm loving it

[Black_Fox1]

Some more numbers
__compressor___output size_____in / out speed (kB/s)
CCMx 1.25 (-6)__11 947 674___ 676 / 679
CCMx 1.23 (-6)__11 947 923___ 592 / 607
CCM 1.25 (-6)___11 974 703___757 / 738
CCM 1.23 (-6)___11 974 852___712 / 731

[SvenBent]

just to answer my own question

NO you cannot use v1.25 to decompress data compressed with v1.23

[Christian]

Thanks for the numbers BlackFox, Nania and Sven!

@Sven:
Sorry, I'm a little late with my answer. As you've already found out - CCM and CCMx are incompatible as well as 1.23 and 1.25.

Only 2,5% is a little bit disappointing, but well, it's better than a bug report. By the way, which platform do you use?

[SvenBent]

WinxP pro 64 SP2 English
C23Q Q 6600 OC 3.2Ghz (9x356Mhz)
965 cchipset
2GB of ram runnig 712mhz (1:! with fsb, Cas 4)

[Hahobas]

Christian, great to have you back. Was this just a quick release, or are you back to developing ccm? These "internal changes" that you made are great.

From what I can tell you're using the GCC compiler. Any chance on compiling for linux i386, and x64? I know 64 bit versions of paq8 show great performance improvements versus their 32bit equivalents. I'd love to test a 64bit linux version to see if it performs better.

Out of curiosity, were some of these "internal changes" having to do with caching from the hard disk? I seem to get less of a performance improvement when working off a RAM disk.

Also, are you still considering making a 3rd mode with higher ratio like we had talked about a few months ago?

Well, here are some tests....

Test System: AMD Athlon 64 3000+ (single core) with 2GIG DDR (dual channel) Corsair CL2 RAM

ccmx123.exe c 7 enwik8 enwik8.ccmx123 (20,819,656 Bytes)- 2m 27sec
ccmx123.exe d enwik8.ccmx123 enwik8 - 2m 15sec
ccmx125.exe c 7 enwik8 enwik8.ccmx125 (20,819,733 Bytes)- 2m 8 sec ( 12.9% faster! )
ccmx125.exe d enwik8.ccmx125 enwik8 - 2m 10sec (3.8% faster)

Other Tests with input and output file located in ramdisk (formated as ext3) to eliminate disk effects:

ccmx123.exe c 7 enwik8 enwik8.ccmx123 - 2m 16sec
ccmx125.exe c 7 enwik8 enwik8.ccmx125 - 2m 7sec ( 7% faster )

ccmx123.exe c enwik8 enwik8.ccmx123 ( 21,601,466 Bytes ) - 2m 12sec
ccmx125.exe c enwik8 enwik8.ccmx125 ( 21,608,042 Bytes ) - 2m 3sec (6.8% faster)

ccm123.exe c 7 enwik8 enwik8.ccm - ( 21,960,016 Bytes ) 1m 41 sec
ccm125.exe c 7 enwik8 enwik8.ccm - ( 21,960,598 Bytes ) 1m 38 sec (3% faster)

[Christian]

Hi hahobas!
Thank you for testing!

Was this just a quick release, or are you back to developing ccm?

This was just a quick release. I "stumbled" over the latest source and I just had to code a few lines.

Out of curiosity, were some of these "internal changes" having to do with caching from the hard disk?

No, nothing disk relevant. I reordered the context mixing part and improved the hashing again.

I seem to get less of a performance improvement when working off a RAM disk.

I have no idea what happend here. But CCM is extremly sensible to memory bandwidth. Even when I play a MP3 on my dual core system while benchmarking, CCM is actually ~3% percent slower!

Also, are you still considering making a 3rd mode with higher ratio like we had talked about a few months ago?

Yep, Im still considering this. But dont expect this to be anytime soon. Measured by the time devoted this will take like a decade.

Any chance on compiling for linux i386, and x64? I know 64 bit versions of paq8 show great performance improvements versus their 32bit equivalents.

Since Im running a 32bit version of Linux Ill have a look at this.

[Hahobas]

But CCM is extremly sensible to memory bandwidth.

I think youre right. Ive been doing a lot of testing, and found something interesting. I have two hard disks on my system, both are SATA, but one is a 10,000 RPM WD raptor, and the other is a 7200 Seagate. If I do the following, I get these timings.

ccmx.exe c enwik8_on_seagate enwik8.ccm_on_seagate - 2m 22sec
ccmx.exe c enwik8_on_seagate enwik8.ccm_on_raptor - 2m 12sec
ccmx.exe c enwik8_on_raptor enwik8.ccm_on_raptor - 2m 12 sec

So it appears that my seagate bandwidth isnt fast enough to hold an upstream, and a downstream without impacting performance.

I wonder if any performance could be gained by doing some type of cached write of the compressed data (since it flows a lot slower than the incoming data), so that people without 10,000 rpm hard drives can have faster performance.

On a different note, I wrote a benchmarking bash script that compares any two versions with all their modes and switches of CCM so if youd like any exhaustive tests done, let me know. Testing a 32 and 64bit linux version would be especially nice

[Piotr Tarsa]

Hahobas
i suspect that pefrormance loss is caused by disk fragmentation or simulteanous use by other programs. hdds have bandwidth of tens of megabytes per second while ccmx compress at no more than a few megabytes per second (on very fast systems).

[thometal]

How about big file support? I mean about 4gb?

[l1t]

somebody say ccm and ppmd are both Symmetry
while lzma or lzpm are not, can you explain it

[thometal]

that mean:

Symmetry: compressions/decompressions speed -> time are nearly equal
Assymmetry: compressions/decompressions speed -> time are very different most compressions is X times slower as decompression

[encode]

Some additional info:

Symmetric (PPM, CM, etc.)
The compression speed is about the same as the decompression, in some cases decompression speed can be even slightly slower. PPM-based compressors and similar, at decompression just build the same data model as with compression - this takes the same amount of time as with the compression.

Asymmetric (BWT)
The decompression is faster than the compression. This is sorting related thing - the reverse sorting is faster.

Highly asymmetric (LZ)
The decompression is much faster than the compression (10X times and more). The LZ coder most time spends on string searching and parsing optimizations. The decoder is simplest - it only decodes literals and copies strings (set of bytes) from previously seen data.

Hope this helps.

[John]

I tried this and it's really fast vs the last version, and now a question/request:
Next version will be optimized for a SMP or dual/quad cores CPu's?

[SvenBent]

YES SMP and big filesupport would be nice,

[Hahobas]

How about big file support? I mean about 4gb?

I completely agree

On another note in relation to optimizing for multiple cores, I know that it is not the easiest thing to do with context mixing. However if you download the latest gcc 4.2.2 you could experiment with OpenMP (auto-parallelization) directives which could offer some multi-core performance boosts with very little work on your part.

Just an idea......

[Nania Francesco]

CCM vs FreeArc?
Comment's!

[Gish]

A very basic question:

Will CCM be released as Open Source (GPL) ?

(I'm asking the same for FreeArc)

Both CCM/FreeArc are good but if they want a chance to success, (and their creators too) I believe the only way is with Open Source.

[Black_Fox1]

CCM vs FreeArc?
Comments!

FreeArc is more flexible in terms of speed vs. ratio, CCM is more powerful

Will CCM be released as Open Source (GPL) ?

(Im asking the same for FreeArc)

Some of promising projects (these two or Precomp for instance) are not yet opensource, because their authors either think that the source code could use some more optimizing or that the source code is rather messy at the moment and needs cleaning up before releasing it.

[Hahobas]

Overal, I think the person who invents a way to make the context mixing algorithm SMP friendly will be the person with the next "big thing" in compression especially since the world is going to multiple cores.

I'm wondering if a pipe-line style approach could be used with CCM to get increased speed and/or ratio on multiple cores. It's pretty obvious that traditional context mixing is hard to split among the cores, but CCM is not context mixing alone. What if one core did the pre-processing (file filters, perhaps work like "precomp", then piped that into another core that did the context mixing etc. Is this type of thing worth it?

[Christian]

Hello there!

At all people requesting features - I'm quite busy with other things right now, thus ccm's development is more or less frozen.

Just a few thoughts on multithreading. ccm's filters are very fast - there's no gain putting them in another thread. The same applies for the data detection which is only done during compression, not decompression (note, decompression is isn't any faster than compression). But as Hahobas proposed - some expensive preprocessing like precomp could be "outsourced" (assuming that it'd be used in ccm).
But if you want to gain speed the main task remains - the creation of a multithreaded compression core. You can do this by just dividing the input data in 2,3,... chunks and you're set. But memory requirements grow linearly with the number of threads and some compression efficiency is lost (this heavily depends on the block size). If the block size is big enough this shouldn't be a problem. But still, this solution is ugly. On the other hand, context mixing is a concept which can be hardly parallelized. So, whatever you do here, you'll have to make some sacrifices.

Finally, I compiled a 32-bit linux version for hahobas. Please do not post this version anywhere else. I hope it works as expected.

link removed...

[Bulat Ziganshin]

Finally, I compiled a 32-bit linux version for hahobas. Please do not post this version anywhere else.

its important step! i hope that you will make it public as dust settles

[Christian]

its important step! i hope that you will make it public as dust settle

Sure. If there arent any problems I dont see a reason for keeping it back. But for now, I just want to make sure, that it is working ok before making it public.
So, if theres anything strange with this compile please leave me a line.
I removed the link above because ccm (without x) was missing. This one includes both versions.

link removed...

[LovePimple]

Please do not post this version anywhere else.

OK

[Hahobas]

Christian, thanks for making this linux version. The performance hit really surprises me though:

Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

linux->ccm c 3 enwik8 (22325509 Bytes) - 1 m 57 s -
linux->ccm d enwik8 - 1 m 59 s -
windows->ccm.exe c 3 enwik8 (22325509 Bytes) - 1 m 34 s - compression 24.4% faster
windows->ccm.exe d enwik8 - 1 m 37 s - decompression 22.6% faster
-----------------
linux->ccm c 7 enwik8 (21960598 Bytes) - 2 m 1 s -
linux->ccm d enwik8 - 2 m 2 s -
windows->ccm.exe c 7 enwik8 (21960598 Bytes) - 1 m 37 s - compression 24.7% faster
windows->ccm.exe d enwik8 - 1 m 40 s - decompression 22.0% faster
-----------------
linux->ccmx c 3 enwik8 (21608042 Bytes) - 2 m 26 s -
linux->ccmx d enwik8 - 2 m 30 s -
windows->ccmx.exe c 3 enwik8 (21608042 Bytes) - 2 m 2 s - compression 19.6% faster
windows->ccmx.exe d enwik8 - 2 m 3 s - decompression 21.9% faster
-----------------
linux->ccmx c 7 enwik8 (20819733 Bytes) - 2 m 32 s -
linux->ccmx d enwik8 - 2 m 34 s -
windows->ccmx.exe c 7 enwik8 (20819733 Bytes) - 2 m 6 s - compression 20.6% faster
windows->ccmx.exe d enwik8 - 2 m 7 s - decompression 21.2% faster
-----------------

In case you're wondering why my timings for the windows version are faster than in some of my previous posts, it's because I upgrade d my kernel.

Looking forward to the source being released.

Also, just for curiosity, I tested my ccm.exe in windows XP as opposed to Wine, and got:
ccm.exe c 3 enwik8 enwik8.ccm 1m 33sec

which is about the same.