CCM 1.3x

[Nania Francesco]

Thanks Magic Cristian ! Hi!

[Hahobas]

Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM

ccm130a-3.4.6 c 3 enwik8 (22351098 Bytes) - 1 m 28 s -
ccm130a-3.4.6 d enwik8 - 1 m 25 s -
ccm130a-4.1.2 c 3 enwik8 (22351098 Bytes) - 1 m 29 s - compression 1.1% slower
ccm130a-4.1.2 d enwik8 - 1 m 29 s - decompression 4.7% slower
-------------
ccm130a-3.4.6 c 3 enwik8 (21646059 Bytes) - 1 m 53 s -
ccm130a-3.4.6 d enwik8 - 1 m 53 s -
ccm130a-4.1.2 c 3 enwik8 (21646059 Bytes) - 1 m 56 s - compression 2.6% slower
ccm130a-4.1.2 d enwik8 - 1 m 58 s - decompression 4.4% slower
--------------
ccm130a-3.4.6 c 7 enwik8 (21980533 Bytes) - 1 m 29 s -
ccm130a-3.4.6 d enwik8 - 1 m 30 s -
ccm130a-4.1.2 c 7 enwik8 (21980533 Bytes) - 1 m 32 s - compression 3.3% slower
ccm130a-4.1.2 d enwik8 - 1 m 34 s - decompression 4.4% slower
--------------
ccm130a-3.4.6 c 7 enwik8 (20857925 Bytes) - 1 m 59 s -
ccm130a-3.4.6 d enwik8 - 2 m 0 s -
ccm130a-4.1.2 c 7 enwik8 (20857925 Bytes) - 2 m 1 s - compression 1.6% slower
ccm130a-4.1.2 d enwik8 - 2 m 2 s - decompression 1.6% slower

Yeah, looks like 4.1.2 is not the best compiler to use in this case.

I rather prefer looking into changes/improvements to the algorithm than tweaking code.

I agree completely. Youve got a lot more performance that way than by code tweaking.

Thanks for your hard work!

[Nania Francesco]

Please you write the link for download the installer of gcc 3.4.6 !

[Christian]

I wonder how you do weighting without a dot product?! (or some vector math)

Actually, I do not have one big mixer, but several small ones. They create some kind of mixing-tree. I simply cant update them at the same time because they all use different approaches (static, dynamic or table driven).

Please you write the link for download the installer of gcc 3.4.6 !

I do not have a link for GCC-3.4.6 binaries for windows. I just compiled the sources tar under openSUSE. But you can download GCCs source from here and try to compile it under Cygwin or MinGW.

[LovePimple]

Hi again!
Sorry for the late reply. So, finally I do have some fresh linux binaries for you. I included binaries produced with GCC 3.4.6 and 4.1.2.

CCM 1.30a (win32+linux32)

Thanks Chris!

Mirror: Download

[Nania Francesco]

INTEL Core Duo 2 2gb Ram

SFC Test
CCMX Opt.6
10.737.064 Bytes
comp. 36,972 sec.-
dec. 37,812 sec.-
CCM Opt. 6
10.820.191 Bytes
comp.31.701 sec.
dec. 32.564 sec.

ENWIK8 opt.7
-> 20.857.925 compr. 1326 KiB/s

[LovePimple]

Quick test...

Test machine: Intel PIII (Coppermine) @750 MHz, 512 MB RAM, Windows 2000 Pro SP4

Test File: ENWIK8 (100,000,000 bytes)

Executable: ccm.exe (model size 4)

Timed with AcuTimer v1.2


CCM 1.26b (Oct 29 2007), Copyright (c) 2007 C. Martelock

Allocated 276 MiB of memory.
97656.25 KiB -> 21565.78 KiB (ratio 22.08%, speed 258 KiB/s)

Elapsed Time: 00:06:21.391 (381.391 Seconds)


CCM 1.30 - Copyright (c) 2007-2008 C. Martelock - Jan 7 2008

Allocated 274 MiB of memory.
97656.25 KiB -> 21586.76 KiB (ratio 22.10%, speed 289 KiB/s)

Elapsed Time: 00:05:41.642 (341.642 Seconds)


CCM 1.30a - Copyright (c) 2007-2008 C. Martelock - Jan 9 2008

Allocated 274 MiB of memory.
97656.25 KiB -> 21586.76 KiB (ratio 22.10%, speed 288 KiB/s)

Elapsed Time: 00:05:43.196 (343.196 Seconds)

[schnaader]

Nice release! Would you be interested in a merge with Precomp
(using Precomp DLL)? This would lead to a compression ratio
like lprepaq, but speed would be better.

[Christian]

Hi Christian!

Nice release! Would you be interested in a merge with Precomp (using Precomp DLL)?

Thanks for the kind offer. Ill contact you via mail.

[LovePimple]

These results show what we can expect from the merge.

Precomp v0.3.7 + CCMx v1.26b (7)

[schnaader]

These results show what we can expect from the merge.

Precomp v0.3.7 + CCMx v1.26b (7)

Perhaps with EXE filters on the CCM side there could be an improvement like LPREPAQ made to beat paq8o8 SFC result.
The main thing that improves comparing to LPREPAQ is the speed, Precomp+CCM will be twice as fast or even faster!

Thanks for the kind offer. Ill contact you via mail.

BTW, have you already send a mail? Nothing in my inbox so far...

[LovePimple]

Perhaps with EXE filters on the CCM side there could be an improvement like LPREPAQ made to beat paq8o8 SFC result.

[LovePimple]

Do you think the merge will cause it to be excluded from the SFC test?

[schnaader]

Do you think the merge will cause it to be excluded from the SFC test?

I dont know. I recently send Werner an e-mail and asked him if
he will add LPREPAQ 1.3 to maximumcompression, but
he hasnt answered yet...
Its obvious why LPREPAQ 1.1 (had a bug that caused it to crash
sometimes) and paq8o8pre (too slow I think) werent added, but
LPREPAQ 1.3 should be fast enough and the CCM+Precomp merge
will be even faster, so there should be a good chance that itll be
added.

However, Precomp is far away from being perfect and I think there
also could be a crash or some bug that slows it down extremely
while processing MFC...
So we will see...

[Asheboy]

I love this tool. I would love It even more if it did folders.

[Intrinsic]

You can always use QFC http://www.geocities.com/jadoxa/qfc/index.html or TAR to group them up before comrpessing.

[Asheboy]

I have been taring them before compression, just it gets annoying sometimes.

[jethro]

I like TarCCM for this purpose. Once you set it up it is very convenient.
http://www.encode.su/forums/index.ph...um=1&topic=412

[SvenBent]

<offtopic>

its there a benefith to use tar instead of just 7-zip with store method ?

7-zip 7z/store gives smalles files then 7-zip tar

[nimdamsk]

7zip gives better results archiving different file types. When files are of the same type .tar gives better results. Files are bigger because of file alignment inside of archive. 7zip compresses list of file names, tar places them as is. So if comressor is stronger then 7zip it takes benefit from raw file names placement and vice versa.

[Asheboy]

+1 to jethro for the link

[SvenBent]

"nimdansk

thanx for the info.


TAR
no comrpession of file names
no sroting of files


7-zip store
compression of filenames (hurts futher compressions)
sorting of files by extension (helps further compression)



Did i get it right ?

[Piotr Tarsa]

you can disable header compression in 7- zip.

[EneergE]

Is there some kind of setting that you have to use to utilize both cores on a dual-core cpu. I can't get CCM to utilize more than one core and I'm only getting ~1500 KiB/s with a C2D e6700 @ 3.2ghz / 2gb ram.

Using "ccmx 7"

[Hahobas]

CCM uses a context mixing algorithm, which is extremely difficult to do in more than one thread. So therefore, CCM does not have a multithreaded switch, and multi-core systems will only use 1 core.

There have been quite a few interesting comments about the subject of multithreading with CCM in some of the older CCM threads.

One exciting aspect of combining precomp with CCM however, is that the precomp work could be pushed to another thread, so those with multicore systems would not see much of a performance hit with "preCCM" versus plain CCM. The only thing they'd notice different is an out-of-this-world ratio.

[Christian]

Here's a new compilation of CCM. Nothing really new, but I moved to GCC 4.3.0 using profile optimization. Thanks again to Hahobas (and of course the GCC guys) who pointed my at this technique. It might be a tiny tiny bit faster - or not. Just try it out.

CCM 1.30b

[Hahobas]

Christian. Thanks for your continued work on ccm. Here's some benchmarks.

Test System: AMD Athlon 64 3000+ (Venice, single core) with 2GIG Corsair DDR CL2 RAM
ccm130a->ccm.exe c 3 enwik8 (22351098 Bytes) - 1 m 25 s -
ccm130a->ccm.exe d enwik8 - 1 m 24 s -
ccm130b->ccm.exe c 3 enwik8 (22351098 Bytes) - 1 m 28 s - compression 3.5% slower
ccm130b->ccm.exe d enwik8 - 1 m 30 s - decompression 7.1% slower

ccm130a->ccm.exe c 7 enwik8 (21980533 Bytes) - 1 m 28 s -
ccm130a->ccm.exe d enwik8 - 1 m 29 s -
ccm130b->ccm.exe c 7 enwik8 (21980533 Bytes) - 1 m 31 s - compression 3.4% slower
ccm130b->ccm.exe d enwik8 - 1 m 32 s - decompression 3.3% slower

ccm130a->ccmx.exe c 3 enwik8 (21646059 Bytes) - 1 m 52 s -
ccm130a->ccmx.exe d enwik8 - 1 m 53 s -
ccm130b->ccmx.exe c 3 enwik8 (21646059 Bytes) - 2 m 3 s - compression 9.8% slower
ccm130b->ccmx.exe d enwik8 - 1 m 58 s - decompression 4.4% slower

ccm130a->ccmx.exe c 7 enwik8 (20857925 Bytes) - 1 m 56 s -
ccm130a->ccmx.exe d enwik8 - 1 m 58 s -
ccm130b->ccmx.exe c 7 enwik8 (20857925 Bytes) - 2 m 1 s - compression 4.3% slower
ccm130b->ccmx.exe d enwik8 - 2 m 2 s - decompression 3.3% slower

Maybe the intel folks will see an improvement with 130b.

[Christian]

Hmmm, that's not good. On my system CCM and CCMx was around ~1% faster for compression and decompression. But on the other hand, the profile feedback was generated on MY box using much more data than just ENWIK.
Anyway, I think I'll continue to use profile optimization in the future. It's much more convenient to use it than hand tuning the source - e.g. for Slug it really helped (no more trial and error coding). RZM's decompression is faster, too. And I still hope that GCC's automagical optimizations will improve further.
Prefetching is another story, but I haven't tried it out yet.

[LovePimple]

Thanks Chris!

The latest CCM and RZM are now available for download from my site.

http://www.geocities.com/lovepimple_mail/

I have been busy today or they would have been available much sooner.

[Christian]

Here's a new compile. I removed some compiler switches and changed the data set on which the profile feedback was created. On my C2D system these executables are a tiny bit faster for compression and a little faster for decompression (than 1.30a). Now, I'll wait for some benchmarks to see if the compiler settings are heading in the right direction.

Hahobas:
Can you rerun your tests just to be sure that the timings are right. Maybe you can check something different than enwik, too?

Download CCM 1.30c