LPREPAQ released!

[LovePimple]

From the MC guestbook...

Great news!

I just uploaded lprepaq to my site. It combines lpaq1 by Matt Mahoney and Precomp v0.3.5 to a complete file compressor/decompressor.

Please report if you find any bugs since it is the first version. Source (just for lprepaq, Precomp is inside a DLL) is included.

Have a look at http://schnaader.info

Greetings,
Christian "schnaader" Schneider

[Black_Fox1]

Thanks LovePimple! Seems like competition for FreeArc has just began

[LovePimple]

Quick (7) test...

A10.jpg > 699,694
AcroRd32.exe > 1,337,906
english.dic > 462,145
FlashMX.pdf > 2,336,792
FP.LOG > 402,877
MSO97.DLL > 1,690,034
ohs.doc > 740,634
rafale.bmp > 728,506
vcfiu.hlp > 514,071
world95.txt > 439,357

Total = 9,352,016 bytes

This is a truly AWESOME compressor!!!

Thanks LovePimple! Seems like competition for FreeArc has just began

[encode]

Craziest!

[Black_Fox1]

Third place in SFC @ hundreds of kilobytes per second? Crazy!

[Bulat Ziganshin]

lprepaq can't be a legitimate competition becuase it violates GPL license of lpaq

[LovePimple]

Another ( quick test...

A10.jpg > 699,694
AcroRd32.exe > 1,335,985
english.dic > 461,798
FlashMX.pdf > 2,329,857
FP.LOG > 402,393
MSO97.DLL > 1,687,335
ohs.doc > 740,378
rafale.bmp > 728,270
vcfiu.hlp > 513,286
world95.txt > 438,541

Total = 9,337,537 bytes


ENWIK8 > 19,797,006 bytes

SUPER-COMPRESSOR!!!

[LovePimple]

Third place in SFC @ hundreds of kilobytes per second? Crazy!

Third place in SFC, very fast compression time, and still lots of room for tuning!!!

Imagine PRECOMP + PAQ8L compression.

[Bulat Ziganshin]

Imagine PRECOMP + PAQ8L compression

yes!

[John]

nice, really, nice, but (always a but...) lpaq1 still really slow in a p4 2.6gHz
And program needs the dll packJPG22.dll

What about to "inject" or use 7-zip compression?

[Bulat Ziganshin]

What about to "inject" or use 7-zip compression?

wait for freearc 0.40

[squxe]

And program needs the dll packJPG22.dll
Where it must be placed? I can't compress my jpg files
edit:
oopps... a10.jpg succesfully compressed. So Lprepaq can't compress all jpg files which can be compressed by PackJPG 2.2

[LovePimple]

First person to combine the power of PRECOMP with awesome compression of PAQ8M will become forever famous!

[Vacon]

Hello everyone,

wait for freearc 0.40

At least _I_ do!

First person to combine the power of PRECOMP with awesome compression of PAQ8M will become forever famous!

Yep! And if he tunes speed of PAQ-part even more.
But the race is opened.

Best regards!

[Black_Fox1]

on my testset:

_______name______________size________in___out speed
FreeArc 0.40 -max ________11 842 821___733 / 1 057 (kB/s)
LPREPAQ 1.0 -7 _________11 454 187___ 83 / ___92
PAQ8M -7 + Precomp0.3.4__9 176 121_____3 / ____3

I will try to provide details in a few days' time...

[LovePimple]

Yep! And if he tunes speed of PAQ-part even more.
But the race is opened.

Highest compression on the planet!!!

Sorry! Just drooling!

[Bulat Ziganshin]

I will try to provide details in a few days time...

lprepaq (lpaq1) is not optimized for binary files

[Matt Mahoney]

I commented on lprepaq in the maximumcompression guestbook:
http://www.maximumcompression.com/guestbook/gb.php

But then I discovered that lprepaq doesn't compress some JPEG files that PackJPG 2.2 does, even though they are based on the same code. These 3 images: http://compression.ca/act/act-jpeg.html

Almost all of the gain of lprepaq over lpaq1 is for flashmx.pdf (1.2 MB). Also 140K for a10.jpg. It does not seem to detect the large JPEG in ohs.doc, which would probably gain another 150K. I think it could be #1 with some x86 models (e8e9 filter), some binary models (sparse order 1-2), and a bmp model.

[LovePimple]

I think it could be #1 with some x86 models (e8e9 filter), some binary models (sparse order 1-2), and a bmp model.

I hope someone capable is taking note!

[LovePimple]

LPREPAQ v1.1 has been released.

This improves JPG detection for JPG files with multiple SOI markers.

[LovePimple]

Quick ( test...

A10.jpg > 699,694
AcroRd32.exe > 1,335,983
english.dic > 461,797
FlashMX.pdf > 2,329,859
FP.LOG > 402,393
MSO97.DLL > 1,687,334
ohs.doc > 740,378
rafale.bmp > 728,270
vcfiu.hlp > 513,283
world95.txt > 438,541

Total = 9,337,532 bytes

[schnaader]

I just uploaded lprepaq v1.2.

Changes:
- Added Precomp switches to lprepaq.
- Fixed lprepaq crashes.

Have a look at http://schnaader.info

[LovePimple]

Thanks Christian!

[schnaader]

I just got a bug report that Precomp crashes under Windows Vista.
So there could be some incompatibility issue for Windows Vista.
Has anybody encountered the same for Precomp or lprepaq or managed running them under Vista?

The bug report was for 32-bit Vista by the way, I don't know if that matters...

[Matt Mahoney]

lprepaq 1.2 seems to be twice as slow as lpaq1 on the large text benchmark. Was it compiled without optimization? The preprocessing step is very fast.

http://cs.fit.edu/~mmahoney/compression/text.html# 1600

Also, there is a new version, lpaq3, plus a version optimized for large text files, elpaq3 (from same source code), both by Alexander Rhatushnyak. These are compiled with g++. He sent me an Intel compile I haven't tested yet.

[Matt Mahoney]

lprepaq cant be a legitimate competition becuase it violates GPL license of lpaq

No, it is legal. It includes source for the modified parts of lpaq1 (calling precomp to a temporary file). precomp is not under GPL and doesnt use any code from lpaq1.

[schnaader]

lprepaq 1.2 seems to be twice as slow as lpaq1 on the large text benchmark. Was it compiled without optimization? The preprocessing step is very fast.

Optimizations were enabled. G++ optimizations could be better than those of MSVC, but I doubt this.
I guess it is the ((ftell(fin) % 3276 == 0) check I use in the main compress/decompress loops. Should be something like act_fin_pos and act_fin_pos++ instead of a ftell after every byte read in ...
Ill compile a corrected version quickly.

Also, there is a new version, lpaq3, plus a version optimized for large text files, elpaq3 (from same source code), both by Alexander Rhatushnyak. These are compiled with g++. He sent me an Intel compile I havent tested yet.

Ill have a look at it and expect to release a version 1.3 that combines lpaq3, Precomp 0.3.6 (or perhaps upcoming 0.3.7? ) and packJPG 2.3 DLL (as soon as I receive it from Matthias Stirner).

No, it is legal. It includes source for the modified parts of lpaq1 (calling precomp to a temporary file). precomp is not under GPL and doesnt use any code from lpaq1.

Thanks for clarification. I assumed it would be that way, but wasnt sure because Im not that familiar with license stuff.
Hopefully Ill get my code cleaned up soon and put both Precomp and the Precomp DLL that will be released at this point under LGPL.

[Bulat Ziganshin]

Hopefully Ill get my code cleaned up soon and put both Precomp and the Precomp DLL that will be released at this point under LGPL.

it will be great day. i wait imaptiently for a day when i will be able to integrate all these great libs into freearc

i have some suggestions for precomp:

1) use multiple threads to check various zlib compression settings. precomp should be an ideal multithreaded program, after all

2) check first the most probable ones - last used, typically used (such as -6/-9 settings) and check in (very) fast mode only these typical settings

[schnaader]

1) use multiple threads to check various zlib compression settings. precomp should be an ideal multithreaded program, after all

Thats on my todo-list, but the way I use (de)flate routines is not optimal, so this will be changed first. These multi-thread/core specific optimizations will most propably be done somewhen after the big code cleanup.

2) check first the most probable ones - last used, typically used (such as -6/-9 settings) and check in (very) fast mode only these typical settings

This is already done for the last/most used modes. Precomp uses a priority list to keep track of the modes used. If one of the recompression steps returns complete success (all decompressed bytes could be recompressed), the remaining modes are skipped.

Using typical settings is also supported - just use "-c69" or whatever suites your needs

[schnaader]

I just recompiled lprepaq without the ftell "bug", but it is only in the compression part and only speeds up compression about 10%...

Recompiled version available at http://schnaader.info

G++ really seems to be faster than my MSVC compile. Even lpaq1 is slower if I compile it myself. Using -Ox or -O2 doesn't make a difference...

Perhaps someone could try to compile it with another MSVC version or with G++ (not sure if the .lib file works in that case)...

lpaq1_self_compiled 3 packjpg22.dll packjpg22.lp3
126976 -> 122160 in 3.96 sec., using 27 MB memory.

lpaq1 3 packjpg22.dll packjpg22.lp3
126976 -> 122160 in 2.62 sec., using 27 MB memory.