paq8pxpre

[Jan Ondrus]

Combined source from
paq8o8pre.cpp v2 aka prepaq v2 file (pre)compressor
using precomp.dll (C) 2007/08 Christian Schneider
and
paq8px_v67 file compressor/archiver. Release by Jan Ondrus, Nov. 5, 2009

             paq8o8pre v2 -7 paq8px_v67 -7 paq8pxpre -7
A10.jpg               638165        637116       637143  (+27)
acrord32.exe          910902        895521       895544  (+23)
english.dic           385584        384202       384193  (-9)
FlashMX.pdf          1821452       3554092      1588181  (-1965911)
fp.log                264893        254154       254062  (-92)
mso97.dll            1293710       1261349      1261481  (+132)
ohs.doc               483033        482511       482153  (-358)
rafale.bmp            551488        528475       528504  (+29)
vcfiu.hlp             401874        395307       395273  (-34)
world95.txt           358516        351914       351983  (+69)

total                7109617       8744641      6778517  (-1966124)

[moisesmcardona]

Thanks!

BTW, I had to compile the source again since the included exe was asking for "libgcc_s_dw2-1.dll".

I include here my compiled version.

[Skymmer]

Combined source from
paq8o8pre.cpp v2 aka prepaq v2 file (pre)compressor
using precomp.dll (C) 2007/08 Christian Schneider
and
paq8px_v67 file compressor/archiver. Release by Jan Ondrus, Nov. 5, 2009

Impressive! Thanks Jan

moisesmcardona, also thanks to you for static compile!

But sad news is that I probably found an issue. For some reasons PAQ8pxPre refuse to process one test file with lenght of 4 244 176 896 bytes. Some other smaller files are accepted but not this one. The message is:

E:\Test3>paq8pxpre -0 -slow Test.dat
Test.dat: not found, skipping...
Nothing to compress

DIR
23.06.2009  05:42            43 008 libgcc_s_dw2-1.dll
20.12.2007  17:59           134 952 packjpg_dll.dll
09.01.2010  12:55           146 944 paq8pxpre.exe
15.01.2008  10:59            87 040 precomp.dll
15.01.2010  13:22     4 244 176 896 Test.dat
20.07.2005  11:48            35 840 zlib1.dll

Can anybody confirm it ?

[Jan Ondrus]

Impressive! Thanks Jan

moisesmcardona, also thanks to you for static compile!

But sad news is that I probably found an issue. For some reasons PAQ8pxPre refuse to process one test file with lenght of 4 244 176 896 bytes. Some other smaller files are accepted but not this one.

Can anybody confirm it ?

I think there is some file size limit in all paq versions (2^31 = 2 GB).
See code below.

// Same as expand() except fname is an ordinary file
int putsize(String& archive, String& s, const char* fname, int base) {
  int result=0;
  FILE *f=fopen(fname, "rb");
  if (f) {
    fseek(f, 0, SEEK_END);
    long len=ftell(f);
    if (len>=0) {
      static char blk[24];
      sprintf(blk, "%ld\t", len);
      archive+=blk;
      archive+=(fname+base);
      archive+="\r\n";
      s+=fname;
      s+="\n";
      ++result;
    }
    fclose(f);
  }
  return result;
}

[schnaader]

I think there is some file size limit in all paq versions (2^31 = 2 GB).

This is true, and even if there wasn't, the Precomp DLL is from version 0.3.8 that perhaps also had a 2 GB limit, at least it gets slower after 4 GB. When merging PAQ and the next Precomp version, I'll try to change the PAQ code so it supports files larger than 2 GB - has this been done by someone else in another PAQ version already?

Anyway, changing the limit even in the standard PAQ versions shouldn't be that hard - just change every file position variable from int (32 bit) to long long (64 bit), handle 64-bit values correctly everywhere (PAQ header, file size display) and use fgetpos/fsetpos instead of fseek (if needed).

[[IO]]

I've been toying with the PAQ routines for a few years now (inbetween other projects). My own variants are probably faster because i'm one of the "fanatical optimising freaks".

The work ALL you guys are doing here is GREAT - thank you

Just tested "moisesmcardona"'s binary (VC++ ? So PEiD tells me) (I don't use that other compiler) against a binary I compiled with my Intel compiler and thought I would share the results with you;

D:\_PAQwerk\paq8pxpre>paq8pxpre -1 gccout Overlord_Title.mp3
Creating archive gccout.paq8pxpre with 1 file(s)...

1/1  Filename: Overlord_Title.mp3 (5494863 bytes)
Precompressing: 100.0% - Time: 764 ms
Block segmentation:
 0           | default   |   5494893 bytes [0 - 5494892]
Transformed from 5494863 to 5494893 and compressed to 4609971 bytes.

Total 5494863 bytes compressed to 4610015 bytes.
Time 221.66 sec, used 37597191 bytes of memory

D:\_PAQwerk\paq8pxpre>p8 -1 intelout Overlord_Title.mp3
Creating archive intelout.paq8pxpre with 1 file(s)...

1/1  Filename: Overlord_Title.mp3 (5494863 bytes)
Precompressing: 100.0% - Time: 655 ms
Block segmentation:
 0           | default   |   5494893 bytes [0 - 5494892]
Transformed from 5494863 to 5494893 and compressed to 4609971 bytes.

Total 5494863 bytes compressed to 4610015 bytes.
Time 196.18 sec, used 37597189 bytes of memory

I used no parameters on compiling, and the old ASM routines in the package. Interestingly, the IC++ binary is 230Kb, whilest compiling it also warned me about the fopen (it's a bit security paranoid) possible holes.

I could'nt test the other binary as I don't use, nor ever will, that compiler - but it would be good to see the speed difference.

Hopefuly I shall get time later today to actually browse the source and see if there is any obvious room for speed improvements.

[[IO]]

Using my TAQ8.asm file and my pregenerated tables (see the includes in the /OVL folder);

D:\_PAQwerk\PAQ8PX~1>paq8pxpre -1 intel_IO_out Overlord_Title.mp3
Creating archive intel_IO_out.paq8pxpre with 1 file(s)...

1/1  Filename: Overlord_Title.mp3 (5494863 bytes)
Precompressing: 100.0% - Time: 717 ms
Block segmentation:
 0           | default   |   5494893 bytes [0 - 5494892]
Transformed from 5494863 to 5494893 and compressed to 4609971 bytes.

Total 5494863 bytes compressed to 4610015 bytes.
Time 171.13 sec, used 37588993 bytes of memory

D:\_PAQwerk\PAQ8PX~1>fc /B intel_IO_out.paq8pxpre gccout.paq8pxpre
Comparing files intel_IO_out.paq8pxpre and GCCOUT.PAQ8PXPRE
FC: no differences encountered

As you can see, binary exacts to the previous precompiled version.

Enclosed is the sources & binary used - I hope this helps development.

[Jan Ondrus]

As you can see, binary exacts to the previous precompiled version.

Enclosed is the sources & binary used - I hope this helps development.

Your version crashes for me.

This is not correct replacement for llog function:

#ifdef GENTABS
  if (x>=0x1000000)
    return 256+ilog(x>>16);
  else if (x>=0x10000)
    return 128+ilog(x>>8);
  else
    return ilog(x);
#else
  return ilogtable[x];
#endif

[[IO]]

No idea why it crashes - works fine on my Vista (32bit) home premium, but feel free to try a recompile.

As for the ilog function - again, feel free to manipulate it anyway you see fit. Just hope that those things help you produce something better, thats all.

[Skymmer]

Just tested "moisesmcardona"'s binary (VC++ ? So PEiD tells me) (I don't use that other compiler) against a binary I compiled with my Intel compiler and thought I would share the results with you;

Thanks for compile! As for moisesmcardona's compile... Judging the linker version (2.56) and sections structure I think its GCC.

Your version crashes for me.

Runs OK on my system. AMD64 4000+, XP Pro SP3.

[Skymmer]

I take my words back. After more serious testing it appeared that [IO] compile also crashes on my system. I've used my TestBed.tar file and seems that IO's compile crashes on 5_HEROES.MOD file from there. At least on it.

[M4ST3R]

seems that IO's compile crashes on 5_HEROES.MOD file

Same Here. XP 32 bits SP3 E8400

Looks promising though

[[IO]]

On some data, it crashes on mine too. Or rather it's the precomp.dll which crashes. Possibly some issue between compilers or I forgot to set a flag during compile or something simple which is my fault. Might even be something in the ASM routine. Feel free to recompile and retest - be interesting to know the speed-results on that small hack.

The essential idea was to demonstrate the advantage of "pre-generated tables" (lookups) over generating "on-the-fly". It's something i've used in all my PAQ variants in the last few years as standard - this is the first i've heard of any crashing. Probably my bad for not checking with the DLL files.

Aside of that, there are numerous "minor" tweaks for performance advantages that i've integrated, aside of the obvious CPU based ones - but i'll leave them for another day - working code is obviously more important.

[LovePimple]

Thanks Jan!

Here are my usual builds for paq8pxpre...

NOTE: _speed_optimised is now renamed _alt.