New Idea for Hybrid 7-Zip Archiver

[DeathTheSheep]

The idea is simple: Combine precomp, 7-zip, and Ultra7z in one modified 7-zip application. Currently, a PAQ-based "lprepaq" exists for a similar purpose, but is slower, less mainstream compatible and less GUI-friendly, whereas 7-zip is more user-friendly, and a .7z file, perhaps containing an extra copy of the drag-and-drop capable precomf.exe if the end user does not have the modified 7-zip, is far more compatible across a wide range of operating systems, programs, and applications.

After testing this combination, I've found it to be quite powerful and versatile. For instance, a typical representative 31621KB tarball containing various sorts of materials is compressed to standard .7z [solid, Ultra, word size=273, mc=1000000000, lzma] at 17117KB. With Ultra7z alone, the size shrinks to 16941KB. With precomp run on the tarball before compression, default .7z produced a 14262KB file, and after running Ultra7z, the size shrinks to 14250KB.

The "modified" .7z specifications should be very simple and backwards compatible as such:
The archived materials are stored in a tarball, which is fed to precomp and compressed inside of a standard .7z file, with the option to include precomf.exe in the root of the archive. Extention can be .pcf.7z or something unique like .gxz (x for 'extreme' compression).
The modified 7-zip program (or 7z file browser, etc) will either recognize the new extension (.gxz) or the .7z->.pcf structure of the archive and automatically parse the .pcf as if it were "opening"/"viewing" the contents of the .pcf directly. The end user will simply open the .7z or .gxz and manipulate it just like a normal .7z archive, as if there were no pcf intermediate.

I cannot code well, but with slight tweaks to the .PCF format to report directory trees and filenames in a tarball, and the introduction of a simple pre-processing (make tarball, run precomp to make pcf) and postprocessing (to run Ultra7z) steps to the standard compression algorithm, the compression-side implementation should be comparatively trivial. For decompression-side, a bit more work is required; tweaking the .pcf spec to allow "parsing" of its contents, detection of the 'modified' archive type, and decompressing from both .7z and .pcf at once, implementation of single-file access handled analogous to 7-zip's handling of single-file extraction in solid archives. However, the incredible compression gain, often surpassing ratios of the slowest archivers, is well worth it for this versatile, fast-decompressing format.

I make this suggestion because it seems quite straightforward in implementation, and would likely prove far easier than developing various integrated filters for 7-zip/lzma.
Any feedback/input/interested takers?

[m^2]

FreeArc -max took this idea a bit farther.
It uses modified 7zip compression + special compression for text together with a bunch of filters, precomp among them.

[DeathTheSheep]

That's what they say, but whatever the case, it's simply not true of my files: using mode 7 and -mx, freearc 0.05a produced a 17233KB archive (17128 if tarballed), which is higher than even the default 7-zip option I mentioned above.

For reference, even nanozip in 'cm' mode produces an archive of 16146KB, which is much larger than the combination I suggested, and takes much longer to compress/decompress as well.

Yet another reason I suggested use of 7-zip is because it is much more widely supported by other archive managers and operating systems, and would be in a sense 'backwards compatible' with existing .7z archives as I've described it.

The only advantages freearc seems to provide over my method are compression speed (but apparently not ratio in this case) and built-in filters for wav and bmp, which I assume can be implemented in 7-zip's lzma (right now there are even stated plans for a wav filter from Igor Pavlov).

[Simon Berger]

The problem is you have to break the 7z format and have to spread it. I am currently developing a new 7-zip GUI and will include something similar to ultra7z optimizer or how the name exactly is .
I thought of using precomp because I like this program and the purpose very much but I don't know yet and also creating archives is far away. A file / archive browser and extracting is way more important as a start.
We will see what the author of 7z allows in the next format of lzma(2)/7z

Yet another reason I suggested use of 7-zip is because it is much more
widely supported by other archive managers and operating systems, and
would be in a sense 'backwards compatible' with existing .7z archives as
I've described it.

Backward compatible with the use of precomp? In no way...

[m^2]

That's what they say, but whatever the case, it's simply not true of my files: using mode 7 and -mx, freearc 0.05a produced a 17233KB archive (17128 if tarballed), which is higher than even the default 7-zip option I mentioned above.

For reference, even nanozip in 'cm' mode produces an archive of 16146KB, which is much larger than the combination I suggested, and takes much longer to compress/decompress as well.

Yet another reason I suggested use of 7-zip is because it is much more widely supported by other archive managers and operating systems, and would be in a sense 'backwards compatible' with existing .7z archives as I've described it.

The only advantages freearc seems to provide over my method are compression speed (but apparently not ratio in this case) and built-in filters for wav and bmp, which I assume can be implemented in 7-zip's lzma (right now there are even stated plans for a wav filter from Igor Pavlov).

It does work. Do the troubleshooting, probably FA doesn't find precomp.

The main (though not the only) advantage of FreeArc is that it almost always generates smaller archives in shorter time than 7zip. Then most archivers actually.

[DeathTheSheep]

Backward compatible with the use of precomp? In no way...

The archive's actual compression and container is pure 7-zip, so this part is. However, the .pcf inside must be processed via another step, supposedly by using the precomp executable provided in the root of the .7z. In this way, the entire process is "pseudo-backwards-compatible" in that the entire operation can be performed without any additional/modified software other than what is provided in the archive itself.

Do the troubleshooting, probably FA doesn't find precomp.

Ah, interesting. I had no idea freearc actually used the program "precomp"--I thought all filtering was built-in. I'll give it a try, but I don't expect much more gain over the 7-zip method in time or compression (also no embedded bitmap or waveform in here to take advantage of freearc's two main filters).

[m^2]

Ah, interesting. I had no idea freearc actually used the program "precomp"--I thought all filtering was built-in. I'll give it a try, but I don't expect much more gain over the 7-zip method in time or compression (also no embedded bitmap or waveform in here to take advantage of freearc's two main filters).

Yeah, it's not going to be much different as essentially it's the same, with several tweaks.
BTW grab extended config created by pat357 and try the following command line switch (instead of -max):

-mecm+precomp+delta+rep:1g+rzm/$exe=precomp+bcj2+delta+rep:1g+rzm/$text=dicte1:p+lzp:768m+pmm:16:1024mb:r1/$wav=ofr:bestnew/$bmp=mm+blizs/$jpg=precomp+rzm/$jpgsolid=precomp+rzm

It needs several external programs in order to work correctly:
-ecm
-precomp
-7za
-rzm
-OptimFrog

Note: I didn't test it, there might be some mistake.

Yeah, ability to tweak settings much better is another advantage of FA.

EDIT: It needs Blizzard too.

[encode]

It needs several external programs in order to work correctly:
-ecm
-precomp
-7za
-rzm
-OptimFrog

I personally dislike FreeArc. First of all, it's not a real archiver - it's like a collection of data compression software with buggy and very strange GUI. In addition, many of these programs are strictly experimental - like RZM or precomp - no guarantee about any stability and correct decompression. The compression engine should be built-in or at least accessed via DLL. Give me ONE executable with WinZip/WinRAR-like interface!!! Otherwise, I'll call it a compression script. Since it's like writing a BAT or script file, collect many third-party command-line compressors, and based on file extension select appropriate executable.

[m^2]

I personally dislike FreeArc. First of all, it's not a real archiver - it's like a collection of data compression software with buggy and very strange GUI. In addition, many of these programs are strictly experimental - like RZM or precomp - no guarantee about any stability and correct decompression. The compression engine should be built-in or at least accessed via DLL. Give me ONE executable with WinZip/WinRAR-like interface!!! Otherwise, I'll call it a compression script. Since it's like writing a BAT or script file, collect many third-party command-line compressors, and based on file extension select appropriate executable.

Yep, I also think that the modes involving external compressors are pointless for real use. I can't wait for a good support for plugins. Good=allowing creation of SFX archives.
But they are cool to play with ideas. The OP asked about precomp, which is alpha anyway, so I think that use of a dozen other experimental tools is OK.

BTW FA has a great GUI, way better than WinZip / WinRar. You can find it in "Addons\TotalCommander MultiArc plugin".

[Vacon]

Hello everyone,

[...]
BTW FA has a great GUI, way better than WinZip / WinRar. You can find it in "Addons\TotalCommander MultiArc plugin".

True if you use it in DoubleCommander (OpenSource-clone of TC, as FreeArc designed for many different plattforms)

Best regards!

[Bulat Ziganshin]

this realy looks more likely to FA talk

1) -mx is maximum built-in compression (lzma+ppmd+filters), -max also adds a few external programs (ppmonstr and precomp)

2) i agree with Ilya that fa is just a collection of best compression algos available - it was its entire goal, read documentation beginning. with -mx you use best open-source algos, while -max or pat357 inifile allows you to experiment more and squeeze out a little more bits. taking into account that there is no one ideal algorithm, and noone (except probably for Christian and Sami) can implement all the algos required to make up good modern archiver, this means that fa has at most one real competitor when we goes to real world

3) i mainly concentrates on archiver part, so GUI (and compression algos) are bastards here

4) lzma + filters cannot compete with specialized MM compression algos

5) until you will write your own precomp implementation (which is unlikely) and include it into 7z sources (which is absolutely impossible) you will not get 7z version that is more widespread than FA. so i think that we will have the same thing forever - 7z is more widespread and FA has better compression.

6) the only way to get precomp into fa is to write its own implementation. i'm pretty sure that its sources will not be released

[SvenBent]

Sorry for going OT

DTS what happend to you X.264 guide ?
seem to be gone from dts unites.

it woulde be nice if it could ba available again and maybe updated with the explaing of the short names ESA,TESA.

-- edit --

what DTS is trying it to do is the same thing I'm doing manually.
using 7-zip as a container and withint that container is all my "exotic" compression programs and a batchfile to decompress it

Im using
precompt
ecm
rzm
delta and rep


I really wish some archiving program would adopt ECM filtering as is really fast simple to do and works great

[DeathTheSheep]

Ah, my guide is still offline, eh? Yes, I should get that fixed...

That is a very interesting bout of chain compression you have there! Excuse my unfamiliarity with many of those programs; would it be impossible to send me a sample archive with all of these components, including the batchfile?

How long does it take on average to compress/decompress (datarate)?
How much space do the "decompression-script" files take up in the archive?

I'm delighted to see that my idea is hardly a radical one. And like Ilia, I much prefer a well-implemented 7-zip over freearc as of right now. Perhaps the situation will improve in the future, but freearc still seems to me a veritable "hodgepodge" in its own right.

[SvenBent]

I don't do the compression with batch files as i do a brute force of the different filtering and compressors ( i also include winrar and nanozip)

butt i do have the extract.bat i made which is automatically decompresses and defilters the files and erase itself and the "exotic" programs leaving back only the real data.

Well heres is my extract.bat

@ECHO OFF
set wait=ping 127.255.255.255 -n 1 -w 255


REM ----- HANDLING NanoZip FILES -----
ECHO nz.exe x %%1 *.* > DcmpTMP.bat
ECHO del %%1 >> DcmpTMP.bat
ECHO exit >> DcmpTMP.bat
for %%f in (*.nz) do start DcmpTMP.bat "%%f"

:checknanozip
%wait% > nul
if exist *.nz goto checknanozip
del nz.exe



REM ----- HANDLING .RZM FILES -----
ECHO rzm.exe d %%1 %%1.tmp > DcmpTMP.bat
ECHO del %%1 >> DcmpTMP.bat
ECHO exit >> DcmpTMP.bat
for %%f in (*.rzm) do start DcmpTMP.bat "%%f"

:checkrzm
%wait% > nul
if exist *.rzm goto checkrzm
ren *.tmp *.
ren *.rzm *.
del rzm.exe



REM ----- HANDLING .REP FILES BEFORE DELTA -----
ECHO rep.exe -d %%1 %%1.tmp > DcmpTMP.bat
ECHO del %%1 >> DcmpTMP.bat
ECHO exit >> DcmpTMP.bat
for %%f in (*.rep) do start DcmpTMP.bat "%%f"

:checkrep1
%wait% > nul
if exist *.rep goto checkrep1
ren *.tmp *.
ren *.rep *.
del rep.exe



REM ----- HANDLING DELTA FILES -----
ECHO delta.exe -d %%1 %%1.tmp > DcmpTMP.bat
ECHO del %%1 >> DcmpTMP.bat
ECHO exit >> DcmpTMP.bat
for %%f in (*.del) do start DcmpTMP.bat "%%f"

:checkdelta
%wait% > nul
if exist *.del goto checkdelta
ren *.tmp *.
ren *.del *.
del delta.exe



REM ----- HANDLING .PCF FILES -----
for %%f in (*.pcf) do Precomp.exe -r %%f
for %%f in (*.pcf) do del  %%f

del precomp.exe
del zlib1.dll
del packjpg_dll.dll



REM ----- HANDLING ECM FILES -----
ECHO unecm.exe %%1 > DcmpTMP.bat
ECHO del %%1 >> DcmpTMP.bat
ECHO exit >> DcmpTMP.bat
for %%f in (*.ecm) do start DcmpTMP.bat "%%f"

:checkecm
%wait% > nul
if exist *.ecm goto checkecm
del unecm.exe





del DcmpTMP.bat
del %0

its has several shortcummings

1 :
Only work in the root of the extracted achieve

2:no errors handligns what so ever

3:
wastes time on unnecessary wait


however it does support SMP

I might improve upon it sometime but since i lack any real programming skills...

sometimes it saddens me to have the ideas but not the skills

i will upload the filters and compressors later

[Black_Fox]

6) the only way to get precomp into fa is to write its own implementation. i'm pretty sure that its sources will not be released

Maybe Christian could release sources of some older not-so-good-now version, that surely wouldn't hurt so much

[Bulat Ziganshin]

>Perhaps the situation will improve in the future, but freearc still seems to me a veritable "hodgepodge" in its own right.

i agree that comparing 7z and fa gui now you will find fa very unfriendly. but said about cmdline versions and especially when you are going to use special batchfiles to decompress data - i don't see why fa is worse

>I don't do the compression with batch files as i do a brute force of the different filtering and compressors

btw, support for brute force search (at least, trying different compressors/settings and selection most compact representation) would be useful for fa too...

[SvenBent]

I hope FA will incorporate some kind of brute force rep+delta filters

i don't mind using cpu time on compressions. but decompression needs to be fast. in this case brute forcing combos is perfect

[DeathTheSheep]

Just as a technical question (putting aside talk of FA, which I think is better suited to a thread of its own ), how difficult would it be from a coding perspective to implement some basic filtering (delta, rep?, etc) in lzma? To me it seems the only reason 7-zip is losing benchmarks is because it is unable to adapt its filtering and compression schemata to different types of data. If this is addressed, there's no rational reason it shouldn't beat out far more compressors. In the maximal practical compression test hosted on freearc.org, 7-zip is seen very close (similar ratio, sometimes better or wose, slightly better compression speed, slightly inferior decompression speed) to FA, even though the following comments are made:

The only program that...doesn?t include multimedia compression. ... Unfortunately automatic selection of compression algorithm is not implemented. FreeArc: supports LZMA, PPMD and multimedia compression algorithms with an automatic choice between them (by file extension) plus additional filters, therefore outperforms 7-zip...But far not reliable as 7-zip. (emphasis mine)

About 7-zip:

"This combination of high functionality and reliability, high compression ratio, rapid decompression made 7-zip the most popular archiver among all the mentioned here... Lot of features, GUI, speed and reliability ? 7-zip."

Perhaps these statements are meaningless or irrelevant (also, 7-zip's compression speed increased quite a bit since that version), but I hope you can see where I'm coming from when I speak of adding built-in filters for 7-zip. Precomp perhaps is a moot point, since the closed sources can only be integrated by Christian himself (much like he did with lprepaq, hint hint ) and ultra7z can perhaps be simply applied by the user afterward if he desires. However, I've learned that many other very simple features can take their place to an extent; as simple as something like if extention = ".txt" compmethod=ppmd, as complex as adding a primitive delta/rep filter before the compression loop, other content filtering or rearrangement prior to compression, etc. We will then have ourselves quite the brilliant hybrid archiver. It can easily be a "competitor" to FA, but with the reliability, GUI, feature-completeness, and ubiquitous nature as 7-zip. Many of these things can even be done within the specs of the .7z format, entailing full backwards compatibility.

[IsName]

DTS, are you sure that rep is _really_ that primitive? We indeed can iclude it as a match finder, indeed. Backward compatible.

[DeathTheSheep]

DTS, are you sure that rep is _really_ that primitive?

No, not at all. I do know that delta filters in general are 'primitive' for wav processing when other 'smart' algorithms exist to greatly optimize the process. I was simply referring to coding primitive versions or implementations of these filters.

Sadly, I don't know which of the "filters" I mention can be made backwards compatible. I do know that a per-extention compression method switch is certainly so, as is organizing files into similar data types types prior to compression so that similar groups can be processed together. Ultra7z is another example, which serves as post-processing instead. These methods of "filtering" are indeed by definition backwards compatible. However, actual coded data-parsing filters as they exist in current form are certainly not. If, however, the ideas from these methods can be implemented into the lzma framework to produce streams within spec, then yes, they would be backwards compatible.

[Black_Fox]

As a little side note, 7zip's GUI is nothing to write home about and all attempts to improve it were so far ignored by Igor.

[Bulat Ziganshin]

As a little side note, 7zip's GUI is nothing to write home about and all attempts to improve it were so far ignored by Igor.

most programs in this class (highest practical compression) - i.e., ccm, uharc, fa 0.40, durilca'light - doesn't have gui at all. so, even its basic GUI makes great difference and means that its user base is much higher than of other - great on all other aspects - programs

[Bulat Ziganshin]

it's possible to add all fa features to 7z. but
1) this will break compatibility with old versions (except for rep-style matches in lzma, but it's only small part of fa improvements)
2) i don't expect such work from Igor, so you should make your own "super-7z" project - how it's better than fa?
3) fa means not only better compression, but also other things lacking in 7z - portable GUI, for example. how many years you will need to duplicate this effort?

so, fa by itself is post-7z archiver. taking into account that Igor has other things to work for, i don't expect that he will seriously improve 7z. and other developers just can't. it's not easy tp understand 7z sources. although adding fa compression libs is easy so i see on this way mainly "political" problems