Radyx archiver

[Conor]

Radyx v0.9.1-beta is released:
https://github.com/conor42/Radyx/releases

Radyx is a multithreaded .7z archiver with a radix sort match finder.

Changes in this version:

• Non-recursed relative paths are now stored, as with 7-zip.
• Filesystem caching of the output file is turned off if free RAM is low.
• Reduced the memory footprint when archiving large numbers of files.
• If some files couldn't be added, a summary is printed at the end.

UPDATE:
v0.9.2 is released: https://github.com/conor42/Radyx/releases

Improvement on Silesia compared to the 2015 release:

v0.9.1:
Compressed size: 50638204 bytes

Commit = 215740 KB = 211 MB
Work Set = 213784 KB = 209 MB

Kernel Time = 0.234 = 1%
User Time = 65.343 = 348%
Process Time = 65.578 = 349%
Global Time = 18.749 = 100%

v0.9.2:
Compressed size: 50183419 bytes

Commit = 113616 KB = 111 MB
Work Set = 110556 KB = 108 MB

Kernel Time = 0.109 = 0%
User Time = 51.859 = 365%
Process Time = 51.968 = 366%
Global Time = 14.195 = 100%

[Stephan Busch]

options:

7-Zip -mx9, fb273, d128m (-mt=3 to reduce memory usage)
Radyx -ma=2 -md=1g -mds=64k -mfb=254 -mx9 -r

                  7-Zip    Radyx 0.9.1
Test_APP        79.554.059  81.564.462
Test_Audio     541.964.587 541.042.215
Test_Camera    499.134.219 497.561.916
Test_Gutenberg 318.256.984 310.560.877
Test_Installer 577.098.508 577.603.111
Test_Mobile    487.373.858 487.467.372
Test_PGM       365.719.047 367.236.108
Test_Sources    36.173.383  36.107.723
Test_XML       197.672.999 192725242
encoding time   2848 sec 1327 sec

Radyx has a weighted average of 123.779 which is rank 71
7-Zip has a weighted average of 124.145 which is rank 79

This means of Radyx: better compression while being twice as fast as 7-Zip..
Interesting work. Very well done.

[Conor]

Thanks for the test results

It's worth noting that 7-zip uses only 2 threads unless you specify -m0=lzma2, and then it will use an even number of threads at -mx5 and above. So with -mmt=3 it was using 2 threads. Radyx defaults to the number of cores in the machine, because it doesn't cost much in compression ratio and the threads have below normal priority.

[Kennon Conrad]

Results on enwik9 for Radyx -ma=2 -md=1g -mds=64k -mfb=254 -mx9 -r:

Compression: 146 seconds, 5901 MB RAM
Compressed file size: 199,859,004 bytes
Decompression with WinRAR took about 14 seconds and verified.

That would rank #67 on LTCB with much faster compression than any higher ranked LZ based compressor.

[Stephan Busch]

I haven't used more threads with 7-Zip because this was my standard setting.
Below is the same test with 7-Zip using 6 threads (could not use 8 because I don't have enough free memory).

                  7-Zip    Radyx 0.9.1
Test_APP        79.552.176  81.564.462
Test_Audio     542.492.381 541.042.215
Test_Camera    499.258.382 497.561.916
Test_Gutenberg 321.069.886 310.560.877
Test_Installer 577.113.358 577.603.111
Test_Mobile    487.604.053 487.467.372
Test_PGM       366.073.937 367.236.108
Test_Sources    36.173.383  36.107.723
Test_XML       200.418.526 192.725.242
encoding time   1784 sec 1327 sec

The numbers still show the benefits of Radix matchfinder - we can use larger dictionary and more threads;
especially on text and xml data we have a compression improvement.
Would it also be possible to make radix matchfinder work for older LZMA and not just for LZMA2?
Do you think there is still room for improvements?
Why are some testsets compressed worse compared to 7-Zip?
Who invented radix matchfinder?

[Nania Francesco]

Radyx is very nice 7zip archiver evolution ! Why Extraction is not supported ?

[Conor]

The numbers still show the benefits of Radix matchfinder - we can use larger dictionary and more threads;
especially on text and xml data we have a compression improvement.
Would it also be possible to make radix matchfinder work for older LZMA and not just for LZMA2?
Do you think there is still room for improvements?
Why are some testsets compressed worse compared to 7-Zip?
Who invented radix matchfinder?

It will work for LZMA too except it would be partially multithreaded. The encoding step would be single threaded, or 2 threads in fast mode, one for match selection and the other for range encoding.
At this point I'm not sure what I would do to improve it further.
I think the main reason for some files compressing worse is the variable dictionary size, from 1/8th of the full dictionary at the start of a block to the full dict at the end, then back to 1/8th.
As far as I know I was the first to use the radix matchfinder in 1997 in the Imp archiver, which was a single threaded LZ77 compressor with Huffman coding.

[Conor]

Radyx is very nice 7zip archiver evolution ! Why Extraction is not supported ?

Thanks
Extraction isn't supported yet because I haven't had time to add it. It will be next.

[a902cd23]

Radyx [64] 0.9 beta Copyright 2015 Conor McCarthy 2015-02-10
>radyx a -ar=off -md=3700m x0.exdupe.7z x0.exdupe
Compression was alot smoother now, some small graphic fault in flash player but no crashes.

7zG.exe test crashed totaly after 1110 MB
Shown in window:
Total size 8098 MB
Processed 3328 MB
Compressed size 1110 MB

7z.exe also crashed.

[Conor]

Have you tried any other programs that can handle .7z files? I wonder if the problem is with 7-zip or the compressed data.

[Nania Francesco]

Conor, in the confirm how much I admire your work done with Radyx, I wanted to ask if there was a chance to help me with LZA project (open source) by inserting the multi-threaded match Finder method in my program!

[a902cd23]

Winrar 5.21 also crashed. Is there any other program I can test with?

[Conor]

Conor, in the confirm how much I admire your work done with Radyx, I wanted to ask if there was a chance to help me with LZA project (open source) by inserting the multi-threaded match Finder method in my program!

Thanks for the kind words Nania. Using the match finder is actually quite simple. Include MatchTable.h and then declare a MatchTable object with the right storage type. For dictionaries <= 64Mb and maximum search depth <= 62 use:
MatchTable<PackedMatchTable> match_table(dictionary_size, 0, search_depth);
otherwise:
MatchTable<StructuredMatchTable> match_table(dictionary_size, 0, search_depth);

Then create a DataBlock struct with your data buffer pointer, start position and end position, and a ThreadPool, and call BuildTable. To read a match, call GetMatch like this:
MatchResult match = match_table.template GetMatch<kMatchLenMax>(block, index);
If the match is longer than search_depth, the data will be compared to get the full length up to kMatchLenMax.

The problem with incorporating it into existing code is that it's block based, so you need to load an entire dictionary size of data, compress it, memmove some data from the end to the beginning and fill the block with more data. The DataBlock structure has a 0 start position for the first block and then you set it to the overlap (which is 1/8th of the dictionary size in Radyx) after that. You are the most familiar with how your code handles input data so you can probably do the mod more easily than me.

You'll need to build MatchTable.cpp, MatchTableBuilder.cpp, Progress.cpp and ThreadPool.cpp in your project.

Hope this helps

[Conor]

Winrar 5.21 also crashed. Is there any other program I can test with?

Maybe zpaq will do it. Winrar probably uses the 7-zip dll or source code.

[Nania Francesco]

Thanks for the kind words Nania. Using the match finder is actually quite simple. Include MatchTable.h and then declare a MatchTable object with the right storage type. For dictionaries <= 64Mb and maximum search depth <= 62 use:
MatchTable<PackedMatchTable> match_table(dictionary_size, 0, search_depth);
otherwise:
MatchTable<StructuredMatchTable> match_table(dictionary_size, 0, search_depth);

Then create a DataBlock struct with your data buffer pointer, start position and end position, and a ThreadPool, and call BuildTable. To read a match, call GetMatch like this:
MatchResult match = match_table.template GetMatch<kMatchLenMax>(block, index);
If the match is longer than search_depth, the data will be compared to get the full length up to kMatchLenMax.

The problem with incorporating it into existing code is that it's block based, so you need to load an entire dictionary size of data, compress it, memmove some data from the end to the beginning and fill the block with more data. The DataBlock structure has a 0 start position for the first block and then you set it to the overlap (which is 1/8th of the dictionary size in Radyx) after that. You are the most familiar with how your code handles input data so you can probably do the mod more easily than me.

You'll need to build MatchTable.cpp, MatchTableBuilder.cpp, Progress.cpp and ThreadPool.cpp in your project.

Hope this helps

I'll try but I hope you give me a help if I need it! Thanks !
Which compiler should I use (GCC, Intel ...) and what options should I set as I've seen that uses a particular C++ dialect ?

[a902cd23]

Maybe zpaq will do it. Winrar probably uses the 7-zip dll or source code.

Zpaq does not support lzma2. And yes, Winrar uses 7zxa.dll (2015-01-03).

This time I tested:
for /l %a in (1024,512,4096) do radyx a -ar=off -md=%am C:\HD-TEMP\TEST_%a.7z x0.exdupe
It all went well until 3584 was about to start, there my USB-disk lost its content. Windows could show its content but not read it, had to pull cable and reinsert it.
7zG test was successful of 1024,1536,2048,2560 but 3072 crashed at compressed 1500 MB.
Retry compress from 3072 crashed again with: Cannot read L:\BACKUP\HD-TEMP\X\x0.exdupe
This time my USB-disk disappeared, again pulled cable and reinsert to get it back.
Although looking at TEST_3072.7z with hex editor I can see filename (x 0 . e x d u p e) at the end of the file, so to the naked eye it looks right, but test crashed again.

This reminds me of the old days then trying to use an old OS function on a new OS, causing bad behaviour.

[Conor]

Oh well, I have started working on a decompressor but it will be a while. I'll sort it out then. Thanks for trying

[Gonzalo]

Very good job

BUT:

Incorrect parameter in switch -md=1024m at 1024m

Always the same, tried various values.
O.S.: MS Windows Vista 32 bits.
The same under Linux/Wine, on different machine.
What happened?

Also, the generated archive (w/o -md option) can´t be opened by 7-zip... ¿?

[Conor]

Very good job

BUT:


Always the same, tried various values.
O.S.: MS Windows Vista 32 bits.
The same under Linux/Wine, on different machine.
What happened?

Also, the generated archive (w/o -md option) can´t be opened by 7-zip... ¿?

The 32-bit version is limited to a 192Mb dictionary. I may be able to increase that a bit. I also need to add a specific error message for this situation.

How big is the archive? It would help a lot if you can send it to me so I can see why 7-zip can't open it. Also the source file (compressed with 7-zip) would help.

[Gonzalo]

The 32-bit version is limited to a 192Mb dictionary. I may be able to increase that a bit. I also need to add a specific error message for this situation.

How big is the archive? It would help a lot if you can send it to me so I can see why 7-zip can't open it. Also the source file (compressed with 7-zip) would help.

Thanks. I wasn't aware of the limit. My error.

The file isn't that big (~192 mb) but currently I have only 1m bandwidth. Working on that... Maybe in 2 or 3 weeks I'll have 6m.
I will try to reproduce the ¿bug? on a smaller archive.

[Conor]

Radyx 0.9.2 is released: https://github.com/conor42/Radyx/releases

This version uses the Fast LZMA2 library v0.9.1 which I released yesterday, adding an improved version of the incompressible data handler from the old Radyx: https://github.com/conor42/fast-lzma...ses/tag/v0.9.1

Improvement on Silesia compared to the 2015 release:

v0.9.1:
Compressed size: 50638204 bytes

Commit = 215740 KB = 211 MB
Work Set = 213784 KB = 209 MB

Kernel Time = 0.234 = 1%
User Time = 65.343 = 348%
Process Time = 65.578 = 349%
Global Time = 18.749 = 100%

v0.9.2:
Compressed size: 50183419 bytes

Commit = 113616 KB = 111 MB
Work Set = 110556 KB = 108 MB

Kernel Time = 0.109 = 0%
User Time = 51.859 = 365%
Process Time = 51.968 = 366%
Global Time = 14.195 = 100%

[joerg]

First thank you very much for such a wonderful piece of software!

I think your radyx archiver is an awesome software!

even your radyx 091 from 2015 is great - wonderful!

Gratulation!

First test with an inputfile of 65 GBytes (database dumpfile)

running on a windows server 2012R2 (8 simultan cores) under vmware:

the new 7zip 18.01 compress better and faster then the 7zip bevor

but in my test the radyx compressor compress even a lot better !

radyx 091 -m2 , result filesize:7016496135 , 136 minutes

radyx 092 -m2 , result filesize:7028869503 , 94 minutes
radyx 092 -m3 , result filesize:6056465033 , 229 minutes

7zip 1801 -m=3, result filesize:8940615341 , 15 minutes
7zip 1801 -m=5, result filesize:7060202412 , 106 minutes
7zip 1801 -m=6, result filesize:6889341349 , 123 minutes
7zip 1801 -m=8, result filesize:6331389375 , 189 minutes

Can we have this radyx archiver in a version 1.0 with even more speed ?

a wonderful compressor! - it is a dream!

if i something miss in the compare with the 7zip

- the 32-bit-version does not work for me under xp or windows server 2003
is this possible?
- then the very fast mode 7zip -m=3
... compress the 65 GBytes in 15 minutes
but this is another field

best regards

[Conor]

Hi joerg, thanks for your message

I'm not sure which level you're using because the results seem inconsistent with the levels. It looks like you're using the encoder mode setting -ma={0-3}, not the level setting -mx={1-12}
The level sets the encoder mode and other settings for the best speed/ratio tradeoff. Level 8 is similar to 7-zip level 5 and uses the same dictionary size. Level 1 is a bit slower than 7-zip level 1 but compresses a little more. I think there's little room for further speed improvements with this algorithm. I've squeezed everything out of it.

The 32-bit version is targeted at Vista and above. It may be okay to change that to XP. I'll try it before the next release.

[load]

unfortunately, it doesn't compile under mingw.

../Path.o:Path.cpp.text+0x366): undefined reference to `__imp_PathMatchSpecA'
collect2.exe: error: ld returned 1 exit status
Makefile:39: recipe for target 'radyx' failed

[AiZ]

@load

Make sure you have package mingw-w64-x86_64-headers-git installed (for 64bit).

Modify last line of Makefile :

$(CXX) -static -pthread -o radyx $(objects) /mingw64/x86_64-w64-mingw32/lib/libshlwapi.a -lm

[load]

thanks, in the meantime i got it working by simply adding -lshlwapi

i just wanted to test which compiler builds faster binaries.

[Conor]

In my tests with mingw64 and gcc 7.2.0 it was a bit faster than VS2015 at level 1, slightly slower at level 9, but the library decompression benchmark was 5-8% slower with gcc. VS2017 also seems to produce slower binaries at compression and decompression. Radyx doesn't use the decompressor yet so it can only be tested in the library benchmark.

[joerg]

@conor: thank you very very much!

your radyx - compressor is for me the very best compressor for now, if i want to compress one or two big files!

awesome! especially the -mx=11 and -mx=12 modes are wonderful! WOW!

very innovative parallel processing!

strong compression (in my tests the resulting files are even smaller then from 7zip)

but, if i want to compress with "radyx a -r ..." a whole directory with subdirectories,
then sadly, it seems to have 2 bugs:

- If a subdirectory is empty, then the subdirectory is not included in the generated archive file
- the date/time stamps of the included subdirectories seems to be not correct


best regards

PS:

in my tests

66.172.170.240 bytes with -mx=12 -mmt8 needs 4 hours - compress to 5.931.260.213 bytes
66.172.170.240 bytes with -mx=12 -mmt6 needs 6 hours - compress to 5.930.482.303 bytes
66.172.170.240 bytes with -mx=12 -mmt5 needs 7 hours
66.172.170.240 bytes with -mx=12 -mmt4 needs 8 hours - compress to 5.929.556.097 bytes

but the mode -mmt2 seems to compress better then -mmt1

[Conor]

@conor: thank you very very much!

You're welcome joerg. One thing currently missing is storage of folder information such as attributes and date/time. This also means that empty folders are not recorded either.

[joerg]

@conor: thank you for your answer

i am using the radyx-compressor to create a daily backup from a database-dumpfile

I have 8 cores and i am using -mmt4
this means 50 % cpu power for 8 hours and it works very well.

But cause the directory-problem
it is sadly unusable for for other backup purposes for me for now.

I think you should focus on the high compression modes like -mx=10 -mx=11 -mx=12

- do you plan to eliminate the problems with timestamps and empty folders in a 1.0 version?
- can you please try to build a binary for linux (redhat)?

- How we can support you?