FreeArc 0.40 released

[MrC]

Thanks, Bulat!

In the meantime - Do you know of any tools out there that do this kind of content segmentation for .iso files? May also work great on other BIGFILES like .pak or .wad which sometimes also contain a "filesystem" and files of different types. Would be a very universal and powerful thing!

[Vacon]

Hello everyone,

the FIRST gui archiver shell for FreeArc...

Thank you Bulat
Nice to see the baby growing

Best regards!

[Simon Berger]

Hi Bulat,
I cannot understand in any way why you released this version which is still unusable for anyone with more then 2gb (unusable with -mx option).

[Bulat Ziganshin]

I cannot understand in any way why you released this version which is still unusable for anyone with more then 2gb (unusable with -mx option).

because i dont found anyone to test it. pre-4 contained two bugs related to this problem. ive fixed one of them. you should remember that you dont found time to finish testing and provide me all the info required to find next bug. hopefully now we have one tester who have more time and patience

In the meantime - Do you know of any tools out there that do this kind of content segmentation

only one, written by DURILCA author - http://www.compression.ru/ds/seg_file.rar . but i dont know whether its the same code as used in durilca (you can ask him)

Thank you Bulat

its not my program

[Simon Berger]

Patience is nothing I still need.
I didn?t notice that you fix a part of the bug . Well. If you have help I look up for a working copy soon

[Vacon]

Hello everyone,

its not my program

No, but you made the stone rolling
Besides:
arc_eng.rar -> 742,6 kB
arc_eng.wrc -> 637,6

Best regards!

[MrC]

Thanks! Will give the DURILCA seg-tool a try.

However, for .iso and other standard containers it might be possible to automate file extraction and rebuilding with FreeArc:

E.g., file.iso --> file.iso...folders_and_files...

One could then make use of FreeArc's build-in file extension-based compression.

It might be able to add this to arc.ini but I don't know how?

????
[External compressor:isoprecomp]
packcmd = isoextract $$arcdatafile$$.tmp $$arcpackedfile$$.tmp
unpackcmd = isorebuild $$arcpackedfile$$.tmp $$arcdatafile$$.tmp
????

Does anybody know how to do this in arc.ini?

[Bulat Ziganshin]

Does anybody know how to do this in arc.ini?

i think its impossible because external compressors system allows to have only one output file. but you provided great idea!!! in next version i want to allow external tools to populate netire directory and then compress these files as usual ones. seems reasonable?

[MrC]

Sounds great! This is exactly what I was looking for. We could then use general extract/rebuild tools for known "bigfiles" and possibly gain more compression.

I'm looking forward to it!

[Bulat Ziganshin]

We could then use general extract/rebuild tools for known "bigfiles"

btw, can you please give me links to such tools so that i will be able to download and test them. preferably small and cmdline. i hope that they provide bit-to-bit identical reconstruction?

[MrC]

...forgot something... Another nice feature (you probably have already thought about is) would be to have a smarter way of identifying filetypes. Identifying a file type by the file's extension works in many ways. However, it may not be reliable enough in practice as the file extension could have been changed or is ambiguous.

For example, I tested the new FreeArc audio compression feature on a set of .WAV (350 MB) files which were ADPCM-coded. Result: almost no compression! I then adapted arc.ini to use lprepaq for these wav files which resulted in shrinking down the set to about 250MB.

So - in conclusion - the analysis of files header information might help to determine the REAL file type more reliably. What do you think?

[Bulat Ziganshin]

the analysis of files header information might help to determine the REAL file type more reliably. What do you think?

overall speaking, using extensions for filetype detection is very well known weakness of FA. it was just perfect idea in terms of implementation_complexity/results ratio, but finally it will be overwritten by more advanced ones. ive already implemented filetype detection by contents, soon i will make new version available for beta testing. then i will go to implement segmentation and so on

i dont like testing headers because this means support only for specific file type. instead, more complex ways should be used: entropy checking and so on. i work on this

[MrC]

Re: Extract/Rebuild tools

Universal Extractor:
http://www.legroom.net/software/uniextract
...rebuilding of some formats might be a problem...

I played around with rasmaker which can extract and rebuild .ras files for a popular game.

There should be free extract/rebuild tools for popular game bigfiles such as CAB, WAD, PAK, BIG, ... Needs some research...

Example:
http://soldieroffortune2.filefront.c..._extractor;149

Commercial tools (this is no ad!):
ZipZag - http://www.zipzag.com/index.html
ALZip - http://www.softaward.com/8296.html - free for private use
UltraISO - http://www.ezbsystems.com/ultraiso/
...

There are probably freeware cmdline tools for many of those file types available. If there is some interest we could start a new thread on collecting some for popular formats...

[MrC]

overall speaking, using extensions for filetype detection is very well known weakness of FA. it was just perfect idea in terms of implementation_complexity/results ratio, but finally it will be overwritten by more advanced ones. ive already implemented filetype detection by contents, soon i will make new version available for beta testing. then i will go to implement segmentation and so on

i dont like testing headers because this means support only for specific file type. instead, more complex ways should be used: entropy checking and so on. i work on this

OK. As you can easily see, Im not an expert in this field. Im looking forward to testing FreeArcs new features...

[Bulat Ziganshin]

ultraiso is a gui tool. cab,pak are compressed formats. for pak(zip) there is bit-to-bit recompression tool - precomp, although it doesn't extract files from archive

please note that first stage - extract files from asrchive is very easy. the real problem is the second stage - how to recompress them again preserving bit-to-bit identity with original file

[MrC]

please note that first stage - extract files from asrchive is very easy. the real problem is the second stage - how to recompress them again preserving bit-to-bit identity with original file

I agree. A solution might be not to extract them but analyse the file for different type of contents or content patterns - i.e., virtually slicing the file to data chunks and then use specialized compression algorithms/external packers for those chunks.

But this might already be your segmentation feature...?

On the other hand, a bit-to-bit recompression might not be necessary for some types of files. The decision could, for example, be left to the FreeArc user.

[Bulat Ziganshin]

But this might already be your segmentation feature...?

yes

On the other hand, a bit-to-bit recompression might not be necessary for some types of files. The decision could, for example, be left to the FreeArc user.

yes, it is possible. this will imeddiately allow to repack zip, cab, 7z and any other type of archives

[MrC]

yes, it is possible. this will imeddiately allow to repack zip, cab, 7z and any other type of archives

...and - I guess - other "bigfiles" for which an external extract/rebuild tool is provided by the user...?

[Bulat Ziganshin]

.and - I guess - other "bigfiles" for which an external extract/rebuild tool is provided by the user...?

of course

[MrC]

of course

[Black_Fox1]

mkisofs, for instance used in nLite for making ISOs... opensource and portable utility, may come handy

[MrC]

mkisofs, for instance used in nLite for making ISOs... opensource and portable utility, may come handy

Nice! It has plenty of options...

Compiled Win32 version available here:
http://www.student.tugraz.at/thomas.plank/

[Arkanosis]

please try http://www.haskell.org/bz/arc-fixed.7z - its 0.40 version with error hopefully fixed

I cannot test it right now, because I have a computer with "only" 512MiB RAM, but Ill test it on Monday.

btw, can you try 64-bit Vista?

It should be possible, maybe even with some additional RAM. But neither until Monday.

Btw, I tested 0.40 (non-fixed) on another computer (Athlon 2000+, 512MiB RAM), on the same file, and it works fine:
<div class=""jscript""><pre>ARC 0.40 Creating archive: test.arc using exe+rep:256mb+delta+tempfile+lzma:32mb:max:bt4:128 ,
&#036;obj => rep:256mb+delta+tempfile+lzma:32mb:max:bt4:128,
&#036;text => dict:128mb:80%:l8192:m400:s100+lzp:128mb:92%:225:h 24:d1mb+tempfile+ppmd:16:384mb,
&#036;wav => tta, &#036;bmp => mm+grzip:8mb:m1:l:a
Memory for compression 384mb, decompression 384mb, cache 1mb
Started: 0.00 secs
Found 1 files, 0 archives: 0.00 secs
Sorted 1 files: 0.00 secs
Joined filelists: 0.00 secs
Compressing 1 file, 142.636 bytes. Processed 0%

Compressing 1 file of 142.636 bytes: 0.02 secs
Using dict:140kb:80%:l8192:m400:s100+lzp:140kb:92%:225:h 18:d1mb+ppmd:16:384mb: 0.02 secs
Memory for compression 384mb, decompression 384mb: 0.02 secs
Solid block compression results (0.109 seconds): 0.14 secs
dict:140kb:80%:l8192:m400:s100: 142.640 bytes in 0.016 seconds: 0.14 secs
lzp:140kb:92%:225:h18:d1mb: 142.644 bytes in 0.016 seconds: 0.14 secs
ppmd:16:384mb: 37.556 bytes in 0.078 seconds: 0.14 secs

Writing directory: 0.14 secs
Found 1 directory names: 0.14 secs
Directory written: 0.14 secs
Compressed 1 file, 142.636 => 37.556 bytes. Ratio 26.3%
Compression time 0.11 secs, speed 1.304 kb/s. Total 0.19 secs
All OK
[/code]



Of course, FreeArc didt try to allocate as much memory, so this test was maybe not so interesting.

[nimdamsk]

WTF with width? Return fixed width please!

[encode]

Write smaller lines next time...

[Simon Berger]

Quoting: Bulat Ziganshinplease try http://www.haskell.org/bz/arc-fixed.7z - its 0.40 version with error hopefully fixed
I cannot test it right now, because I have a computer with "only" 512MiB RAM, but Ill test it on Monday.

So I can say to you. it still crashes But the precentage showing seems to be a little bit different. Also it crashes at 10% I think before it was at 10,5.

[Bulat Ziganshin]

Of course, FreeArc didt try to allocate as much memory, so this test was maybe not so interesting.

the whole problem what i have only 1gb. and here everything works ok. so i need someone with 3+ gigs to debug this problem

[Bulat Ziganshin]

WTF with width? Return fixed width please!

i prefer current behavior

[nimdamsk]

i prefer current behavior

May be i am wrong, but making horizontal scrolling doesnt make me much happy

[Arkanosis]

So I can say to you. it still crashes But the precentage showing seems to be a little bit different. Also it crashes at 10% I think before it was at 10,5.

Yes, I can confirm. As I test it on a small file, I cannot see any difference.

FreeArc 0.40-fix, Core2 Quad, 3GiB RAM:
<div class=""jscript""><pre>
ARC 0.40a Creating archive: test.arc using exe+rep:1gb+delta+tempfile+lzma:128mb:max:bt4:128,
&#036;obj => rep:1gb+delta+tempfile+lzma:128mb:max:bt4:128,
&#036;text => dict:128mb:80%:l8192:m400:s100+lzp:128mb:92%:225:h 24:d1mb+ppmd:24:1536mb,
&#036;wav => tta, &#036;bmp => mm+grzip:8mb:m1:l:a
Memory for compression 2gb, decompression 2gb, cache 1mb
Started: 0.00 secs
Found 1 files, 0 archives: 0.00 secs
Sorted 1 files: 0.00 secs
Joined filelists: 0.00 secs
Compressing 1 file, 142.636 bytes. Processed 0%

Compressing 1 file of 142.636 bytes: 0.00 secs
Using dict:140kb:80%:l8192:m400:s100+lzp:140kb:92%:225:h 18:d1mb+ppmd:24:1536mb: 0.00 secs
Memory for compression 2gb, decompression 2gb: 0.00 secs
[/code]


Thinking about this, I wonder if it is not the same kind of problem as the one Igor talked about. WinXP doesnt seem to be able to allocate big memory blocks. He simply decided to remove some options for LZMA.
Ill try it on Vista64, so well be doubtless about it.