Mounting a .zip file

[jaclaz]

Maybe this can be of interest to compression developers:

Mount ZIP, ISO, and Private Folder files as virtual folders to the Windows file system using the Pismo File Mount Audit Package. Free download.
Pismo File Mount

Pismo File Mount is a Windows system extension that enables application controlled virtual and user mode file systems. Using Pismo File Mount, applications can expose all kinds of program and user data through the Windows file system interface.

http://www.pismotechnic.com/pfm/

I can possibly be expanded to allow for other formats....

jaclaz

[jlowe48]

Pismo File Mount Audit Package includes features that use LZMA compression.
- Compact ISO, an indexed compressing, spanning, encrypting, signable, stubbable, ISO image container.
- Compact File Set, restricted ISO image in Compact ISO container.
- Self Mounting Executable (SMX), like a self extracting installer except can run contained applications without extracting files to hard drive.

To meet licensing goals, and because I started work on this before Igor Pavlov released his C LZMA compressor implementation, I chose to not use Igor Pavlov's LZMA SDK. Instead I implemented an LZMA decompressor from scratch, using Pavlov's implementation as format documentation and for performance and compatibility testing. The decompressor is +10% slower than Pavlov's. I still need to profile and tune the code.

Later I also implemented an LZMA compressor. Currently it only does "block mode" compression, meaning streams that are the same size or smaller than the dictionary size. Performance is terrible, less than half of Pavlov's performance. Compression efficiency is also worse than Pavlov's implementation, but it is pretty close. I have not finished the performance work, but I am already using my LZMA compressor in my SMX creation tool.

The code for my LZMA decompressor and compressor, and the hacked up Pavlov code I use for reference and for compression in ptiso, are in the ptiso source package on the download page at www.pismotechnic.com . Anyone who wants to take a look is more than welcome. The compression code is C, though much of ptiso is C++. The license is unrestrictive (non-viral), in the spirit of the zlib and bzip2 licenses.

[nanoflooder]

This could be interesting.
Any chance for this to make it into Linux?

[Bulat Ziganshin]

actually, i don't found anything better than lgpl. author just don't provide sources - only dll

[Bulat Ziganshin]

jlowe48. seems that you an author?

how about

1) using original lzma.dll
2) support Standard for compression libraries API

[jlowe48]

@Bulat,

actually, i don't found anything better than lgpl. author just don't provide sources - only dll

The LGPL license can prevent closed source developers from dealing with portability issues, such as tool sets, operating systems, and hardware.

My application for LZMA was for use in an open data format. My definition of an open data formats is that it is publicly documented and can be used without restriction, including in closed source applications. It is unreasonable to expect application developers to create their own compression or encryption implementations, so the license on the only available implementation of a chosen algorithm becomes a license on the use of the data format. The second implementation of LZMA at least minimally improves the documentation situation, and it provides a BSD-like licensed implementation.

1) using original lzma.dll

I can not use any DLLs in the self mounting stub. The LGPL+exception license on the LZMA SDK would have allowed static linking the C decompressor, but technically only if I never had a need to modify the code.

2) support Standard for compression libraries API

I might be able to use this standard API for integrating some specific codec into a format reader, like PPMd support for ZIP. This API would probably be most useful in an independently developed formatter that was distributed separately from PFM Audit Package. This would allow full flexibility over licensing issues. Note that most archive data formats and compression schemes are not random access friendly and so are difficult to use in file system applications.

p.s. Great avatar photo. I have uploaded mine.

[Bulat Ziganshin]

1. bsd/lgpl/gpl is *source* ocde licenses. you just ship dll, so you can call this a free library. also i don't think that you can say about open format if it's important part (lzma) doesn't have free open implementation

2. i tjhink, in this case you can use lzma code in your app as far as you don't modify it but just call existing functions such as LzmaEncode

3. btw, it's an interesting idea - provide meta-compressor which implements indexing over any other compressor

[jlowe48]

3. btw, it's an interesting idea - provide meta-compressor which implements indexing over any other compressor

For random access, a compressed stream needs to be broken up into blocks. The largest effective size for each block varies depending on the application, but any more than a few megabytes and the random access performance drops off. The upside is that you can much more effectively use multiple cpu cores since the blocks can be compressed and decompressed independently.

A special purpose random access compression algorithm could be even more interesting. It would work kind of like video compression, with A-blocks and B-blocks. The A-blocks start with reset compression state and so can be decompressed independently. The B-blocks start with the compression state left from an arbitrary earlier A-block. The decompressor at most has to decompress 2 blocks to get any 1 block from the stream, caching would help with this. Compression efficiency could increase compared to stream compression since the compressor could choose a better matching compression state than what happens to be left from the previous block. Could add C-blocks to give the compressor more options. Some of the same techniques used in pre-sorting filters could be used for choosing A-block compression states.