Bigger dictionaries without extra effort

[Mexxi]

In the last few days I experimented with the safe mode's command prompt to allow the use of bigger dictionaries with 7z. My comp sports 4GB of RAM and in Windows I'm usually limited to a 256MB dictionary. I tested both the safe mode of XP x64 and Win7 x64. Xp x64 allowed me to use dictionaries up to 338MB and Win7 up to 333MB. I checked the compression gain with two file sets. One was vm.vdmk from Bulat's High Compression Benchmark and the other one was the install directory of Mass Effect.

vm.vdmk:

7z - 791.513.501 - (Win gui: 256MB dic, solid, wordsize 273, LZMA2)
7z - 789.677.324 - (-t7z -m0=LZMA2 -mx9 -md337m -ms8g -mhc=on -mf=on -mfb=273 -ma=1 -mmc=1000000000 -mmf=bt4 -mc=880803840b)
7z - 789.513.857 - (-md350m)

The gain between the different dictionary settings was smaller than I had anticipated, yet it beat the compression that Bulat managed to get. The bigger dictionary had a more tremendous impact on the Mass Effect archive. I didn't log the numbers down to the last byte, but the size decreased from 4509MB to only 4473MB just by improving the dictionary size (syntax was the same as above).

Now it's no wonder that the compression increased. That's what to expect. My point is rather to find a way to get an even better performance out of an archiver without upgrading the computer and without code changes. Currently I'm working on a bootable x64 command shell (based on Win2003's kernel) that uses less RAM than the safe mode's command prompt to enable even bigger dictionaries. My goal is to get at least 350MB out of a 4GB machine. For developers this is certainly less interesting, still I decided to share my experience anyway. If someone wants to give my command shell a go, just drop me a message.

Edit: Taking Minwin as a basis would enable the use of 372MB dictionaries on 4GB machines. That would be a great testing platform.

[DARcode]

Interesting, thank you, would appreciated updates here in this thread on the prosecution of your work.

[Shelwien]

http://google.com/search?q=define%3Aprosecution

[Mexxi]

Interesting, thank you, would appreciated updates here in this thread on the prosecution of your work.

Thanks for the interest. I switched to a WinPE 3.0 environment and started to streamline it. So far I reduced the file count from 9080 down to 170 and the folder count from 2050 to 15. I lowered the file system size from 825MB to 55MB, getting close to Microsoft's MinWin size of 25MB. Currently I can compress my new PE which I christened MicroPE to 16MB with 7z standard settings (Ultra compression).

Thanks to the streamlining, the minimal amount of RAM required to run PE was lowered from 256MB to 108MB where I still have 40MB of free RAM after booting. I hope to get down to 96MB. RAM usage on my 4GB machine went down from over 400MB to 235MB, leaving me with 3960MB of free RAM which currently enables me to use 348MB dictionaries in 7z. So I'm getting pretty close to my initial goal of 350MB. I guess I'll manage to reach that tonight, but of course I'll try to slim down MicroPE further.

MicroPE in its current form is still pretty unoptimized. It loads some useless services which I will try to remove. Also, the registry is too big and will be reduced as a final cosmetic step. Right now, several driver sets are being loaded to ensure compatibility to several machines. However, once I'm finished, I'll slim my personal version down to a level that it will only run on my machine, removing any non-essential drivers, including keyboard and command interpreter support. I already experimented with it. Instead of giving you a command prompt after booting, MicroPE directly executes the command line in question which has to be defined beforehand in the primary operating system. After the job has been completed, MicroPE will reboot to the primary OS. This lowers the file count by about 15 and the RAM usage by may be 10MB.

[m^2]

Thanks for the interest. I switched to a WinPE 3.0 environment and started to streamline it. So far I reduced the file count from 9080 down to 170 and the folder count from 2050 to 15. I lowered the file system size from 825MB to 55MB, getting close to Microsoft's MinWin size of 25MB. Currently I can compress my new PE which I christened MicroPE to 16MB with 7z standard settings (Ultra compression).

Thanks to the streamlining, the minimal amount of RAM required to run PE was lowered from 256MB to 108MB where I still have 40MB of free RAM after booting. I hope to get down to 96MB. RAM usage on my 4GB machine went down from over 400MB to 235MB, leaving me with 3960MB of free RAM which currently enables me to use 348MB dictionaries in 7z. So I'm getting pretty close to my initial goal of 350MB. I guess I'll manage to reach that tonight, but of course I'll try to slim down MicroPE further.

MicroPE in its current form is still pretty unoptimized. It loads some useless services which I will try to remove. Also, the registry is too big and will be reduced as a final cosmetic step. Right now, several driver sets are being loaded to ensure compatibility to several machines. However, once I'm finished, I'll slim my personal version down to a level that it will only run on my machine, removing any non-essential drivers, including keyboard and command interpreter support. I already experimented with it. Instead of giving you a command prompt after booting, MicroPE directly executes the command line in question which has to be defined beforehand in the primary operating system. After the job has been completed, MicroPE will reboot to the primary OS. This lowers the file count by about 15 and the RAM usage by may be 10MB.

108 MB? There's a lot to do. Win XP can run with 18 MB, though then it uses page file intensively (and detects 20 MB^^). With 24 MB it run a simple java service w/out killing my HDD...though page file was still in use.

[Mexxi]

108 MB? There's a lot to do. Win XP can run with 18 MB, though then it uses page file intensively (and detects 20 MB^^). With 24 MB it run a simple java service w/out killing my HDD...though page file was still in use.

Four reasons why your comparison doesn't work at all:

- I use an x64 environment which requires different files with different dependencies and different memory usage.

- 7z requires extra system files in XP to work at all, raising the memory consumption significantly (We're talking about 3MB ole-ddls here).

- You use a pagefile, I don't (yet) since this wouldn't allow to measure the improvements accurately. I will enable a pagefile as a final step though.

- You're comparing XP to a Windows 7 core. Win 7's kernel has a lot more dependencies than that of XP. So of course it will take up more memory. In that regard: You have a lot of work to do on your XP, because I could run Java on Win95 with only 4MB RAM :P


Anyway, my goal is not to use MicroPE in a minimal environment. On the contrary. I only posted the differences in minimal requirements to showcase the massively lowered memory consumption. If you wanted to make a valid comparison, then run your XP with 4GB RAM and see how much of it you can access. I promise you, that this seemingly sleek operating system will suddenly eat up 10% of the resources easily. This is where MicroPE shines. While it might be outrun on minimal memory usage, its overall memory consumption is a lot lower than that of XP in bigger environments.

Still, currently MicroPE only eats 70MB RAM (out of the 10 without pagefile, so I assume its memory consumption could easily be halved with one, making your 24MB for such an antique operating system look pretty excessive lol. Especially if you compare it to WinMin. It ran an http-server while only eating 32MB RAM (90MB pagefile) and offering all features of a modern kernel that XP lack.

[m^2]

Four reasons why your comparison doesn't work at all:

- I use an x64 environment which requires different files with different dependencies and different memory usage.

- 7z requires extra system files in XP to work at all, raising the memory consumption significantly (We're talking about 3MB ole-ddls here).

- You use a pagefile, I don't (yet) since this wouldn't allow to measure the improvements accurately. I will enable a pagefile as a final step though.

- You're comparing XP to a Windows 7 core. Win 7's kernel has a lot more dependencies than that of XP. So of course it will take up more memory. In that regard: You have a lot of work to do on your XP, because I could run Java on Win95 with only 4MB RAM :P

These are your design choices. x64 is certainly important, but others are not. If your choices are suboptimal, so will be the result, even if you manage to reach your goals.

Anyway, my goal is not to use MicroPE in a minimal environment. On the contrary. I only posted the differences in minimal requirements to showcase the massively lowered memory consumption. If you wanted to make a valid comparison, then run your XP with 4GB RAM and see how much of it you can access. I promise you, that this seemingly sleek operating system will suddenly eat up 10% of the resources easily. This is where MicroPE shines. While it might be outrun on minimal memory usage, its overall memory consumption is a lot lower than that of XP in bigger environments.

Lol. No, XP doesn't implicitly use large amounts of RAM. Possibly things like filesystem buffers differ, but even if, it can be adjusted. I can't think of any other reason for XP using different amounts of memory on different machines. Though I admit I never run it with 4GB.

Still, currently MicroPE only eats 70MB RAM (out of the 10 without pagefile, so I assume its memory consumption could easily be halved with one, making your 24MB for such an antique operating system look pretty excessive lol.

I'll pass on it to avoid XP-Vista war, because it's pointless.

Especially if you compare it to WinMin. It ran an http-server while only eating 32MB RAM (90MB pagefile) and offering all features of a modern kernel that XP lack.

Neither of us will reach MinWin (you misspell it) levels for the simple reason that neither of us has access to Windows sources and neither of us will bother to decompile binaries.
And, honestly, saying that MinWin has more feature-rich kernel than XP is utter BS. That's exactly the opposite of what WinMin was.

[Mexxi]

These are your design choices. x64 is certainly important, but others are not. If your choices are suboptimal, so will be the result, even if you manage to reach your goals.

What are you talking about?? The sole purpose of my project is to utilize as much memory as possible for 7z. This is a very specific project, so don't act like the way I try to accomplish things is sub-optimal, unless you prove there's a better way...

Lol. No, XP doesn't implicitly use large amounts of RAM. Possibly things like filesystem buffers differ, but even if, it can be adjusted. I can't think of any other reason for XP using different amounts of memory on different machines. Though I admit I never run it with 4GB.

So you admit you never used it with such a setup, yet you claim the impossibility of my statement? Way to go! Go ahead and adjust what you assume can be adjusted before making any claims that are little more than vague assumptions. Unlike you, I actually slimmed down XP to a handful of files and didn't find a way to lower its memory usage. There's also no documentation and also the gurus that have been working on slim XP versions for years offer no solution. So I suggest to do what you claim is possible, otherwise your argument isn't worth the webspace it takes up...

I'll pass on it to avoid XP-Vista war, because it's pointless.

There is no XP-Vista war. If you actually read my statement then you would have seen that I tried XP as well as Win2003 before switching to a Win7 core. I'm neither a fanboy, nor biased. In fact, I tried what I wanted to achieve on about every kernel architecture. Win7 actually proved to be easily configurable and a lot more efficient due to its componentized nature. There's no reason to argue about that. The numbers speak for themselves.

Oh and just for the sake of completeness: An XP-based PE doesn't support pagefiles contrary to a Win7-based one, lowering its performance significantly, but I'm sure you already knew that... Speaking of sub-optimal design choices lol...

Neither of us will reach MinWin (you misspell it) levels for the simple reason that neither of us has access to Windows sources and neither of us will bother to decompile binaries.
And, honestly, saying that MinWin has more feature-rich kernel than XP is utter BS. That's exactly the opposite of what WinMin was.

I misspelled it due to a letter switch I spelled it correctly in my very first post, so thanks for this unnecessary correction lol. While we're at it: The correct conjugation would be "misspelled" lol

You're wrong on MinWin. It serves as core for Win7 and PE 3.0 and therefore is available, though not in its slim form. The sole purpose of MinWin is to offer a minimal kernel with a componentized nature which offers an easy way to expand functionality without the need for kernel updates and rewrites. Thanks to that, it's possible to actually expose the core without the need to disassemble libraries or sources. For XP that certainly would have been necessary, but not for Win 7/PE 3 and I'm on the best way to prove it. I already removed very essential services that couldn't be removed under XP due to them being low-level requirements for the system.

And yes the kernel is more feature rich, though not in the literal sense. The fact that it can be enriched with features so easily while offering a more modern approach to system management justifies this description.

Neither of us will reach MinWin...

I never claimed I wanted to. I said I used it as a frame of reference instead of a much older Windows based on a totally different kernel architecture. I never said it was my goal to achieve MinWin's performance...

[Cyan]

What about MicroXP for example ?
I remember having a copy of it which would boot at 40MB.
It's not PE-based though.

[m^2]

What are you talking about?? The sole purpose of my project is to utilize as much memory as possible for 7z. This is a very specific project, so don't act like the way I try to accomplish things is sub-optimal, unless you prove there's a better way...

I did not say they are suboptimal. I suspect they are, but avoided definitive statement because proving it is not worth my time.

So you admit you never used it with such a setup, yet you claim the impossibility of my statement? Way to go! Go ahead and adjust what you assume can be adjusted before making any claims that are little more than vague assumptions. Unlike you, I actually slimmed down XP to a handful of files and didn't find a way to lower its memory usage. There's also no documentation and also the gurus that have been working on slim XP versions for years offer no solution. So I suggest to do what you claim is possible, otherwise your argument isn't worth the webspace it takes up...

I did use XP on machines with different amounts of memory. And when you talk about gurus, show your sources.

There is no XP-Vista war. If you actually read my statement then you would have seen that I tried XP as well as Win2003 before switching to a Win7 core. I'm neither a fanboy, nor biased. In fact, I tried what I wanted to achieve on about every kernel architecture. Win7 actually proved to be easily configurable and a lot more efficient due to its componentized nature. There's no reason to argue about that. The numbers speak for themselves.

Calling XP "antique" certainly sounds like you don't admire it as a great OS for modern times...But it's good that you don't want war either.

Oh and just for the sake of completeness: An XP-based PE doesn't support pagefiles contrary to a Win7-based one, lowering its performance significantly, but I'm sure you already knew that... Speaking of sub-optimal design choices lol...

I did not know this. Though using PE is a design choice too. And I still suspect that it could be better.

You're wrong on MinWin. It serves as core for Win7 and PE 3.0 and therefore is available, though not in its slim form. The sole purpose of MinWin is to offer a minimal kernel with a componentized nature which offers an easy way to expand functionality without the need for kernel updates and rewrites. Thanks to that, it's possible to actually expose the core without the need to disassemble libraries or sources. For XP that certainly would have been necessary, but not for Win 7/PE 3 and I'm on the best way to prove it. I already removed very essential services that couldn't be removed under XP due to them being low-level requirements for the system.

I did not say anything contrary to the fact that Vista / 7 use it.
Question of curiosity: what are the removed things, the ones that would have to stay on XP? I remember several fatalities that I got by removing some seemingly unnecessary things.

And yes the kernel is more feature rich, though not in the literal sense. The fact that it can be enriched with features so easily while offering a more modern approach to system management justifies this description.

Actually the thing you're probably talking about (the fact that it's layered) doesn't make it more modern, MS just had a lot of mess in the kernel that they cleaned up. Unless you mean something else....

[Skymmer]

Mexxi, your experiments are quite interesting. I have one suggestion but not sure if thats what you want. You can try v4.32 of 7-zip. The deal is that later versions have increased speed at the cost of increased memory consumption. Some later versions also include tweaks of LZMA to improve compression. I better illustrate this with small test conducted on pak0.pak file from Half-Life (Gordon says hello to Shepard ). Tested on XP Pro 32-bit with 32 bit versions and
-m0=LZMA:d=128m:lc=3:fb=64:mf=bt4. Results are in form of version\resulted size\memory consumption.

v4.32 = 98 225 268 = 1 256 MB (9.1% lower)
v9.07 = 97 799 032 = 1 382 MB

Not sure if it will be the same with x64 versions but I think it worth trying. By the way I just noticed that 32-bit version of 7-zip (at least 9.07) is limited to 128 MB dictionary. That's why I love FreeARC - it allowed me to use 153 MB dictionary with bt4 match finder. Another thing is that I usually have 1670 MB of free memory under XP. I rebooted into safe-comand-promt-mode and got 1863 MB of free mem but the maximum allowed dictionary for FreeARC remained same. Why? Something related to largest address space or how it called?

[Bulat Ziganshin]

freearc also allows to change hash size, while 7zip make it fixed. but it's not the option for Mexxi since he needs 64-bit version (and also freearc doesn't have bcj2)

[Cyan]

Would x64 version double the cost of pointers (64bits instead of 32bits) ?

If that is the case, then there would be a real disadvantage using x64 version, at least with less than 4GB memory.

[Bulat Ziganshin]

no. lzma stores offsets raher than pointers. btw, it also helps solving problem of sliding window maintenance that you have asked in other topic

[Cyan]

I agree. Pretty neat to instantly check distances. Although i've not released a PC-compression software based on this technique, I've used it for several unreleased ones (and for Saturn-based compressors).
As an interesting side effect, it can be used to lower memory usage (for example, with a 2-bytes relative offset, but that means distance cannot be more than 64KB, which is enough for some usages).
I guess that 4 bytes is even simpler, with little risk of 2 consecutive matches at more than 2GB distance...

Unfortunately, this doesn't work for my next candidate compressor, which is not based on LZ-search iterations. So i will have to settle with another method for maintenance of the ring-buffer...

[Mexxi]

What about MicroXP for example ?
I remember having a copy of it which would boot at 40MB.
It's not PE-based though.

I didn't try MicroXP, but started off with PicoXP. Problem is that there is no slim X64 XP and what I could see from MicroXP is that it is also x86. Great for systems with up to 3GB, but pretty much a waste for comps with more RAM. Still, the other problem at hand is that XP (and Win7/Vista/PE) start to cache files the more RAM is available. So even if you have an XP that runs on very small amounts of RAM, it will occupy hundreds of MB on systems with vast amounts of memory. For XP, slimming down doesn't really help getting around this problem. I made my own ultra-slim XP x64 and got the same amount of free RAM as when using the safe mode's command prompt and I also didn't find a way to turn off the caching.

For WinPE 3.0, slimming down actually does help to level this problem, even though PE still caches files. Unlike XP, there is a direct relation between the "slimness" of the system and the amount of memory it occupies.

Mexxi, your experiments are quite interesting. I have one suggestion but not sure if thats what you want. You can try v4.32 of 7-zip. The deal is that later versions have increased speed at the cost of increased memory consumption.

Thank you for this tip! I didn't know that! I just finished MicroPE for now, so I will make a comparison between 4.32 and 9.10 tomorrow

I rebooted into safe-comand-promt-mode and got 1863 MB of free mem but the maximum allowed dictionary for FreeARC remained same. Why? Something related to largest address space or how it called?

May be the amount isn't accurately displayed? You could try to close all unnecessary processes in safe mode. You could also give PicoXP a shot, although the amount of free RAM would most likely remain the same. Still, worth a try.

freearc also allows to change hash size, while 7zip make it fixed. but it's not the option for Mexxi since he needs 64-bit version (and also freearc doesn't have bcj2)

What kind of hash-size do you mean? I'd love to use a x64 version of FreeArc, so consider me one of the happy guys should you ever release one



As already mentioned above: I'm finished with MicroPE for now. The filesystem has a size of 45MB now spread among 140 files. The system boots from an HDD to avoid the usage of a Ramdisk. The system supports all major mainboard chipsets to ensure compatibility to basically all systems. It does not support any extras (CD-ROM, Mouse, PCMCIA, PCI, AGP, USB, FAT32 etc). It could be even slimmer if all the additional drivers were kicked out including keyboard support. The registry has a size of 1.29MB. I guess I'll slim that down to less than 1MB sooner or later.

As for the memory usage, I'll let a screenshot do the talking:




As you can see, MicroPE only occupies 111MB of memory, allowing 7z to use 356MB dictionaries with no pagefile involved.

I did use XP on machines with different amounts of memory. And when you talk about gurus, show your sources.

Funny that you think I would have to prove an absence of information about something that I already moved away from just because you want to argue about it. But just to save you a sleepless night: I'm talking about the creator of PicoXP and WinBuilder. You can even find my posts regarding this subject on several forums that deal with mini-sized operating systems, his being one of them

Calling XP "antique" certainly sounds like you don't admire it as a great OS for modern times...But it's good that you don't want war either.

No I don't admire it. It's a workhorse and not my "bestest friend" whom I have to defend to the point where the choice for it becomes irrational...

If I had to choose an operating system, I'd always go with the server ones. I went from Win98 to Win 2000 and loved it and switched to 2003 when it came out. XP is overloaded with crap compared to Server 2003. That's also why the latter has a better gaming performance, even though that's the last thing it was intended for. If I want a sleak and quick system I'll choose Server 2003 over any bloated XP any day.

And don't get me started on the "modern times". the NT5 technology is so old that it didn't support most of today's standard hardware when it came out. NT6 came out 4 years ago, so unless you consider some 5 year old hardware modern, there's no reason to call it that. XP modern? Thanks, but I prefer to use the DX10 and 11 capabilities of my hardware, let alone all the other goodies that XP lacks.

I did not know this. Though using PE is a design choice too. And I still suspect that it could be better.

Well, If you go for a slim operating system then it does make sense to use a 100MB sized PE instead of a 1.5GB sized XP, especially since PE's transportability makes it pretty much universal, while you'd have to manually install XP every time you wanted to use it on a different comp. The amount of work related to that is just not worth the effort. Question is also if XP could ever be so slimmed down as PE.

Question of curiosity: what are the removed things, the ones that would have to stay on XP?

This depends on your approach, needed drivers, the actual function you want to get out of the system etc. There's a slim XP called NCLI which is basically just a booting recovery console. It can't run external tools, but is also just 5MB in size. Other systems such as PicoXP come with a valid command line interpreter, yet lack essential files to run certain applications. For example it lacks the ole32.dlls to run 7z, so depending on what you want to do, you might need to satisfy certain dependencies. Other approaches use shells from other operating systems. For example someone slimmed down XP to less than 10MB by using the shell of an embedded XP instead of that of an original one. The embedded one seems to have less dependencies.

Anyway, there is no general answer. Your best bet is to use PicoXP as a basis and to go from there. I can give you one hint though: you either need to find the first file in the dependency chain, or the last one. Removing the first one will allow you to remove all other files that solely serve the master file. Removing the last file in the chain requires the process to be non-critical, but either way it can go wrong. It's basically just trial and error most of the time, but after a while you'll get a pretty good idea how the operating system works and how to slim it down without losing much functionality.

[Wladmir]

There's how to do it using bzip2 (dictionary size>900kb)?