LzTurbo: The fastest now more faster

[ZakirSheikh]

@Anyone Who Can Help
How can I make Lzturbo Show progress in inno...
How can i use lzturbo stdin stdout..Thanks

[Shelwien]

See http://nishi.dreamhosters.com/u/mpzapi_v1b.rar (hookapi.inc)

[Chirantan]

I've tried many commands for lzt with stdin stdout in the arc.ini but i just couldnt find it. It'd be great if someone could send me the lzt arc.ini commands with stdout stdin support.

[hunman]

Stdout in windows binary is buggy!
The author told me.

[Sportman]

I tried Lzturbo 1.2 with enwik10 but decompress file is only 1,073,741,824 bytes.

227,112,649 bytes, 2,342.967 sec. - 10.514 sec., lzturbo -49 -p1 (v1.2)

With compression I see a row:
VirtualAlloc failed

No other errors.

OS page file is disabled.

[CompressMaster]

OS page file is disabled.

That´s the problem. Lzturbo 1.2 probably needs more RAM than you have available. Enable pagefile and see if it helps. Or upgrade your RAM. But compression will be slooooow, though.

[Sportman]

Lzturbo 1.2 probably needs more RAM than you have available.

There is 32GB and 30GB+ memory available.

[dnd]

Try restarting your system.

[Sportman]

Try restarting your system.

It didn't help, but I found the problem.

It looks like that Lzturbo 1.2 (-49) use system TEMP path to write something while compressing, but this TEMP path is at an other much slower storage device and had less then 10GB free space. I increased the temp space to 10GB+ (still at the slow storage device) and now it work without errors:

2,028,356,329 bytes, 8,212.226 sec. - 33.829 sec., lzturbo -49 -p0 (v1.2)

Lzturbo 1.2 -32 option hadn't this problem.

[dnd]

LzTurbo is not using any temporary files.
The issue has to do with "VirtualAlloc" on windows.
It's possible that the Virtual Memory Size is set to 16GB or lower on your system.
If you want, you can change the settings in "Computer Properties/Advanced System Settings/Performance/Advanced/Virtual Memory"
It can be other solution possible:https://github.com/nicehash/NiceHash...ved/issues/652

Please make sure that decompression is done from the faster drive.

[Sportman]

It's possible that the Virtual Memory Size is set to 16GB or lower on your system.

Page file was in all tests disabled because I wanted an error instead of using a page file (memory swapping).

Strange enough Lzturbo 1.2 -49 only use 2,089,672KB memory during compression and it looks like the same for decompression, so no swapping expected.

Maybe Windows use a lot of memory for caching (when reading a 10GB file and writing a 2GB+ file) and when the page file is disabled use the temp folder. I need to research this more.

For now I enabled the page file and moved the temp folder both at/to the fast fast storage device.

Please make sure that decompression is done from the faster drive.

Compression and decompression are always done at the same fast storage device.

Update 1:
I tested with a 30GB fixed page file and no temp file space and got the same "VirtualAlloc failed" error with Lzturbo 1.2 by options -19, -29, -39 and -49 only.

Update 2:
I tested with a 30GB fixed page file and 32GB temp file space and got the same "VirtualAlloc failed" error with Lzturbo 1.2 by options -19, -29, -39 and -49 only.

[dnd]

Strange enough Lzturbo 1.2 -49 only use 2,089,672KB memory

LzTurbo is using a custom memory mapped allocator (based on VirtualAlloc).
It can use up to ~16-17 GB for 1GB block (option -b1000).
You must look at the virtual memory used.