Hi all – Is there a fundamental or theoretical reason why there isn't a better than gzip compressor that is as light on memory and CPU? According to the LTCB, gzip only uses 1.6 MB of RAM.
If you had to use at most the resources used by gzip (memory and CPU), could you build a better compressor? Is there an insurmountable problem of computational complexity or something? Like if you use context modeling or FSE or ANS...
What if we had full use of the modern CPU instruction set, so things like carryless multiplication, CRC32, AVX2, Bit Manipulation Instructions, SSE 4.2 string matching, etc.? And whatever their counterparts are on an ARMV8 server CPU.
(Note that the "gzip" that Mahoney uses on the LTCB is an extremely obsolete Windows application that no longer exists, but may be based on a predecessor to the current GNU gzip. His app is from 2006 or so. I'm not clear on what the implications are for memory use and CPU, but the modern zlib gzipper will be faster, and might compress better. The Cloudflare and libdeflate implementations of gzip will be faster still, not to mention SLZ.)
This article is interesting in that Expedia thought LZMA2 was the winner of their testing, but they discovered that with on-the-fly compression it used too much resources and gzip was actually the winner...