Amazon AZ64 compression better than Zstd

[SolidComp]

Hi all – Amazon's new AZ64 codec looks interesting. They say that it compresses better than LZO and is 40% faster, and compresses better than Zstd and is 70% faster.

Does anyone know more about it and how it works? The only thing they say is that they leverage SIMD instructions, and they imply that it shines with small data in particular. It's used in their Redshift database, which I think is based on PostgreSQL.

The SIMD angle is interesting. I think there's still a lot of headroom with compression codecs because surprisingly few of them leverage SIMD very well. Are Zstd and brotli leveraging SIMD?

[Jyrki Alakuijala]

Are Zstd and brotli leveraging SIMD?

It looks like that compressor may be using the properties of specific data types and searches done into database tables -- i.e., possibly data is ordered. Compressing ordered uint32s is a different game than general purpose compression. Riegeli might be closer to that than zstd or brotli, but most likely not quite the same use case.

[Bulat Ziganshin]

At its core, the AZ64 algorithm compresses smaller groups of data values and uses single instruction, multiple data (SIMD) instructions for parallel processing. Use AZ64 to achieve significant storage savings and high performance for numeric, date, and time data types.

it's the so-called integer compression

[Jarek]

Indeed this is only for SMALLINT, INTEGER, BIGINT, DECIMAL, DATE, TIMESTAMP, TIMESTAMPTZ ... where LZ is not very useful - basic techniques for time series, separate data types should do a better job.
Here are some benchmarks: https://stackoverflow.com/questions/...at-performance


Recently zstd was added to ROOT they use in most of high energy physics experiments ( https://arxiv.org/pdf/2004.10531 ) - again LZ makes no sense for this kind of data, specialized compressor should do a better job ( https://arxiv.org/pdf/1511.00856 ).

[Lucas]

Ah the old comparing apples to oranges trick, never fails to impress .

[JamesB]

It's a reasonable comparison to make if you're running a database engine and want to justify the extra complexity of adding in type-custom compression engines over using the generic ones you already have available, but it's probably been taken out of context here.

[MegaByte]

For these types, you'd often use Delta coding or another specialized algorithm, so I wonder how this compares. That thread mentions that their own ANALYZE system currently doesn't test the new algorithm and they haven't given any numbers for the other methods.

[SolidComp]

I wonder if they're using Daniel Lemire's integer SIMD compression libraries.

[djubik]

I don't think it's apples to oranges, as JamesB said it shows better performance for some domains.

My observation as an outsider in the compression scene is that most focus seems to be on human words compression. All the main corpus have a big skew towards this type of data, rather than scientific data.

Scientific data is the most volumous and would benefit the most from compression IMO. I actually think the reason oodle has made such strides is they benchmark on game data which is basically scientific data.

​

[Lucas]

One thing these benchmarks often fail to mention is the class of compressor, integers compressors can be lossy if they sort records before delta coding, as in they cannot restore the original order of the data. At which point comparing it to a lossless codec is no longer fair, since there is no such thing as an "unsort" unless we're dealing with burrows wheeler transforms, which we aren't.
Hence why comparing the two can be an apples to oranges comparison.

But yes from a broad perspective compression is compression, as for being able to restore the exact original content, that's another question .