Activity Stream

Hello, Most recent papers on image compression use the Bjontegard metric to report average bitrate savings or PSNR/SSIM gains (BD-BR, BD-PSNR etc.). It works by finding the average difference between RD curves. Here's the original document: https://www.itu.int/wftp3/av-arch/video-site/0104_Aus/VCEG-M33.doc I am a little confused by the doc.The bitrate they show in the graphs at the end of the document seems to be in bits/sec. At least it is not bits per pixel (bpp) which is commonly used in RD curves. As per this site https://www.intopix.com/blogs/post/How-to-define-the-compression-rate-according-to-bpp-or-bps, converting from bpp to bits/s would require information of fps which might not be known(?). I want to know if it really matters if the bitrate is in bpp or bits/s? Or does the metric give correct values no matter which one uses? Here's a Matlab implementation that seems to be recommended by JPEG: https://fr.mathworks.com/matlabcentral/fileexchange/27798-bjontegaard-metric . I ran a few experiments and it seems bpp gives plausible results. Though a confirmation would be nice. Thanks!

I've wanted this with 286 and 386 laptops for decades. Do you feel it will live up to the promises from the press release? https://arstechnica.com/gadgets/2021/02/framework-startup-designed-a-thin-modular-repairable-13-inch-laptop/

Is there a way to make ANS FIFO? I'm assuming it's either impossible or no one has tried it, since we're stuck with a LIFO fast coder since 2014. If so, what's the reason making it impossible? ANS basically maps numbers to bits and bigger state means more bits. This is why the state is increasing as we encode. Normalization helps us with infinite precision. Can't we just reverse this process? Instead of starting at a low state and increasing as we go, start at some high state and decrease it. Normalization would make sure we don't get to negative states - it will increase the state. In ANS we're asking the question "What is the next state that encodes all the bits we have and one more bit?". Now we can ask "What is the closest previous state that encoded the information in the current state, preceded with the new bit?" I'm imagining some hypothetical (for the binary alphabet): Let's say we want to encode 00110100 First symbol is 0 x_0 = 16 encodes 000100 The last symbol of this sequence is 0 and it matches the end of this code. We don't switch states. The next symbol we read is 0. The closest smaller state that matches is: x_1 = 13 encodes 00100 The next symbol is 1. The closest smaller state that matches 001 is: x_2 = 5 encodes 001 The next symbol is 1. But we can't encode anything - our state is too low. So we normalize - we output some bits and go to let's say state x_3 = 40 x_3 = 40 encodes 0010010001001 This matches our symbol 1. We read the next symbol to be 0. x_4 = 38 encodes 0100100101 and matches "01" And so on... That's the basic idea. I'm trying to understand deeper how ANS works and how exactly it relates to range coding. Notice the string encoded by our state is the reverse of what we read. We're still reading in the correct direction but the state encodes in the opposite - we're adding data to the most significant bits of the state, not the least significant. That's just me brainstorming. Has anyone else tried this? I would find great pleasure in a proof it doesn't work as well. Thanks!

I started to work on codecs bindings for the new CELS framework with what I thought the simplest one - LZ4. But it turned out to be more complex work than I expected. My main surce of information is lz4 manual, library source code and examples. I have not found any tutorial-style guides for using the library, and the documentaton provided is centered around individual functions rather than usage scenarios. My goal is to avoid mallocs inside the library (providing all memory buffers by myself), and reusing compression contexts between operations in order to maximize perfromance. I will put my questions in bold, mixed with commentaries describing my setup. Q1: It seems that decompression context is so small and quickly initialized, that keeping and reusing it between operations is meaningless? For compression of the single buffer, I use sequence of `malloc(LZ4_sizeofState()) + LZ4_compress_fast_extState() + free()` Q2: Does LZ4_compress_fast_extState guranteed to never call malloc inside (in current 1.9.3 version)? It seems that the next step to improve perfromance is to have malloc + LZ4_compress_fast_extState + LZ4_compress_fast_extState_fastReset (many times) + free? Q3: Am I correct that I should call LZ4_compress_fast_extState with freshly allocated state object, but on the following operations using the same state object, I can use LZ4_compress_fast_extState_fastReset to improve perfomance? Q4: In lz4.c:819 I see that the fast reset is actually used only for input blocks less than 4 KB. Why it don't used for larger blocks? Now, going to the stream compression. Q5: The LZ4_resetStream_fast docs stated that despite having same size, context used for single-shot compression, shouldn't be reused for the stream compression and vice versa. Is it true or is it my misunderstanding? The proper approach to maximize speed by reusing context with streaming compression looks like: 1. malloc + LZ4_initStream 2. call LZ4_compress_fast_continue multiple times, processing single stream 3. LZ4_resetStream_fast 4. call LZ4_compress_fast_continue multiple times, processing single stream repeat steps 3 and 4 again for each subsequent stream and finally 5. free the memory Q6: Is it correct approach to ensure max speed? Q7: LZ4_decompress_safe_continue documentation mentions three decompression scenarios, and LZ4_compress_fast_continue mentions various scenarios as well. Is any of them has speed advantage? In particular, I have implemented double-buffering with adjancent buffers (as implemented in blockStreaming_doubleBuffer.c although this contradicts Note 3 on LZ4_compress_fast_continue) - is it correct and can I improve the speed by using other approach to compression/decompression? By default I use two buffers of >=64 KB, but I also interested to know answer for smaller buffers. PS: at the end of day, it seems that internal LZ4 API is already well-thought for my needs, not so much for the manual.

Hi all – I just realized that I don't know anything substantial about encryption. Am I correct in assuming that we can't generally compress encrypted data? I assume that effective encryption produces random-looking data to the naive sentient being, therefore the sequence of steps has to be compression first, then encryption, then decryption, then decompression right? I've heard a bit about homomorphic encryption, which I think means a type of encryption that enables one to extract useful data from an encrypted dataset without decrypting the dataset. At that level of description, this can mean a few different things. Homomorphic encryption might mean that data in its informative form exists at different levels of abstraction or something, some of which is not actually encrypted (?), or there might be something about the nature of the search algorithms – how you find what you're looking for without decrypting... Or it could be much more interesting than this. But the general thrust of homomorphic encryption hints at the possibility of compressing encrypted data, if the surfaced information was somehow useful to a compressor, could be acted on by a compressor, but I don't know enough about it. Has anyone worked on the theory around this? Thanks.

libsais is my new library for fast (see Benchmarks on GitHub) linear time suffix array and Burrows-Wheeler transform construction based on induced sorting (same algorithm as in sais-lite by Yuta Mori). The algorithms runs in a linear time (and outperforms divsufsort) using typically only ~12KB of extra memory (with 2n bytes as absolute worst-case extra working space). Source code and Benchmarks is available at: https://github.com/IlyaGrebnov/libsais

Fazip uses TTA but it doesn't work for compound content, like a .wav somewhere inside a TAR archive. Precomp was supposed to include wavpack like a year ago but nothing has happened yet and it doesn't seem like it's going to. So, do you know of something like this, that detects and compress audio chunks and makes a copy of the rest of the data? Thanks in advance!

I assume many here are into file compression for the greater good more than money. But up to what ratio as the poll shows. Also to the extent how far you are willing to go to screw another as well if someone hired you to benefit from the idea. Kind of like how Xerox got screwed over by Apple, or Apple and others got screwed by MS, or the original created of FB got screwed over, The McDonald brothers got screwed over by McDonald, Nikola Tesla screwed over by JP Morgan, Etc. like most big corporations. So what would make you different it seems. Think about it for a few days before you answer.