How important is compression?

[Trench]

Everyone talks about how to compress but why compress is not mentioned much or maybe I missed it.

What would be the benefits of a better compression? What would be a detriment of better compression? Obviously not all compression are equal and the better you compress the better results. Hard drive space has increased more than file compression from the 1980.

I remember hard drives under 30MB and now over 30TB while compression did not gain that much in percentage maybe due to lack of financial gains. If file compression was better HD technology would not have increase as fast.

If you make a file be compressed 10% how much of a benefit would it be?
What if 100% or 1000% ? does it matter and if so in what way?

Is this a game for most how some kind of indicated? Or some use this as a stepping stone for a better job to put on resume? Can it benefit or save the world? Can it make CPU faster? Or is it something like to just to store all movies in your pocket?

Everyone has a reason so what are yours since everyone has a different perspective.

To get the ball rolling.
A 10% gain might help with maybe 1% of ewaste, less pollution, less files being lost. Can CPU also go faster?
The bad thing about file compression is it can hurt economy since if no one buys an updated phone with more space or HD then less sales, stock market less, less taxes.

A 1000% gain might help with maybe 10%?? My percentage can obviously be wrong but its a guess. Would higher be better to make a big change or just nice to have?

What is your perspective the better a file is compressed or not??

[Shelwien]

> How important is compression?

Very, but there's a lot of duplicated terminology.
Any prediction, generation, recognition, optimization are closely related to compression,
basically any AI/ML/NN too.

> Everyone talks about how to compress but why compress is not mentioned much or maybe I missed it.

To save storage space or communication bandwidth, which have significant costs.

> What would be the benefits of a better compression?

More saved money.

> What would be a detriment of better compression?

Extra latency on data access, maybe obscure bugs/exploits.

> Obviously not all compression are equal and the better you compress the better results.

Actually there's no one clear metric of "compression goodness".
Different use cases require different solutions.

> Hard drive space has increased more than file compression from the 1980.

The technology at that time was simply that rough.

> I remember hard drives under 30MB and now over 30TB while compression did
> not gain that much in percentage maybe due to lack of financial gains.

Its not about money. Random compression methods don't exist,
and lossless compression improvements are quite hard to discover and implement.

> If file compression was better HD technology would not have increase as fast.

Hardware technology improvements would stop soon enough, because of physical limits...
in fact storage now is only ~2x cheaper than it was in 2011,
and the sequence is logarithmic:
https://www.backblaze.com/blog/wp-co...er-gb-2017.jpg

> If you make a file be compressed 10% how much of a benefit would it be?

Is that "to 10%" or "by 10%"?
In any case, if file size is reduced by 10%, then 10% of its storage cost is saved,
it directly maps to money: without compression you needed 10 SSDs, now you need 9.

> Is this a game for most how some kind of indicated?

"For most" of whom? Some people like it as a hobby, some have related jobs,
some like competitions and benchmarks.

> Can it benefit or save the world?

Well, its more likely to destroy it... we'd keep increasing density of randomness
(that is, compressed/encrypted data) until universe breaks :)

> Can it make CPU faster?

It already does - branch prediction in recent cpus is pretty similar to CM.

> Or is it something like to just to store all movies in your pocket?

Newer movies would just have higher resolution, so they'd never all fit in any case :)

> What is your perspective the better a file is compressed or not??

Actually its better if it isn't - you won't lose it from a single-bit error then.
But at this point its hard to fully avoid it - for example, HDDs always use some entropy
coding to store the data, it won't work otherwise:
https://en.wikipedia.org/wiki/Run_le...for_RLL_coding
and SSD sometimes have integrated LZ compression.

[Trench]

Thanks But for so many to not give their reasons it shows it is not that important, especially for the person that is reading this that did not reply. lol


Your reply is interesting. You say its very important but you mainly show money are the important core cause a few times. If that is the case then it is not important in a way in how may see it. Yet I assume you want it free to some degree despite it would save people money.


As stated in my other post HD manufacturers don't make that much money. But if its to save money then how much money will people pay to save money. But only if it gets significant results. Some people pay for winrar which the gain is not better and even worse than the free which doesn't justify but they are magically afloat.


When I said if its a game I kind of find a hobby and competition a bit of a game to do it for fun.


Well I was hoping for more than money reason but at least you stated 1 good thing besides money which is a faster cpu. I guess everyone says the same which I am a bit disappointed. I telling people about the internet before anyone went online but it was hard to describe something that has not been around but I did say it can do anything of everything. Hard to describe things no one experienced yet. The same thing with file compression we only know what we experience since creativity is not as easy as many may assume, which is why I keep saying a programmer can not discover a far better file compression. They have the ability to write the code but not create it.

It reminds me a computer science professor / Entrepreneur David Gelernter says.
"The thing I don’t look for in a developer is a degree in computer science." He beat Apple in court to win his patent case which he doesn't go with free mentality since as it shows corporations use other that have good intentions to gain from their hard work.
It is not an insult to programmer in what he said but a limitation of mastering another field which would be an insult to the other other field of work if anyone can do it. Jack of all trade is a mater of nothing as the saying goes. It's not a discouragement but and encouragement how everyone should focus their skills to be more efficient to get results. Their are the few exception but that does not make the rule.



On a side note this free mentality hurts people that want things for free when corporations benefit from it to crush the competition that make free things so that they can sell their product that doesn't have that much passion and innovation gets stuck. People with passion have to not give their passion away and should be as valuable as their desire. Just my view.

Maybe if I ask people in another field of work like philosophy, math, theology I might get a different answer.

[Shelwien]

> But for so many to not give their reasons it shows it is not that important,

More like nobody here cares about the question - I suppose you can try asking
it on Quora instead: https://www.quora.com/profile/Matt-Mahoney-2

Also your posts are kind of hard to read.

> You say its very important but you mainly show
> money are the important core cause a few times.

I don't understand why'd you expect some kind of ideology for people to work on data compression.
Yes, with current tech we can avoid using any compression algorithms at all,
so its certainly not something essential, just a useful option.

On other hand, in cases without hardware solutions (eg. we want to add some new features
to device's firmware, but can't replace the firmware flash chip which has a limited size)
we might have to look for software solutions.

> Yet I assume you want it free to some degree despite it would save people money.

There's a significant difference between a compression algorithm and
a product based on it, which is actually designed to save money in some specific use cases.

> As stated in my other post HD manufacturers don't make that much money.

Well, you'd have to look at media content providers instead (netflix, youtube etc).
Online video won't be possible without significant effort invested into
video codec development (which is also compression).

> Some people pay for winrar which the gain is not better and even worse than
> the free which doesn't justify but they are magically afloat.

There's some weird logic in play, but its actually easier for corporations
to buy commercial software for common tasks rather than use free equivalents.

> Well I was hoping for more than money reason

There're plenty of cases where compression is useful
(eg. improvement of encryption security) or is the only solution to some
technical problem (sending a picture in a twitter post).
Soon enough (maybe in 10-20 years) it would also be the only way
to improve storage density (once switching to a higher density tech
becomes too expensive).
But since there're technical workarounds for using compression in most cases,
money can be said to be the only real reason.

> I guess everyone says the same which I am a bit disappointed.

Well, your brain does a lot of data compression and translation
between different coding methods (google "data compression human brain"),
so it can be said that its unavoidable?
On other hand, you don't have to know the implementation details
for your brain to work.

> which is why I keep saying a programmer can not discover a far better file
> compression. They have the ability to write the code but not create it.

That's not how it works at all.

1) Programming is not a mechanical task like you see it.
In most cases there's no one best solution which just requires
"writing the code".

2) In mathematical terms, compression is a solved problem
(find a shortest set of cpu instructions that outputs given data),
but practically that solution is impossible to use.
So its up to programmers to find ways to make efficient algorithms
for available hardware - there's plenty of creativity involved,
in facts that's one of the main reasons as to why compression is interesting.

3) Everything is up to the volume of external information.
In theory, we can replace any file with a reference to some global storage system -
that's already near-infinite compression in practice, for most users.
Again, the problem is purely technical - we need a solution based on cheap resources,
not just something theoretical.
This means that to compress a file we need to enumerate all versions of its content
compatible with additional known information, then encode the index of one specific
version which matches the file data.
If you'd ever try using combinatorics to compute the numbers of content versions
in some simple cases, you'd see that compression ratio is very limited
when the volume of external information is low (dictionaries etc) -
log2 of the number of possible data versions would be close to original size of the file.