No hope for better compression even if possible

[Trench]

Imagine 1000 movies ore more fitting in a 1gb flash drive. I am not saying it possible but imagine.

1 What would be the negative issues with that? None? Are you sure?
2 What do you hope to achieve? Money? for how long?
I assume a small percentage gain a year to please everyone and not rock the boat.

Even if you are able to compress every file at 99% logical it won't matter for a few reasons but 2 main things are.

A Can't compete against free versions to sell it
1 Companies won't most likely buy it since again you have to go against free. Which people are not waiting in line to buy winrar.
2 Can be reversed engineered and other multiple alterations can change it which makes it risky.
3 Current devices are enough to transfer files for a while now.
4 money easy come and easy go, due to suits against you to mess up companies without warning and even if warned not allowed.
and many more

B Can't release it for free since it would mess up the market and nations
1 It would cause more people being able to download entire libraries, programs in an instant which would be like wilfe fire which can not be stopped easily which companies would hare that.
2 The stock market for many companies would fall which would have a market panic which goes against fiduciary laws to some degree. Countless of companies like of storage devices, politicians, online storage companies, phone companies, internet providers,
3 it will put many out of the job which will also cause many angry people and see it as an international threat to mess with trade.
4 If a company buys it it will be a target against many and no want to be seen as the bad guy
and many more

Anyone remember this guy from the 1990's about which said he can have an entire movie on floppy drive and was going to sell it. The day before he sold it he just mysteriously died of a heart attack. His files could not be found.

You against millions of people, plenty of rich and powerful peoples lively hood which they will feel like they are threatened and one out of the million might be nuts. Since as the saying goes "hurt people hurt people", or in other words people that have been hurt will hurt others.

More money to be made to find a cancer cure than the cure.
More money to be made for the effort of world peace than world peace.
And the list goes on.

Some say even free energy is against the law which would destabilize nations, tax revenue, billions lost, a "national security" threat, etc.
Even Edison and his financiers tried everything to stop Tesla and cut off funding. Most have not heard of Tesla before the internet.
Money makes the world go round, and to stop that is going against the world it seems. And the masses wont be with the smart people since they do as the people on top say since they present security and money for them to live.

People are not morally ready and can't handle such great difference.

It feels like if one rock the boat and the boat will rock you off the boat. Since this seems like a logic issue.

OR maybe I can be wrong. So tell me how what I just said is wrong and what guarantee you have if you do it?

[michael maniscalco]

I and almost every long time user of this board, I would wager, are here because we love what we do. There is no money, fame, nor glory involved. It is, for the most part, just another example of the "hacker ethic". We follow our passion, work our craft, and trust that it will be appreciated or beneficial in some way.

Those who come here for money, fame, etc are soon forgotten. Those who remain are likely the finest kinds of people.

- Michael

[Shelwien]

We can easily extrapolate the results of sudden compression breakthrough from previous similar cases.
In 199x people were busy collecting plaintext books and floppy-sized games.
Then hardware improved and it became possible to quickly download whole collections with millions of books at once.
The result? Book downloading is not a mainstream topic anymore, also OCR'ers started posting books as 10-100M pdfs
instead of <1M plaintext.

Also now the main content type is video, it takes 90%+ of traffic and storage.
But its lossy, so better compression methods would just lead to increase in quality until same balance is reached.

P.S. I'd be moving this and your other thread to "random compression".

[schnaader]

My vote is yes, but I guess you won't like my explanation. It is possible to compress even millions of movies below 1 GB, yes. And there are many services and companies that do so. One of the currently best known is YouTube. They take the movie, compress it to a fraction of its size (roundabout ratio is 1:100 from uncompressed -> video codec). After that, they make the movie accessible to users by assigning an URL to it that is only about 30 bytes (for example: youtube.com/watch?v=NN75im_us4k) or even shorter (youtube.be short URLs).

Yeah, I know. This is not what you meant. But this is the problem with all those random compression stuff here - you just don't ask the right questions. It would be "Can I losslessly compress all possible files by 99% and get the original file back afterwards by decompression?" And the answer to that is a straightforward and easy "No", backed by the pigeonhole principle. Note that both your post and the poll don't even mention those both words that are essential here.

Now let's address some other things from your post:

Anyone remember this guy from the 1990's about which said he can have an entire movie on floppy drive and was going to sell it. The day before he sold it he just mysteriously died of a heart attack.

I guess you mean Jan Sloot. Well, let me explain my theory: 1. It was a hoax. 2. He died. Two totally unrelated things. As a matter of fact, everyone will die sometime regardless of his inventions or things he discovered. And it's not that I deny that people might sometimes get killed by others, it's just that I don't believe in the claims he made. Can I prove my theory? No. Should I? I don't think so, there are better things to do for me than hunting some mysterious stories.

Similar recent story is the death of Mad Mike. I don't think his death implies that the earth is flat or that he got killed by someone to hide this. And this is because some basic science can show that the earth is in fact not flat.

Some say even free energy is against the law which would destabilize nations, tax revenue, billions lost, a "national security" threat, etc.

Same thing here: What if those "free energy" concepts just don't work? The laws of thermodynamics say so and I think they're valid, so I don't care "researching" "free energy". Of course you can go that way, deny science facts and go on a hunt for some magical things using all those pseudo-science buzzwords. But don't expect others to follow your way, especially in a forum filled with data compression experts.

Don't get me wrong, you should think out of the box, that is was made me develop Precomp - which does a good job compressing most of the files you'll throw at it to about 30-50% less than pure LZMA2. In a way, it was me denying the statement that "compressed files can't be compressed further". Well, again, watch your wording. Compressed files can be compressed further if you decompress them first. Also, compressing them further losslessly does work, but involves some advanced techniques and work because usually, compression is not bijective.
And now we can proceed to your "Can't compete, can't release, evil companies" part. Did I get killed because I developed Precomp? No, but of course you could say that Precomp isn't one of the revolutionary and foundation-shattering random compression things you're talking about. Did people try to "steal" my idea? Well, in the beginning there was some tool using Precomp without my permission, but it quickly went away. Also, a Google employee had a failed attempt with Grittibanzli. Back when Precomp was closed source, AntiZ was a more successful open source attempt. Dirk Steinke made Preflate and PowerArchiver integrates a commercial solution. And I even got a job at Ocarina Networks because of my work in the field, so yeah, I indirectly made money with the ideas from Precomp. But did I sell my soul to some evil company and did they steal all my ideas? No.
All this "competition" isn't threatening me, quite the contrary - I'm happy for each one of them to exist. First, because the recompression idea is spread and second, because smart people do hard work to push and promote the field of (re)compression. So, why doesn't recompression get more mainstream very quickly, isn't this evil and unfair? No. It's hard work to replace existing solutions (in a safe and robust way that fulfills different requirements), even if your core techniques are clearly superior. Same for FLIF->FUIF->JPEG XL, for example. Smart people doing hard work. For me, those are the people I look up to and not some random compression techno-babble kiddies discussing bullshit. Sorry for addressing the elephant in the room that harsh, but that's the truth I believe in.

[Trench]

Compression with decompression is implied. since their is no point of compression without decompression.

But if you need a another example to have the entire internet with lossless videos, music, images, programs, etc in 1gb or 1mb.
I am not saying if its possible to achieve I am saying if it was possible would it be allowed when so many people of power and influence will get upset over it that have invested Billion in other forms directly or indirectly.


The examples of people may or may not be true but the point is the reason behind not allowing it. If their enough evidence even to 1% that someone of power can prevent it?


If you invest 1 billion dollars in something today and tomorrow someone comes up with something to make your investment useless would you be happy that you lost your money? Would nations be happy if they also lost money and revenue? obviously no. People lives are risk to a degree to have built up an infrastructure that sustains million? Won't at least 1% of someone that loses to be enraged do something? Obviously not right but as the saying goes when something could go wrong can go wrong.


You may be content if someone takes form you but another might not be that you are taking their lively hood. Not everyone is the same.


Existing solutions can be replaced overnight in a way if the difference is that great. Look at the stock market now how closing the boarder to china billions were lost but that is temporary and many people are not happy. It will bounce back but will take time. But if it was permanent people can lose their minds.


You can't speak for someone else when they have their own experiences which validates the contrary. And I am not talking about 2nd hand experience but 1st hand. Everything is political from the food you eat, the cloths you have, the news you are presented, to the science discoveries, etc.

Most voted that it would not be allowed. I would prefer to be wrong.
If what I say is true at 1% how would you deal with it than dismiss it? Can it be?
How many people are willing to take that risk? And what guarantee does one have?
I could be wrong and would like to be wrong, but would like something more convincing.

[Piotr Tarsa]

Regarding the conspiracy theory from first post where "it would mess up the market and nations". There are two possibilities:
1) There is indeed a conspiracy. In such case random posts like this have about 0% chance of ending the conspiracy. Instead of fantasizing about some secret revolutionary algorithm's capabilities, you need to find out how and who hides it and we have no idea even where to start.
2) There is no conspiracy. In such case random posts like this only delay the search for better compressors as knowledgeable people have to read through someone's delusions instead of coming up with constructive ideas.

In either case it's actually counter-productive to spam this forum with thread like this one.

[Self_Recursive_Data]

Poll: Can a 99% file compression of any file be allowed?



yes

If your file has a trillion trillion 'A's in it, all you have to store is a small number as bits ex. just 30 bits can store 1,048,576,000 'A's! That's 279,620,266 bits stored as just 30 bits.

And the more data you compress the more you can compress it - this goes for wikipedia text.

[zyzzle]

Youtube is the absolute bane of video compression. It is a scourage, one of the most irresponsible and greedy companies around. They compress EVERYTHING to hell, all lossy compression of course, and the site is does a great disservice to those who appreciate good-quality (I mean high-bitrate, well encoded) movies and films. They're not archivists, they're destroyers of quality. The average person used them because "it's convenient" and doesn't know or care that any (already lossy compressed!) films they upload undergo a 2nd generation of "digital loss" and a ludicrious lowering of paltry video bitrate) before appearing on the site. They only support a *maximum* of 160 kbps lossy (AAC, codec, I believe) audio, which is absurd in 2020. And again, any compressed audio you upload undergoes a further generation of re-compression, resulting in a tinny, artifact-ridden mess of audio and video.

So, wishing to get 100 movies on a 1GB memory stick is an absolutely terrible idea which would do far more harm than any good.

[Trench]

Piotr Tarsa
That's not a convincing argument. If more people voted yes with a reasonable argument then I would have asked the moderator to take this post down. But if it upsets the few then despite the masses say no then fine ill ask the moderator to take it down in a day despite no can can give valid argument against each point.


Self_Recursive_Data
funny but that's not the point.


zyzzle
That is not the point but youtube works since no other site can work as well which is why so many failed and even google video failed which was made by google programmers and the same with google plus. Obviously plenty of room for improvement on youtube but people at google are not smart enough in creativity despite very smart programmers which are generally not creative people like a Steve Jobs. Steve Jobs is like Steven King as a programmer is to a translator. You can not have a translator make great novels as likely, and Steve Wozniak cant be the top creative designer and Steve Jobs the head programmer which people seem to forget that.

Everyone has to know their role which most don't and think they are superman which is an insult to every other profession ans very none humble. People can argue the point but their is even more evidence again it. I dealt with plenty of high end programmers which work for the top companies but their arrogance brought their own personal company down since they think they know it. Many programmers but not all think they are jack of all trades master of everything, which does not exist.

Companies do well when they hire the skill of all fields and not have a programmer do it all. Communists did that to to make a doctor be the farmer and the farmer be the doctor and we are creeping up to that mentality again which does not work well. A kitchen knife will always beat a Swiss army knife in the kitchen when doing professional cooking.

[Piotr Tarsa]

Trench:
I got into lossless compression scene because:
1) You don't need any extra money, power, people, special permits, access to restricted technology, etc What you need to develop lossless compression algorithms is just an ordinary computer. All compression programs can scale down their requirements so you don't necessarily need e.g. a 500 TB RAM machine to run a neural network compressor. lstm-compress uses less than 20 megabytes of RAM and still does the job.
2) You can objectively compete because lossless compression, is well, lossless. That means there's no room for imperfection (loss of details), subjective judgment or so. Your program has to compress and then perfectly decompress. Only then it's deemed working.
3) There's not much required for creating a compressor except implementing your more or less revolutionary idea. A simple compressor that compresses one file at once is equally as interesting to compression community as a super complex archiver that does many extra things in addition to compression if the compression efficiency is very high.

Because of above, comparisons to world peace, curing cancer or building power grid are flawed. World peace needs significant effort of multiple people. Same for other two. You can't just build power grid. You need permit from government. You also can't claim you've cured cancer without expensive experiments in expensive laboratories that prove your statement and also others need to be able to replicate. OTOH for starting a great software project you only need an ordinary computer. Look at Linux kernel. It's a massive undertaking started by a single guy that soon was supported by bigger and bigger community. Killing Linux in its infancy could mean a lot more money to Microsoft, Apple, Unix variants vendors, etc but that didn't happen.

Summing up, lossless compression scene is a really bad target for conspiracies.

[well]

Killing Linux in its infancy could mean a lot more money to Microsoft, Apple, Unix variants vendors, etc but that didn't happen.

Summing up, lossless compression scene is a really bad target for conspiracies.

because, money is just a game, all casinos may be hacked - just double stakes all times communists are winners!
i has a dilemma: one way is to hack bitcoin - my fortune is around 40 billions dollars in the best case, other way is keep the silence... what to do?

[Trench]

Piotr Tarsa
You are not directing your comments to me but to most that outnumber your yes vote.

If you were a billion dollar storage company how happy would you be? would you take legal action if you are responsible for so many jobs and shareholders?

I generally agree with some things you said , but that is still no assurance or solution. Or maybe you are still not understanding what I am saying since you disagree but you still are not giving solid answers.

Even, silly politicians have delusions how one side says the other worked with another gov and the other side says the others are very corrupt. one conspiracy is true no matter what side you choose since one side is conspiring which means to talk behind closed doors.

if you are right great but if you are wrong that is the risk since how will you fix it when its too late? You are gambling on your assumptions without facts dismissing the other point fully. But if thats all you can explain then ok but the majority wont agree with you in this forum and if someone fos have a better idea they wont talk. I am not to blame since they had that belief before. All you had to do is convince people which I am trying to say that is not good enough of a reason.

Ill ask the mod to erase this post then since I have not seen any solutions so far.

[Piotr Tarsa]

If I had a compressor that "allows 99% file compression of any file" while being too scared to announce it publicly, I would use secret channels like Tor network to publish fully working source code anonymously to several places around the globe. Others would continue the job of integrating that source code with other programs and therefore humanity would fully benefit from my invention. OTOH if my intention was to make money on that algorithm then maybe some conspiracy would happen. But in that case (not conspiracy, but selling my program) I could jack up the price very high to maximize my profits and people wouldn't see huge benefits from my invention. Then my company could be considered a monopoly and government would step in and so on...

[michael maniscalco]

Jesus wept.

[suryakandau@yahoo.co.id]

Imagine 1000 movies ore more fitting in a 1gb flash drive. I am not saying it possible but imagine.

1 What would be the negative issues with that? None? Are you sure?
2 What do you hope to achieve? Money? for how long?
I assume a small percentage gain a year to please everyone and not rock the boat.

Even if you are able to compress every file at 99% logical it won't matter for a few reasons but 2 main things are.

A Can't compete against free versions to sell it
1 Companies won't most likely buy it since again you have to go against free. Which people are not waiting in line to buy winrar.
2 Can be reversed engineered and other multiple alterations can change it which makes it risky.
3 Current devices are enough to transfer files for a while now.
4 money easy come and easy go, due to suits against you to mess up companies without warning and even if warned not allowed.
and many more

B Can't release it for free since it would mess up the market and nations
1 It would cause more people being able to download entire libraries, programs in an instant which would be like wilfe fire which can not be stopped easily which companies would hare that.
2 The stock market for many companies would fall which would have a market panic which goes against fiduciary laws to some degree. Countless of companies like of storage devices, politicians, online storage companies, phone companies, internet providers,
3 it will put many out of the job which will also cause many angry people and see it as an international threat to mess with trade.
4 If a company buys it it will be a target against many and no want to be seen as the bad guy
and many more

Anyone remember this guy from the 1990's about which said he can have an entire movie on floppy drive and was going to sell it. The day before he sold it he just mysteriously died of a heart attack. His files could not be found.

You against millions of people, plenty of rich and powerful peoples lively hood which they will feel like they are threatened and one out of the million might be nuts. Since as the saying goes "hurt people hurt people", or in other words people that have been hurt will hurt others.

More money to be made to find a cancer cure than the cure.
More money to be made for the effort of world peace than world peace.
And the list goes on.

Some say even free energy is against the law which would destabilize nations, tax revenue, billions lost, a "national security" threat, etc.
Even Edison and his financiers tried everything to stop Tesla and cut off funding. Most have not heard of Tesla before the internet.
Money makes the world go round, and to stop that is going against the world it seems. And the masses wont be with the smart people since they do as the people on top say since they present security and money for them to live.

People are not morally ready and can't handle such great difference.

It feels like if one rock the boat and the boat will rock you off the boat. Since this seems like a logic issue.

OR maybe I can be wrong. So tell me how what I just said is wrong and what guarantee you have if you do it?

Just to the point what you mean n want...

[CompressMaster]

Can a 99% file compression of any file be allowed?

In short, yes. It´s possible to compress even already compressed data today - with prior recompression of course (for example schnaader´s PRECOMP). Otherwise, they´re random - i.e. no useful patterns here - impossible to compress even today. As I said prior, it may be possible to achieve that in the future, althought I´m little bit skeptical, but - who knows???

As to multiple filetypes - there are 256 possible patterns, yet scattered randomly across input file. Thing is, prepare random data for better compressibility...