Why not open source?

[nemequ]

There is a lot of great open source compression software that gets posted here, but there is also a lot of great closed source compression software. I'm pretty deeply involved in open source so perhaps I'm just having trouble seeing things from another perspective, but I'm having trouble understanding the reasons why people would choose to develop closed source compression software. Therefore, I thought I should ask those of you who are doing so and releasing it for free here: why not release the source code?

The potential reasons which I could come up with, as well as why I have trouble accepting them, are listed below. I'd love to hear your thoughts, especially if you disagree or have reasons I haven't thought of.

Commercialization

Obviously, commercializing open source software is more complicated. That said, there are some really great open source solutions across the spectrum (from fast to slow but with high compression ratios), which makes me wonder how much of a market there really is for proprietary solutions. Are there many people choosing to pay to use your software when there are open competitors which are very, well, competitive? Why not use a dual-licensing scheme (like LZO) to try to popularize your implementation while maintaining commercial viability?

Code quality

I know a lot of people are, frankly, a bit ashamed of some of their code quality. This has certainly stopped me from releasing source code on more than one occasion, but wouldn't it be better to just include a disclaimer that the code quality is questionable and hope that someone else finds the code interesting enough to use, and maybe even improve, in spite of that?

Inclusion of 3rd-party IP

Lots of proprietary code includes content licensed from third parties which cannot be released. I have trouble believing this is a significant issue given how low-level most of the software in question is.

Maintaining trade secrets

Isn't the whole purpose of this forum to share information on data compression? Why even post anything at all?

So, what am I missing? I know some people have some strong opinions on the matter, and I really hope this doesn't turn into a flame war—if it does I'm sorry, and hopefully the mods will close the thread.

[Matt Mahoney]

Well, I agree. There is basically no market for software any more. Software is something that controls a product or service, not something you sell by itself. I realize the old business model of writing and selling programs used to work, but that was before the internet.

With regard to data compression, there is no good reason for people to create archives with proprietary products when there are free, open source products like zip, 7zip, freearc, zpaq, etc. Closed source projects tend to get abandoned, and then you can't extract your files because the archiver won't run on your new computer or you can't find the program anymore. Does anybody remember Compressia or WinRK?

You might wonder how anyone can make money writing software when almost all of it is free. The way it works now is you publish code to advertise your programming skills. The better your code, the better your job offers will be. If you want good code, then you want lots of users to give you feedback. If you want a lot of users then your code should be free with an unrestricted license. Here are two examples:

1. I made PAQ open source, which allowed many other people to make modified versions of it. Early versions were not very good. Without help from others, it would not have moved to the top of the benchmarks and I wouldn't be getting a high salary for working at home doing what I considered a hobby.

2. I know that ZPAQ works on a lot of platforms like on an ARM, a Mac, and on a backup server with terabyte sized archives, not because I had such systems for testing, but because others had the source code.

Also, I think that open source code is always going to be higher quality than closed source, simply because more people can look at it and find bugs. This is pretty important for anything involving security, like an archiver with an encryption feature, or a website that decompresses data received over the internet.

[nburns]

I assume that the people here that don't share their source are either aiming for prizes or are reserving the option to try for a prize at a later time. I can't think of any other reason.

[Matt Mahoney]

Well, I think that people don't publish their code so they can retain the rights to sell it to companies that want to use it. If they published source, people could just steal it and use it in their own products. But that doesn't make sense to me. Understanding someone else's source code is a lot of work, almost as much as writing it. If a company wants it, they have to hire the person who wrote it or else it's mostly worthless.

Even when competing for prizes, I don't see the advantage. When I was competing in the SequenceSqueeze contest, which required publishing our entries on SourceForge, James Bonfield and I went even further by discussing results and describing our algorithms, which I think helped both of us finish in the top 2 spots. Well, he won, but we both got a co-authored published research paper out of it. We hardly even looked at each other's code, and I don't think the other contestants looked either.

[nburns]

The other obstacle to selling it is finding someone that wants it, when compression is already a cheap commodity. Google wanted a fast compression format, so they made their own. Then they let people download it for free.

we both got a co-authored published research paper out of it

My interest is academic in nature. There's no value in doing research and not telling anyone about it.

[nemequ]

Well, I think that people don't publish their code so they can retain the rights to sell it to companies that want to use it. If they published source, people could just steal it and use it in their own products.

Right, this is basically the "commercialization" argument I mentioned above. The problem is that it only makes sense for software which is better than the competition, and even then it usually has to be a lot better for someone to choose to pay for it. However, I don't really see that happening. With LZ4, LZMA, ZPAQ, etc., available, the open source compression software is either the best-of-breed or close enough that I simply can't see why people would use something proprietary. Unless it's a huge improvement, I just can't imagine there being much of a market. Or is there some market for these data compression algorithms which I'm not seeing?

But that doesn't make sense to me. Understanding someone else's source code is a lot of work, almost as much as writing it. If a company wants it, they have to hire the person who wrote it or else it's mostly worthless.

That's assuming they actually need to understand the code. I wrote a library (Squash) which integrates a lot of compression libraries. I've even helped fix a couple of bugs in those libraries, but I don't have a particularly deep understanding of any of them.

Then again, I think anyone wanting to integrate compression software deeply enough into their code base to justify purchasing a license (and support agreement) for something just a little bit better than the open source offerings would also have no problem paying someone as a consultant for open source code. The people who don't need such deep integration work would probably rather use something which is free/open, even if it isn't quite as good.

Even when competing for prizes, I don't see the advantage. When I was competing in the SequenceSqueeze contest, which required publishing our entries on SourceForge, James Bonfield and I went even further by discussing results and describing our algorithms, which I think helped both of us finish in the top 2 spots. Well, he won, but we both got a co-authored published research paper out of it. We hardly even looked at each other's code, and I don't think the other contestants looked either.

I have trouble believing competing for such prizes is a sound business model, given how uncommon they are and how much they pay. Even if you won them all, could you really make enough money to justify all the time you put in? Keep in mind that you need to be pretty smart and well-educated to win, so there are probably plenty of other opportunities for you to make money.

The other obstacle to selling it is finding someone that wants it, when compression is already a cheap commodity. Google wanted a fast compression format, so they made their own. Then they let people download it for free.

Well said, though I'd like to add that in a lot of situations LZ4 is better than Snappy. Even a huge company with the best software engineering talent money can buy only managed to make something which is competitive. And now that Snappy is out there (and free), it will be even harder for the next person.

Zopfli is, perhaps, a bit more interesting for the purposes of this conversation, but it's such a small niche that I don't see a lot of room for a commercial offering. And the market for something like zopfli for anything other than zlib/deflate is virtually non-existent.


I'd really love to hear from someone releasing closed source compression software, since all the rest of us can really do is speculate.

[nburns]

I have trouble believing competing for such prizes is a sound business model, given how uncommon they are and how much they pay. Even if you won them all, could you really make enough money to justify all the time you put in? Keep in mind that you need to be pretty smart and well-educated to win, so there are probably plenty of other opportunities for you to make money.

I don't think it's a business model. It's more like an intellectual sport. It seems to be popular among Europeans, who have lots of time off from work.

Well said, though I'd like to add that in a lot of situations LZ4 is better than Snappy. Even a huge company with the best software engineering talent money can buy only managed to make something which is competitive. And now that Snappy is out there (and free), it will be even harder for the next person.

That kind of makes the point, because Google whipped up their own and it's not even the best, but I don't think they care. That shows how little it's worth to do it incrementally better. Also, the world is full of compression software, with numerous options for doing the same thing, all free.

[encode]

Snappy has a much greater compression potential - IIRC 16 MB window vs 64 KB one of LZ4. Anyway, LZ4 is great and I think I will try to write an "Optimized LZ4" - same compression format with a higher compression, I'm pretty sure it is possible. Actually my LZF can't be slower than LZ4 - I'm just using slow getc()/putc(), and author of LZ4 definitely did something with I/O.

As to Open Source - if I opened the source each time, then I've never earn even a cent from my hard work! At least with Closed Sources, time at a time, large companies purchase licenses of my software. And again that companies already hired and have Data Compression professionals - to mess with my code and other stuff. Clearly no reason to hire me! On the other hand, for many times I see how people steal ideas from my Open Source projects - despite of license prohibition. That's why now I'm releasing "Public Domain" code.

Well, if I lived in Silicon Valley, have my own company and lawyers - that would be a different story...

[thorfdbg]

Well, I think that people don't publish their code so they can retain the rights to sell it to companies that want to use it. If they published source, people could just steal it and use it in their own products. But that doesn't make sense to me. Understanding someone else's source code is a lot of work, almost as much as writing it. If a company wants it, they have to hire the person who wrote it or else it's mostly worthless.

Depends a bit how the software is made, though I agree in general. "Using" such a software is less a problem, but keeping the API stable, keeping the design stable and fitting it to the company product line is more of a problem. That is, if you depend on source code by somebody else without paying her or him, you cannot ensure that the design and goals of that software stay stable enough to make parts of your products dependent upon. There are a couple of very good stable open source software products (zlib comes to my mind) that are stable and great, and other products with have been modified with questionable intent or even leaving the original design goals.

You do not pay programmers to write programs anymore. You pay them to ensure that the software fits to the overall "picture" or "infrastructure" of your company.

Even when competing for prizes, I don't see the advantage. When I was competing in the SequenceSqueeze contest, which required publishing our entries on SourceForge, James Bonfield and I went even further by discussing results and describing our algorithms, which I think helped both of us finish in the top 2 spots. Well, he won, but we both got a co-authored published research paper out of it. We hardly even looked at each other's code, and I don't think the other contestants looked either.

I believe any type of research depends on openness, and I also believe that there is currently something seriously wrong in publication and scientific research. Namely, nowadays it is no longer sufficient to publish papers. You should *also* publish the source code you used to generate such results. After all, the central point in science is reproducability of results (this is what science is after), but given the complexity of software today, you cannot really reproduce anything without having the source.

[thorfdbg]

Well, I agree. There is basically no market for software any more. Software is something that controls a product or service, not something you sell by itself. I realize the old business model of writing and selling programs used to work, but that was before the internet.

Bingo! Software is a tool to provide a service you sell, but not the product itself.

Also, I think that open source code is always going to be higher quality than closed source, simply because more people can look at it and find bugs. This is pretty important for anything involving security, like an archiver with an encryption feature, or a website that decompresses data received over the internet.

It depends. Yes, I agree that the overall quality of the code (as it is) is typically quite good as far as open source goes. However, the stability of the software design is better in the commercial model as the commercial model depends on the satisfacction of the customers. The open source model depends on the satisfaction of the programmer - something very different.

Consider for example Linux: I'm a great fan of it, and I'm using it (as I do right now). But the overall problem is that it remains a moving target. The Open source community does not have enough work power, leave alone motivation, to generate a stable kernel API for device drivers, for example. If you want support for a piece of hardware for Linux, it basically means that you need to keep updating your software over and over again because the internal kernel API changes from revision to revision. This is different for commercial software: As crappy as windows might be, but the device driver API is pretty stable and changes very slowly, ensuring workking devices.

When compiling a linux kernel, I never know whether my hardware keeps working, and if it doesn't, I get no service whatsoever to keep it working. I can write bug reports, or try to fix it myself, or... but there is nobody that I can make reponsible (as for example "under warranty") to give me support I need. Thus, this is not a usable model for end-user support that require computer equipment "as a tool" and not "as a goal by itself".

[Cyan]

I will try to write an "Optimized LZ4" - same compression format with a higher compression, I'm pretty sure it is possible.

Indeed, it is => http://cbloomrants.blogspot.fr/2012/...mal-parse.html
The place is still up for grab...

[nemequ]

As to Open Source - if I opened the source each time, then I've never earn even a cent from my hard work! At least with Closed Sources, time at a time, large companies purchase licenses of my software.

Has this happened recently? We haven't always had good open-source compression software (like LZ4 and Snappy) for fast compressors which didn't require purchasing a license for proprietary use (like LZO). Now that we do, I'm wondering if that market has dried up to some extent.

What are their use cases? How can they justify spending money on a license when there are free software alternatives which are very competitive, or are these corner-cases where there isn't any competitive free software?

Does it happen enough to justify the effort you spend writing the software, or is it just an occasional perk for something you enjoy doing?

And again that companies already hired and have Data Compression professionals - to mess with my code and other stuff. Clearly no reason to hire me!

Why not? If they're buying a license from you they obviously recognize the quality of your work, so what makes you think they wouldn't be interested in hiring you?

On the other hand, for many times I see how people steal ideas from my Open Source projects - despite of license prohibition. That's why now I'm releasing "Public Domain" code.

What license were you using? I'm not aware of any open source license which prohibits "stealing" ideas. They're all about protecting the implementation. Actually, I'm not sure it's possible to have an open source license (under the OSI definition) which prohibits such things. If your ideas are truly novel (and you don't want to share them freely), then you should apply for a patent. Of course, patents are a great way to make sure ideas never get seriously considered, but if you're trying to keep the ideas to yourself anyways it doesn't seem like that would really be a concern for you.

Well, if I lived in Silicon Valley, have my own company and lawyers - that would be a different story...

If you're talking about people taking your ideas, I don't think there is anything to take legal action over. If you're talking about people using your code in violation of the license, there are lawyers who will help at no cost to you: the Software Freedom Law Center comes to mind.

[encode]

Well, I'm here not since yesterday. I'm selling my programs for quite long time (since 2003 I believe). That is before LZ4, Snappy or BSC. A company spend money if it founds that nothing is better. Or I do a custom design for e.g. Nintendo DS ROM compression, or custom CM for quite specific data... Just today I have no spare time to write documentation for my programs or run comparison charts. Please check out my homepage for more: http://compressme.net/

[JamesB]

Even when competing for prizes, I don't see the advantage. When I was competing in the SequenceSqueeze contest, which required publishing our entries on SourceForge, James Bonfield and I went even further by discussing results and describing our algorithms, which I think helped both of us finish in the top 2 spots. Well, he won, but we both got a co-authored published research paper out of it. We hardly even looked at each other's code, and I don't think the other contestants looked either.

There were a considerable number of downloads of fqzcomp during the competition itself, so some people were clearly looking for it. However I'm still in favour of open source and the vast majority of my work is public. I may perhaps be in a lucky situation though in that I am employed by a research organisation who gives me a certain degree of flexibility. If I was self employed I may feel differently about releasing my code.

However I agree that there is little to gain in non-public code in the compression sector now unless it's something very specific such as an optimised zlib equivalent which is API and format compatible by 3x faster. (That'd be worth real money to a lot of organisations as they don't like changing legacy software.) Generally I'd expect most freelance money in compression is going to come through consulting and writing custom compressors for a specific data set, and this is where having at least some of your code public is a good thing. It's advertising!

[Matt Mahoney]

I have trouble believing competing for such prizes is a sound business model, given how uncommon they are and how much they pay. Even if you won them all, could you really make enough money to justify all the time you put in? Keep in mind that you need to be pretty smart and well-educated to win, so there are probably plenty of other opportunities for you to make money.

Well, there is a business model in organizing the competitions. http://www.kaggle.com/competitions

But winning a contest, even one with no prize money, is still good for your career.

[nburns]

Well, there is a business model in organizing the competitions. http://www.kaggle.com/competitions

But winning a contest, even one with no prize money, is still good for your career.

And good for your ego, I'm sure.

[Matt Mahoney]

I have no trouble finding people wanting to compete on my benchmarks, even though there is no prize money.

[nburns]

What license were you using? I'm not aware of any open source license which prohibits "stealing" ideas. They're all about protecting the implementation. Actually, I'm not sure it's possible to have an open source license (under the OSI definition) which prohibits such things. If your ideas are truly novel (and you don't want to share them freely), then you should apply for a patent. Of course, patents are a great way to make sure ideas never get seriously considered, but if you're trying to keep the ideas to yourself anyways it doesn't seem like that would really be a concern for you.

I don't think it's possible (at least in the US) to own an idea without going through the patent process. This is different than a piece of writing, which AFAIK you have automatic ownership of -- you own the copyright automatically without having to do anything. International law is consistent in most cases with US law.

[thorfdbg]

A company spend money if it founds that nothing is better. Or I do a custom design for e.g. Nintendo DS ROM compression, or custom CM for quite specific data... Just today I have no spare time to write documentation for my programs or run comparison charts.

But what is the market for compression nowadays, and which data do you need to compress? I mean, even big committees like MPEG are fiting with the problem that bandwidth is available nowadays, and storage is available nowadays, for good prices. If you need to store more data, buy a bigger disk. Thus, I wonder, which market demands does it address? Given an archiver compresses 1% better than its competitor, why would anyone bother to pay for that 1%?

[nemequ]

Well, I'm here not since yesterday. I'm selling my programs for quite long time (since 2003 I believe). That is before LZ4, Snappy or BSC.

Yes, and I understand the reason for releasing closed source compression software then. However, you (and others) are still releasing closed source compression software, which I still don't understand.

A company spend money if it founds that nothing is better. Or I do a custom design for e.g. Nintendo DS ROM compression, or custom CM for quite specific data...

I think this makes sense, I still don't see the benefit of closed source here. If a company needs custom CM, how does the software you implement the custom CM for being closed source help?

If they need a custom algorithm, how do other algorithms being closed source help? If you're lucky it wouldn't hurt, but I don't see how it would help.

Just today I have no spare time to write documentation for my programs or run comparison charts. Please check out my homepage for more: http://compressme.net/

I'm not sure why you're mentioning this. You don't have to write documentation to release software as open source; it's nice, but it's not a requirement. Benchmarks are even less important—other people will run those for you, and people are more likely to trust an outside source because there is chance of bias.

I don't think it's possible (at least in the US) to own an idea without going through the patent process. This is different than a piece of writing, which AFAIK you have automatic ownership of -- you own the copyright automatically without having to do anything. International law is consistent in most cases with US law.

Right, I was referring to the idea of a license which says you can't use the knowledge you gain from reading the code to write a competitor. It's not an uncommon thing to see in proprietary software, although in my experience they generally rely on NDAs. I don't recall ever seeing anything similar in an open-source license, and I'm not sure it's really possible to have something like that and be compliant with the OSI definition.

[encode]

Yes, and I understand the reason for releasing closed source compression software then. However, you (and others) are still releasing closed source compression software, which I still don't understand.

I'm releasing Open Source as well

I think this makes sense, I still don't see the benefit of closed source here. If a company needs custom CM, how does the software you implement the custom CM for being closed source help?

If they need a custom algorithm, how do other algorithms being closed source help? If you're lucky it wouldn't hurt, but I don't see how it would help.

I'm selling the source of a custom algorithm. And these algorithms are completely non public. And I can't make it public or open the sources...

[nemequ]

I'm releasing Open Source as well

Right, I understand that (and appreciate it!). And I understand the older stuff being closed source because, at the time, they were significantly better than the open source alternatives.

What I'm still having trouble understanding the reason for is the new closed source stuff. Do you feel that it's significantly better than the best existing open source competitors, by enough of a margin that it's likely a company would be willing to pay for a license?

I'm selling the source of a custom algorithm. And these algorithms are completely non public. And I can't make it public or open the sources...

If someone else pays you to create a compression algorithm (and/or implementation of one), it's obviously up to them whether or not to open source it. What I'm wondering about is the software you don't write for a particular customer, where you already distribute the executable for free. In other words, those times when you have a choice of whether or not to open source, and choose not to.

Perhaps it would be easier to communicate using a specific example, so let's use your new optimized LZF as that example. As far as I can tell, your reason for not releasing the source code is hope of commercialization. So, where do you see potential commercialization opportunities for this new software? I haven't looked at Optimized LZF closely, but AFAICT its open source competition would really be LZ4 or Snappy, correct? Does it outperform LZ4 and Snappy by a significant enough margin that you feel a company might be interested in acquiring a license instead of using a free, popular, stable, and proven alternative?

I really don't mean to pick on Optimized LZF—I only mention it because you're the author. The same question applies to pretty much every piece of closed source compression software out there, although maybe their competition is LZMA, ZPAQ, bsc, LZO, etc., not LZ4 or Snappy.

[encode]

Well, Optimized LZF to LZF is like the KZIP to Deflate. It achieves a higher compression within the same compression format (LibLZF). It is not finished yet - I'll continue improving it. And it's not about any commercial success - I just like to run the project myself! Data Compression is just the subject of my interest - I'm not trying to gain any money or popularity...

[nburns]

Right, I was referring to the idea of a license which says you can't use the knowledge you gain from reading the code to write a competitor. It's not an uncommon thing to see in proprietary software, although in my experience they generally rely on NDAs. I don't recall ever seeing anything similar in an open-source license, and I'm not sure it's really possible to have something like that and be compliant with the OSI definition.

If that language was in an open source license, it would probably not be enforceable. IANAL, but I think this comes down to contract law. If you have entered into a contract, such as an NDA, then you can put whatever stipulations you want into it, and then sue for breach of contract if one party fails to abide by the contract. If code has been published on the internet under an open source license, then it would probably be impossible to claim that anyone that comes into contact with the ideas in the code has agreed to a contract.

[Matt Mahoney]

That is what patents are for. But I think that patents are a bad idea if you want people to use your software.

[thorfdbg]

That is what patents are for. But I think that patents are a bad idea if you want people to use your software.

Why? It all depends on the licences you provide to grant users access to your IPs. If they are friendly enough, I don't see so much of a problem.

[Matt Mahoney]

Patents are the reason that JPEG uses Huffman coding instead of arithmetic coding and images are 10% larger.

[Mangix]

Well, arithmetic coding is also ~3x slower to decode. Some benchmarks:

Stock Huff: real 0m0.680s
Optimized Huff: real 0m0.675s
Progressive: real 0m1.169s
Arith: real 0m2.145s
Arith prog: real 0m2.285s

libjpeg-turbo is of course faster but there is greater SIMD speedup with the huffman images, probably because of better optimized code.

[thorfdbg]

Patents are the reason that JPEG uses Huffman coding instead of arithmetic coding and images are 10% larger.

No. Patents only granted under RAND conditions are the reason that AC coding has not been used. If access to the same patents would have been granted under FRAND conditions, things would be different. It's not the patents that caused the problem, but the greedy patent holders.

This being said, look how MPEG works: It's all patented, lots of IPs only available under RAND conditions, yet succesful.

[thorfdbg]

Well, arithmetic coding is also ~3x slower to decode.

This is, in this generality, just a wrong statement. It is a matter of the implementation how fast it performs, and even then, the Huffman coding is only one part of the actual decoding process. Also, the model is quite different, so this is pretty much an apples to oranges comparison. For progressive, it even more depends on how the progressions are organized. With proper tricks, the "baseline Huffman JPEG mode" can be implemented with about ~10 times the speed. AC has never received enough market demand to justify any similar optimization.