Reduced Length LZ (RLLZ): One way to output LZ77 codes

[compgt]

Hi all!

This is my first post, though i have been visiting this site recently every now and then, watching professional and expert programmers exchange lively ideas and suggestions to come up with the most powerful compressors.

Let me share here my one way to output LZ77 codes, which may improve LZ77/LZSS/ or LZW.

LZ77:

LZ77 coding transmits <offset, length> codes. Many times in the compression process, the same matching string is coded with the same <length> code. We can avoid transmitting the <length> code for many strings as well as not outputting a bit to identify a literal (as in LZSS) by the following:

Instead, read the whole input or block (or history buffer) and gather matching strings and encode the <<the whole string>> e.g., <<offset>> plus <<length of the string (n>=2)>> only this one time, <<the number of the same matching strings (ie., number of following offset codes)>>, and its succeeding occurrences in the input block by transmitting only the said <<offset>> codes. This is actually a "Reduced Length LZ (RLLZ)".

The literals are outputted last *without bit flags* since they fall into the block buffers not covered or "not activated" by encoded strings.

During decoding, the strings are written first in the output buffer using the offsets, and the literals "fill in" the unwritten positions in the output buffer.

Thanks,

-- Gerald R. Tamayo

[compgt]

No need to output bit flags for literals or the number of following consecutive literals in some algorithms.

Then the completely filled output buffer is written to file.

That is, e.g. after gathering all distinct repeated strings on a block,

1) no transmission of <length> code for the next occurrences of the same string (but, in the simplest way, you have to output the number of succeeding strings);

2) transmitting the literals last (in the output buffer) means no need to transmit *bit flags* for literals (and matches). This is the novel idea here: deferred literals output;

3) if you know LZT (2008 ) algorithm where it was demonstrated complete exclusion of the length code, this might as well be "LZT2". Should improve LZ77/LZSS/LZW based compressors. Decoding is also straight-forward.

Sorry that only now after a decade of LZT (2008 ) i am releasing this. I stopped coding compression in 2010 or 2011.

My question is: Is this new, or already in some ROLZ implementations?

[Shelwien]

All popular LZ implementations these days seem to use "rep-codes" for match distance.
I think I'd seen also attempts to use rep-codes for match length too, but it didn't show much improvement.
Thus, reusing a whole distance;length pair probably won't be very helpful - well, maybe on logs or some such.

Btw, the further generalization would be secondary LZ compression of LZ tokens.
It was tested before at least in form of bsdiff+lzma, and does show some potential.

[compgt]

Yes, I have similar ideas as "rep-codes" in 2000s. My compression ideas are from the late 90s when i became interested again in data compression.

The ones i call here "holes" they call "gaps" even in LZW. But the idea of deferring transmission of literals for an output buffer avoids bit flags for both literals and matches which is still used in some explanations of ROLZ. Other algorithms still need to output the number of following literals. Deferred, meaning this is not an "online" algorithm.

Yes, i understand LZ recompression for LZ tokens needs well-crafted tokens; this might be a huge algorithm.

Thanks, Shelwien!

[Lucas]

I experimented with this back idea in March 2018 but the problem I encountered was enumerating all subsets of the string since the starting point would end up fixed after a match was found, but optimal parsing requires multiple rotations of the string to be able to find the shortest path.

I never could get my implementation to beat LZ77, it was pretty much on par. I found that the odds of re-using a whole word is as probable as lz77 inserting a new word into the dictionary. By new word I mean a word which is either 1 byte shorter or longer than the next longest matching entry.

I used a 3 symbol flag: literal (no match), lz77 match (insert word), and reuse word via offset code. I guess it'll be worth revisiting but it'll need a global match finder like a suffix array to find the best matches across the whole input unlike the binary tree I was using.

There may be potential with this system for text but it is quite a challenge to get it working well.

[compgt]

...

I never could get my implementation to beat LZ77, it was pretty much on par. I found that the odds of re-using a whole word is as probable as lz77 inserting a new word into the dictionary. By new word I mean a word which is either 1 byte shorter or longer than the next longest matching entry.

I used a 3 symbol flag: literal (no match), lz77 match (insert word), and reuse word via offset code. I guess it'll be worth revisiting but it'll need a global match finder like a suffix array to find the best matches across the whole input unlike the binary tree I was using.

There may be potential with this system for text but it is quite a challenge to get it working well.

I guess, you're thinking about lz77's history buffer as "dictionary"?

> a global match finder like a suffix array to find the best matches

Indeed. Matt Mahoney did something like this in analyzing enwik9 strings...

[Lucas]

Yes I used the LZ history buffer as my dictionary, except I used the most recent 1MB of history to reference words, it was implemented as a 1MB array of uint32_t's to store the length which was referenced in the history, while the rest of the history buffer could grow to 512MB, this was done mainly to constrain decode memory requirements to 1N + 4MB

You've got me wondering if we were both onto something with this encode technique because in theory it can flip between traditional LZ77 and an LZ78 style compressor almost freely, at least I haven't seen it ever compress worse than LZ77, it might just be a matter of determining proper switching between the 2 methods.

[Dresdenboy]

What is the state of implementing this idea?

I had some thoughts and played it through in my mind (knowing my match/literal statistics of compressing binaries, no text though) in different variants (for example: not storing the matched string literal bytes with the offset list, but referencing it from the literals block, avoiding the first explicit copying from the string list to some first position (code bits!) and leaving that to the literal filler instead).

So far I think that saving the literal/match flags isn't enough to offset the needed pointers to either the first occurence or the literal block, even with some kind of delta coding, either going sequentially forward through referenced literals or target positions will cause more randomized references on the other side.

But for text compression it might actually work due to more frequent reuse of matched strings. In my files I mostly see single time matches (bytes at same absolute position only copied once for a match). There is surely a way to calculate the benefits based on reuse probabilities for strings.

[compgt]

> What is the state of implementing this idea?

Not implemented yet. I stopped coding in 2010. I just know this makes very compact LZ.

If i recall correctly that James Storer and Jennifer Chayes of Microsoft went to me in my high school (Philippines) in the early 1990s, then it was at that time that i mentioned to Storer the idea of transmitting the literals last, in LZ77. I didn't elaborate though, as i hadn't been programming anymore that time. I didn't have access to an own computer in the 1980s. Mostly theoretical.

I had no achievement in high school except when Hollywood people (and Jennifer Chayes!) went to me in school to sing or maybe re-record the official songs for Bread and America bands, maybe Nirvana too, etc. I didn't get some million$ for my Hollywood music and movies that time, not even thousand$. See, the lead singer in Bread is a "Gates" though it's my voice in the band's modern "official" songs. Others called me "David Gates" or "Bill Gates". To some, i was the real Bill Gates.

[JamesWasil]

> What is the state of implementing this idea?

Not implemented yet. I stopped coding in 2010. I just know this makes very compact LZ.

If i recall correctly that James Storer and Jennifer Chayes of Microsoft went to me in my high school (Philippines) in the early 1990s, then it was at that time that i mentioned to Storer the idea of transmitting the literals last, in LZ77. I didn't elaborate though, as i hadn't been programming anymore that time. I didn't have access to an own computer in the 1980s. Mostly theoretical.

I had no achievement in high school except when Hollywood people (and Jennifer Chayes!) went to me in school to sing or maybe re-record the official songs for Bread and America bands, maybe Nirvana too, etc. I didn't get some million$ for my Hollywood music and movies that time, not even thousand$. See, the lead singer in Bread is a "Gates" but it's my voice in the band's modern "official" songs.

Did you invent KITT for David Hasselhoff to play on Knight Rider as Michael Knight? Or did Wilton Knight come to you to get the design ideas and understand how the molecular bonded shell works?

[Dresdenboy]

> What is the state of implementing this idea?

Not implemented yet. I stopped coding in 2010. I just know this makes very compact LZ.

It looks like LZ77 wants to give RLLZ a try.

Did you invent KITT for David Hasselhoff to play on Knight Rider as Michael Knight? Or did Wilton Knight come to you to get the design ideas and understand how the molecular bonded shell works?

I think we can't help him here.

[danlock]

I wonder if it relates to the new Knight Rider series I heard was coming?

[JamesWasil]

I wonder if it relates to the new Knight Rider series I heard was coming?

Yes, Gerald Tomato will tell us that the newest Knight Rider was already done and voiced by him during the cold war in the Philippines and not by William Daniels, and that it was done during the 70's even though he never knew about it. Funny how that works.

Speaking of Knight Rider though, you know KITT and KARR had to use a great deal of data compression to store data with limited space and for transmissions with signals and frequencies. I wonder why the show never mentioned that or tried to (the original 80's show or the horrible reboots that came later). I figure Huffman or LZ77 might have been worth a mention at least once when talking about a state of the art computer and car of the future, but none of the episodes ever did?

[compgt]

Yes, Gerald Tomato will tell us that the newest Knight Rider was already done and voiced by him during the cold war in the Philippines and not by William Daniels, and that it was done during the 70's even though he never knew about it. Funny how that works.

Speaking of Knight Rider though, you know KITT and KARR had to use a great deal of data compression to store data with limited space and for transmissions with signals and frequencies. I wonder why the show never mentioned that or tried to (the original 80's show or the horrible reboots that came later). I figure Huffman or LZ77 might have been worth a mention at least once when talking about a state of the art computer and car of the future, but none of the episodes ever did?

Hurling insults to my name now, attacking me personally when you can just have an open mind?...

The corrupt people using my Hollywood billion$ for themselves it seems created that profile "Gerald Tamayo math wizard" in the news here in the Philippines, also a child math whiz kid?! who was good in calculating square roots of numbers, and who was begging on the streets for money. That's in the news in 2014 or 2015, though i think i saw it already earlier. This they did to discredit me when they found out i have an internet presence already via my "The Data Compression Guide" in 2008.

I don't despise the whiz kid, of course, no offense to him; He was just being used by these corrupt people. This will make people who google "Gerald Tamayo" think and perplexed that i'm that square root kid. I am the "Gerald Tamayo" whiz kid by that name, but in the 1970s to the 80s, pioneering modern computing and quantum computing, and was prolific in composing songs for the Hollywood singers and bands and making the Hollywood blockbuster movies. These hardcore corrupt people will do everything to discredit me or destroy me, at the same time shamelessly partaking of my Hollywood and tech billion$.

Yes, i made some "Knight Rider" episodes too and programmed its analog AIs, when i was Hollywood.

[JamesWasil]

Hurling insults to my name now, attacking me personally when you can just have an open mind?...

The corrupt people using my Hollywood billion$ for themselves it seems created that profile "Gerald Tamayo math wizard" in the news here in the Philippines, also a child math whiz kid?! who was good in calculating square roots of numbers, and who was begging on the streets for money. That's in the news in 2014 or 2015, though i think i saw it already earlier. This they did to discredit me when they found out i have an internet presence already via my "The Data Compression Guide" in 2008.

I don't despise the whiz kid, of course, no offense to him; He was just being used by these corrupt people. This will make people who google "Gerald Tamayo" think and perplexed that i'm that square root kid. I am the "Gerald Tamayo" whiz kid by that name, but in the 1970s to the 80s, pioneering modern computing and quantum computing, and was prolific in composing songs for the Hollywood singers and bands and making the Hollywood blockbuster movies. These hardcore corrupt people will do everything to discredit me or destroy me, at the same time shamelessly partaking of my Hollywood and tech billion$.

Yes, i made some "Knight Rider" episodes too and programmed its analog AIs, when i was Hollywood.

Oops, I thought your last name was Tomato not Tamayo. I must have misread. I apologize for getting your name wrong.

As for the rest of it, professional help of some sort may be needed, but I doubt that a lawyer to sue the tech firms is what's required for that. I sure hope that compgt is a troll of sorts. Otherwise, this is a very sad thing.

[Dresdenboy]

Oops, I thought your last name was Tomato not Tamayo. I must have misread. I apologize for getting your name wrong.

As for the rest of it, professional help of some sort may be needed, but I doubt that a lawyer to sue the tech firms is what's required for that. I sure hope that compgt is a troll of sorts. Otherwise, this is a very sad thing.

He created that data compression site with sources and interesting articles. For the other thing - not everything is in one's own hands. I know a similar case.

[compgt]

Oops, I thought your last name was Tomato not Tamayo. I must have misread. I apologize for getting your name wrong.

As for the rest of it, professional help of some sort may be needed, but I doubt that a lawyer to sue the tech firms is what's required for that. I sure hope that compgt is a troll of sorts. Otherwise, this is a very sad thing.

You're an idiot to not know the consequences of your insults. Karma knows the truth.

[JamesWasil]

He created that data compression site with sources and interesting articles. For the other thing - not everything is in one's own hands. I know a similar case.

Yeah, he copied things from other sites. Not original. He has big problems.

[JamesWasil]

You're an idiot to not know the consequences of your insults.

Nope. You're an idiot to not know anything but how to make false claims, you need mental help and a one-armed straight-jacket.

You're a piss poor troll. Stop polluting the compression forums with your bullshit. It doesn't amount to anything and only takes away from the value of what people come here for.

And yeah buddy, I'm just going to say it: you're absolutely insane. You invented nothing. Hollywood owes you nothing, because you didn't do anything. You just make bogus claims all over the place and need medication.

It's not my fault or anyone else's here that you're an idiot and the world failed you, and we're really tired of hearing about it on every single thread you interject and spam message you've sent to people unsolicited.

There were no insults until now. If you want to make insults, I'll give you some tomato. I was being nice to you but no more.

That's the consequences of your ad-hominem after I apologized for getting your name wrong. I take back my apology.

There are no consequences for telling it like it is, so there you go.

[compgt]

Nope. You're an idiot to not know anything but how to make false claims, you need mental help and a one-armed straight-jacket.

You're a piss poor troll. Stop polluting the compression forums with your bullshit. It doesn't amount to anything and only takes away from the value of what people come here for.

And yeah buddy, I'm just going to say it: you're absolutely insane. You invented nothing. Hollywood owes you nothing, because you didn't do anything. You just make bogus claims all over the place and need medication.

It's not my fault or anyone else's here that you're an idiot and the world failed you, and we're really tired of hearing about it on every single thread you interject and spam message you've sent to people unsolicited.

There were no insults until now. If you want to make insults, I'll give you some tomato. I was being nice to you but no more.

That's the consequences of your ad-hominem after I apologized for getting your name wrong. I take back my apology.

There are no consequences for telling it like it is, so there you go.

I am not used to this, people attacking me, my person, in online forums. I thought people are civil enough to be polite here. I have not attacked you, JamesWasil. You're helpless, you don't know what you're saying. You're one of those uncontrolled. Me, i was stating the truth.

You're the one who started insulting my name! Your apology is also an insult, undermining my intellect. I was telling the truth, of how modern Hollywood and modern computing industry came to be, by me, with me. Hollywood and the tech industry owes me bigtime. Their mistake was to totally betray me, left me with nothing to own. They were overkill, blinded by the billion$, fueled by their greed and envy. They're corrupting my Hollywood billion$. I made modern Hollywood. I pioneered modern computing. That was classified top secret, now declassified by me.

[compgt]

Yeah, he copied things from other sites. Not original. He has big problems.

Original text, original implementations. GPL was new that time. And I have to make my programs different from others because they were intended to be sample code programs for The Data Compression Guide.

[JamesWasil]

I am not used to this, people attacking me, my person, in online forums. I thought people are civil enough to be polite here. I have not attacked you, JamesWasil. You're helpless, you don't know what you're saying. You're one of those uncontrolled. Me, i was stating the truth.

You're the one who started insulting my name! Your apology is also an insult, undermining my intellect. I was telling the truth, of how modern Hollywood and modern computing industry came to be, by me, with me. Hollywood and the tech industry owes me bigtime. Their mistake was to totally betray me, left me with nothing to own. They were overkill, blinded by the billion$, fueled by their greed and envy. They're corrupting my Hollywood billion$. I made modern Hollywood. I pioneered modern computing. That was classified top secret, now declassified by me.

Well maybe you need to get used to it, because you've only posted crap and spammed people with nonsense and your delusional rants without provocation, and quite frankly, I've had enough of your bs and how you turn every single thing others have made or done into "ah yes, I remember doing/making that during the Cold War in the 1960's, 70's, and 80's!" Enough.

I'm not helpless, I know exactly what I'm saying, and I don't go around spamming people or writing bullcrap like you have. It makes the forum look bad, and it detracts from both the conversation and the progress of things when you do it. I tried to ignore it when it was contained to the off-topic threads, but then you go and put it all here, too.

None of this crap was around in 2007-2017 until you got here. I didn't make an account until a decade later when I felt like chiming in here and there and just read the forum between then and now. But even with agreements and disagreements back then that members have had, NO ONE was forced to smile and nod at your delusional behavior as they are now. It's ridiculous.

Yeah, I'm one of the "uncontrolled", because I don't need medication and I have enough basic respect for 99% of the members here not to do the crap you're doing whether I know them or not.

If you're some other user on here and you think this is funny, you just need to stop. This is not what this forum was about, and it shouldn't be about that now, either. This is not Facebook or Twitter.

You've mentioned things about compression here and there and that's great. I support that and appreciate it. But what is not great is how you interject your delusions about having made everything that other people have done and claim it as your own from here to Hollywood.

It's demeaning and undermining to the intelligence of everyone else here and elsewhere when you do that, even if you have some sort of psychological problem that makes you think that is ok and acceptable to do it. It is not.

You have 0 proof of your claims in the off-topic thread, and that remains a constant here as well.

You just need to get help and stop disrespecting people here by saying you did everything they ask about or discuss, because it's only going to irritate people and make them angry and not come back, rather than fostering a decent discussion of data compression techniques, information theory, or anything else.

This entire thread should probably be moved to the Off-Topic area to not clutter up the main thread since it's more about your fantasies than it is about the subject itself.

[compgt]

Well maybe you need to get used to it, because you've only posted crap and spammed people with nonsense and your delusional rants without provocation, and quite frankly, I've had enough of your bs and how you turn every single thing others have made or done into "ah yes, I remember doing/making that during the Cold War in the 1960's, 70's, and 80's!" Enough.

I'm not helpless, I know exactly what I'm saying, and I don't go around spamming people or writing bullcrap like you have. It makes the forum look bad, and it detracts from both the conversation and the progress of things when you do it. I tried to ignore it when it was contained to the off-topic threads, but then you go and put it all here, too.

None of this crap was around in 2007-2017 until you got here. I didn't make an account until a decade later when I felt like chiming in here and there and just read the forum between then and now. But even with agreements and disagreements back then that members have had, NO ONE was forced to smile and nod at your delusional behavior as they are now. It's ridiculous.

Yeah, I'm one of the "uncontrolled", because I don't need medication and I have enough basic respect for 99% of the members here not to do the crap you're doing whether I know them or not.

If you're some other user on here and you think this is funny, you just need to stop. This is not what this forum was about, and it shouldn't be about that now, either. This is not Facebook or Twitter.

You've mentioned things about compression here and there and that's great. I support that and appreciate it. But what is not great is how you interject your delusions about having made everything that other people have done and claim it as your own from here to Hollywood.

It's demeaning and undermining to the intelligence of everyone else here and elsewhere when you do that, even if you have some sort of psychological problem that makes you think that is ok and acceptable to do it. It is not.

You have 0 proof of your claims in the off-topic thread, and that remains a constant here as well.

You just need to get help and stop disrespecting people here by saying you did everything they ask about or discuss, because it's only going to irritate people and make them angry and not come back, rather than fostering a decent discussion of data compression techniques, information theory, or anything else.

This entire thread should probably be moved to the Off-Topic area to not clutter up the main thread since it's more about your fantasies than it is about the subject itself.

I made modern Hollywood! I made the Hollywood blockbuster movies! The actors and actresses are just my people. I just made them. But they're popular more credible, they figured that out so they betrayed me and denied me.

Same in tech. I pioneered modern computing! I'm here for the money, for my Hollywood and tech billion$ which they stole from me. I don't like lawsuits; i can't stand being interrogated. I just want to get paid my due billion$. I don't want to be the evil sorcerer next time to destroy the world to avenge myself!! I can't annihilate all those people who steal from me. Somehow i must stop. They must understand in their hearts that they owe me. Somethings gotta give. And it must be them, paying me.

It's useless and senseless interacting with you here, @JamesWasil!!

[Dresdenboy]

Dear Gerald,

once I landed on your blog because of RLLZ and was quite a bit confused by your other postings there. A new check already brought up some dangerous stuff, like:

"These PAQ compressors were still being developed in the 1980s when i was in grade school. But it was actually started earlier than that, in the 1970s when i was a really young boy. It was our project for the US information theory education materials for the future, funded by us. What happened (the researchers' results) will be part of the history of US information theory, i said. These materials included Data Compression The Complete Reference (Salomon's book), Data Compression Explained and LTCB programs, released only in 2000s. I was running LTCB in the 1970s that some compressors were developed by my aunts or uncles, and really hardcore militaristic people. These programs were just homework assignments for advanced students before in Masbate, Masbate, Philippines. I am thinking DCE and the LTCB compressors were just given again to Matt Mahoney by the government in 2000s and officially launched them, or he already possessed, developed, or added on those materials since the 80s, like some people here (encode.su) owning 1980s compressors but published or released only in 1990s or 2000s.

I remember this encode.su website and GUI (1970s), as the names Shelwien, David Scott, Bulat Ziganshin, dnd, Matt Mahoney, Michael Maniscalco, Jyrki Alakuijala. Some of them would come to me in the Philippines in the 80s...if it was really them. By early 1990s i was forgetting already. In the mid 2000s i was strongly remembering again. If i hear my voice in the bands "official" recordings of Bread, Nirvana, America, Queen, Scorpions etc, i then strongly believe these computer science memories.

(Source: https://grtamayoblog.blogspot.com/20...-programs.html)

I've seen claims like that before. And this is serious stuff. Be very careful with such claims. There are two possible truths behind this. One is highly unlikely (p ~ 1), while the other is quite the opposite (1-p). In both ways there is just one way to solve this and show all those people, who stole your ideas, that you are right: Ask a psychiatrist to attest you full mental health, setting p to 1, so that billionaires, companies and countries have to accept this.

Otherwise you'll continue to live a life full of such thoughts probably making you unhappy. Clear this mess up, find the truth!

[compgt]

Dear Gerald,

once I landed on your blog because of RLLZ and was quite a bit confused by your other postings there. A new check already brought up some dangerous stuff, like:



(Source: https://grtamayoblog.blogspot.com/20...-programs.html)

I've seen claims like that before. And this is serious stuff. Be very careful with such claims. There are two possible truths behind this. One is highly unlikely (p ~ 1), while the other is quite the opposite (1-p). In both ways there is just one way to solve this and show all those people, who stole your ideas, that you are right: Ask a psychiatrist to attest you full mental health, setting p to 1, so that billionaires, companies and countries have to accept this.

Otherwise you'll continue to live a life full of such thoughts probably making you unhappy. Clear this mess up, find the truth!

@Dresdenboy, i am telling the truth. I planned the timeline of data compression and encryption history, to a moderate and steady pace of development.

It's classified top secret, now being declassified by me. I was the one who classified it in the first place anyway, since i owned the Hollywood and tech companies. I was on top anyway. I can live with my name not in songs as composer or co-composer, for example, because i own the Hollywood media (music and movie) companies anyway. Do you understand? But they took advantage of this fact, that my name is not there in the music bands albums and songs (and movies), they ousted me in the corporate. My voice is in the "official" songs of the bands Bread and America, and some songs of Nirvana, Scorpions, Queen and many other songs. We calculated and forecasted a billion$ revenue before for one high-selling Beatles music album, maybe with Bread music album/s too etc, i think. I made the Hollywood blockbuster movies Star Wars, Star Trek, Transformers, and the Marvel movies etc. to document how advanced we were during the Cold War. It was war, yet i was also this lovely boy singing to mellow everything.

The same in tech, i was supposed to be co-founder of Yahoo, Google and Facebook (already owning Apple, Microsoft, IBM, Intel, AMD), though i chose and made popular Jerry Yang, Sergey Brin and Larry Page, and Mark Zuckerberg. It's the power and popularity of American globalism i understood early on but it was me, a Filipino Asian, orchestrating everything during the 1970s to the 80s, this timetable of science and technologies. Do you understand?

@SCOTUS should just order these Hollywood and tech companies to pay me. It's that simple.

[compgt]

I've seen claims like that before. And this is serious stuff. Be very careful with such claims. There are two possible truths behind this. One is highly unlikely (p ~ 1), while the other is quite the opposite (1-p). In both ways there is just one

Ask a psychiatrist to attest you full mental health, setting p to 1, so that billionaires, companies and countries have to accept this.

Otherwise you'll continue to live a life full of such thoughts probably making you unhappy. Clear this mess up, find the truth!

There's no need to state that, Dresdenboy. I've hinted that already in another thread:

ivan2k2, you sound young and inexperienced. A smart man will not mention about insanity, especially in public online forums. I am not used to that, somebody mentioning "insane" and "mental health". It saddens me. Too brave of you to even mention that. Corrupt, hardcore people (corrupt military and politicians) will set you up, will put you in a mental institution because you are simply in their way. They don't care about you, about your life. They have no ethics for that. They are paid big money for that. (I reckon they had been corrupting and partaking of my Hollywood billion$ up to now since the 1980s.)

I suppose you're not one of them, paid to attack me? They made huge efforts to put me in a mental institution in the 1980s, trying to dispose of me quickly, but i was a known child genius or honor student that i was supported and protected by teachers in grade school and high school. So they didn't succeed; i continued my schooling, in between years this grade school time managed to sing and record the official Bread and America songs etc. you're now used to listen to.

I was my own psychiatrist. I was very good psychiatrist assuaging and appeasing people to forget the Cold War. I influenced the field of modern psychiatry too because i was wizard. I designed pharmaceutical drugs and medicine, owning pharma tech companies but taken from me too.

They succeeded in early 2010s to set me up for the mental institution. They set me up by "threatening" me. If i retaliate they will tag me as "violent" fit for a mental institution. These people are so hardcore corrupt that they will really live near me, to disturb me, to partake of my Hollywood billion$ they are corrupting. I recently remembered he was messing with me in college too, in the 1990s. He was an addict, rapist maniac, and hardcore killer up to 1980s, i now recall. That's what corrupt powerful politicians do to their enemies or those in their way, sending them to jail or mental institution. These are the corrupt people we should be afraid of.

So now, Dresdenboy, i actually see my psychiatrist regularly just for the meds. Just like any other rock star, huh?

If i may add, how secure are today's singers and bands?

P. S.

I am now living peacefully here in the Philippines.

It would be more peaceful and serene and i would be glad if i get my Hollywood million$ first, in the meantime, then my tech billion$.

[lz77]

All popular LZ implementations these days seem to use "rep-codes" for match distance.

Do you mean using history buffer for offsets? How can you shorten the length of offset with it? I tried a buffer for 8 latest offsets and delta coding and did not get a gain in the ratio...

[Dresdenboy]

Do you mean using history buffer for offsets? How can you shorten the length of offset with it? I tried a buffer for 8 latest offsets and delta coding and did not get a gain in the ratio...

You might reserve some very short codes for them. Look how LZX is doing it, or what fgiesen wrote here: https://www.google.com/amp/s/fgiesen...n-bitknit/amp/

[lz77]

Thanks, but it looks like a promotional article about nothing...
How to distinguish a codeword with full offset from codeword with a link to repeated offset? This bit flag will eat all the benefits. What result shows LZX on enwik8 with and without this feature?

And to improve the ratio of special files like .img in the Calgary corpus can be used better methods.

Repeated offset like THEN,THAT abcd... THEN THAT when a long match is separated by different literals occurs, but rare...

And I do not see how I can squeeze 150 Mb of the TS40.txt after preprocessing and LZ77 compression to 120 Mb with entropy coding like lzturbo...

It seems that the more economical the LZ77 encoder encodes, the worse the final ratio after entropy encoding...

[Dresdenboy]

Thanks, but it looks like a promotional article about nothing...
How to distinguish a codeword with full offset from codeword with a link to repeated offset? This bit flag will eat all the benefits. What result shows LZX on enwik8 with and without this feature?

Fabian at least provided some statistics. And checking the probabilities of the rep-offsets and regular offsets of course makes sense. For texts they might even be less useful (as you mentioned already), as they actually cover structured binary data.
Charles Bloom also wrote a lot (google search link) about them.

And to improve the ratio of special files like .img in the Calgary corpus can be used better methods.

Of course. That's the good thing about compression - there is still room for improvement.

And I do not see how I can squeeze 150 Mb of the TS40.txt after preprocessing and LZ77 compression to 120 Mb with entropy coding like lzturbo...

It seems that the more economical the LZ77 encoder encodes, the worse the final ratio after entropy encoding...

The latter makes sense - because that encoding used for the LZ77 variant already is some kind of pre defined entropy encoding without adapting to actual statistics. That is where my idea from running an optimization of the encoding parameters came from, which I mentioned in the TS40 help thread.

Do you have the amounts of short and long matches? This is useful for estimating the potential for even better encoding (w/o full force entropy coding).