Quasi-Unlimited Packer with no data loss ( seems serious )

[uglyvan]

this is a yabasic programc;

try it on small size zip files ( like as of 1Mb ) to check its efficiency

then a serious programmer will enhance my program for professional use

dim t(100000000)
dim u(100000000)
dim v(1500)

i=1

open 1,"d:dummy.zip","rb"
open 2,"d:dummy1.txt","wb"
open 3,"d:dummy1_log.txt","w"

after each execution of the program, pack the file dummy1.txt which will label dummy1.zip, and then a dummy2.txt will appear

actually, I used this method on a 800 kb zip file which, after 15 executions, drops to 4 kb...

give it a try, I hope it will enlight your journey

while(!eof(1))
t(i)=peek(1)
i=i+1
rem print i
wend

close 1

20
print "pass 1..."

j=1
k=1
o=0
m=1

if (t(1)-128>=0) then
f=1
else
f=0
endif

while (j<=i)
rem print j,t(j)
zz=256
while (zz>1)
zz=zz/2
if (t(j)-zz>=0) then
40
f=1
else
f=0
endif
if (f=1 and o=0) then
k=k+1
u(k)=1
o=1
else if (f=1 and o=1) then
u(k)=u(k)+1
if (m<u(k)) then
m=u(k)
endif
t(j)=t(j)-zz
elseif (f=0 and o=0) then
u(k)=u(k)+1
if (m<u(k)) then
m=u(k)
endif
60
elseif (f=0 and o=1) then
k=k+1
u(k)=1
o=0
endif
endif
wend
j=j+1
wend
print "almost finished"

print
print m

for l=1 to k
poke #2,u(l)
v(u(l))=v(u(l))+1

next l
80
close 2

print "finish"

for m=1 to 1500
if (v(m)>0) then
print m,v(m)
r=r+m*v(m)
print #3,m,v(m),m*v(m)
endif
next m

print #3,r
close 3

end

[Bulat Ziganshin]

decompressor implementation is obvious, so it left as exercise to the reader

[AiZ]

[uglyvan]

hint: the last line from dummy_log is, with a one-byte difference, exactly 8 times the original dummy.zip; that is to say, multiply the file size by 8 and you will get the bit_original_filesize...

( obscure, isn't it ?? )

[Sportman]

I used this method on a 800 kb zip file which, after 15 executions, drops to 4 kb...

I tried it with a zipped yabasic.exe as input:

yabasic.exe 372,224 bytes
dummy.zip 186,836 bytes
dummy1.txt 360,001 bytes
dummy1.zip 134,730 bytes
dummy2.txt 261,831 bytes
dummy2.zip 96,542 bytes
dummy3.txt 187,265 bytes
dummy3.zip 69,726 bytes
dummy4.txt 135,351 bytes
dummy4.zip 50,280 bytes
dummy5.txt 97,539 bytes
dummy5.zip 36,543 bytes
dummy6.txt 70,869 bytes
dummy6.zip 26,695 bytes
dummy7.txt 51,685 bytes
dummy7.zip 19,672 bytes
dummy8.txt 38,099 bytes
dummy8.zip 14,665 bytes
dummy9.txt 28,345 bytes
dummy9.zip 10,928 bytes
dummy10.txt 21,137 bytes
dummy10.zip 8,329 bytes
dummy11.txt 16,017 bytes
dummy11.zip 6,421 bytes
dummy12.txt 12,337 bytes
dummy12.zip 5,012 bytes
dummy13.txt 9,625 bytes
dummy13.zip 3,967 bytes
dummy14.txt 7,601 bytes
dummy14.zip 3,180 bytes
dummy15.txt 6,087 bytes
dummy15.zip 2,575 bytes

How to reverse it

[uglyvan]

to bring on the unpacker we need to know the program's philosophy;
here it comes:

1° it acts as a complement to any classic packer ( i.e. winzip etc )

ir is based on words, ranging from 1bit to n bits;
a word is anything like that:

say you have 3 bytes of data,
137-22-49
137=128+0+0+0+8+0+0+1
22=0+0+0+1+0+1+2+0
49=0+0+1+1+0+0+0+1

now merging data
128+0+0+0+8+0+0+1+0+0+0+1+0+1+2+0+0+0+1+1+0+0+0+1

first byte of the file=1 ( 128=8th bit=1 )
1+3+1+2+1+3+1+1+2+3+2+3+1

then print to disk
1st byte = sign of the 1st bit
2nd byte = 1
3rd byte = 3
then 1
2
1
3
1
1
2
3
2
3
1


( grmpf! hope it doesn't confuse you, you easy readers )

[Sportman]

now merging data
128+0+0+0+8+0+0+1+0+0+0+1+0+1+2+0+0+0+1+1+0+0+0+1

first byte of the file=1 ( 128=8th bit=1 )
1 + 3+1+2 +1 +3+1+1 +2+3 +2 +3 +1

One 1 become 1 and one zero become 1
How you can distinct with decoding 1 from one 1 from 1 from one zero?

Two 1's become 2 and two zero's become 2
How you can distinct with decoding 2 from two 1's from 2 from two zero's?

[Bulat Ziganshin]

Sportman, runs of 0s and 1s are going after each other

i guess that he is using non-prefix code to encode lengths

[uglyvan]

wtf??

rename words to n-length bits

with same value like in 255=1+1+1+1+1+1+1+1
now if your file begins with a 255 1st word ( after assigning a 1 for the value of the 8th bit ( 255=128+64+32+16+8+4+2+1 ) will have a min value of 8;
say 2nd byte is 137
137=128 (8th bit set )+ 8 ( 4th bit set ) + 1 ( 1st bit set )
137=1+0+0+0+1+0+0+1
merge 255+137
1111111110001001
now writing to disk
111111111=9
000=3
1=1
00=2
1=1

grmpf ugly is bad teacher :/

[uglyvan]

... by the way ...

some of them say " it's worth a rough 25 Billion Dollars !! "

- and how much money for little Yvan ??

- hmpf! 4 Billions .

[Stefan Atev]

Finally, we've defeated the pigeonhole principle! Take that, Dirichlet!

[uglyvan]

very good! thank you!!

[Sportman]

Sportman, runs of 0s and 1s are going after each other

I knew that, but I talked about how to determine the first value type of the start encode byte, because in his examples he also encode the start zero value(s), you can skip them but then you need to know how many where skipped.

Edit:
Looks like he stream it, then only the first value type need to be known, I see he named it "sign of the 1st bit".

[Sportman]

and how much money for little Yvan ??

How old are you?

[Sportman]

some of them say " it's worth a rough 25 Billion Dollars !! "

It can make some people in this forum jobless.

[JamesB]

Give up - infinite compression has already been done. Sorry, but you're slow to the market.

https://github.com/philipl/pifs

[Sportman]

Dummy14.zip start with:
0101 0000 0100 1011 0000 0011 0000 0100 0001 0100 0000 0000 0000 0000 0000 0000 0000

I expect:
(0) 1 1 1 1 5 1 2 1 1 2 6 2 5 1 5 1 1 1 30

But dummy15.txt start with:
1 7 1 7 6 2 5 3 3 5 28 4 10 6 1 7 1 7 1 7 2 6 4 4 3 5 1 9 1 5 4 4

[uglyvan]

Dummy14.zip start with:
0101 0000 0100 1011 0000 0011 0000 0100 0000 0111 0000 0000 0000 0000 0000 0000

I expect:
(0) 1 1 1 1 5 1 2 1 1 2 6 2 5 1 7 3 24

But dummy15.txt start with:
1 7 1 7 6 2 5 3 3 5 28 4 10 6 1 7 1 7 1 7 2 6 4 4 3 5 1 9 1 5 4 4

well,if my program is broken, just write your own using the philosophy I posted above; a program written in 100 lines can't be hard to rebuild

[uglyvan]

How old are you?

well, if you wanna see my face, check out "uglyvan iron maiden piece of mind" on youtube

[uglyvan]

Dummy14.zip start with:
0101 0000 0100 1011 0000 0011 0000 0100 0000 0111 0000 0000 0000 0000 0000 0000

I expect:
(0) 1 1 1 1 5 1 2 1 1 2 6 2 5 1 7 3 24

But dummy15.txt start with:
1 7 1 7 6 2 5 3 3 5 28 4 10 6 1 7 1 7 1 7 2 6 4 4 3 5 1 9 1 5 4 4

no no, there is no error here:

each word is assigned to one byte

1+7=8
1+7=8
6+2=8
5+3=8
3+5=8

[Sportman]

no no, there is no error here

Then I do not understand what you do.

When I do what I thought you do, then it's almost the same I code 15 years ago, I found some old code (without the starting type) and got:

dummy.zip 186,836 bytes to byte values 739,391 bytes with different statistics and different value list then your dummy1.txt 360,001 bytes.

Compressing:
236,690 bytes zip
186,686 bytes paq8pxd18
186,582 bytes emma
186,162 bytes cmix

[uglyvan]

Then I do not understand what you do.

When I do what I thought you do, then it's almost the same I code 15 years ago, I found some old code (without the starting type) and got:

dummy.zip 186,836 bytes to byte values 739,391 bytes with different statistics and different value list then your dummy1.txt 360,001 bytes.

Compressing:
236,690 bytes zip
186,686 bytes paq8pxd18
186,582 bytes emma
186,162 bytes cmix

then, my program must be broken

[SvenBent]

then, my program must be broken

Says kinda in the title "Unlimited Packer"...

[uglyvan]

... or, if you prefer, I have absolutely no knowledge in the fields of the business ...

[uglyvan]

... or, if you prefer, I have absolutely no knowledge in the fields of the business ...

shall I drop it as a GPL product ?

[Sportman]

I have absolutely no knowledge in the fields of the business

None of the persons who got something similar working, so never stop trying and LawCounsels shall be happy with this outcome

[Nania Francesco]

Honestly I have not well understood if the program manages to actually compress already compressed data. If so it is right to give help to those who had this idea.

[Sportman]

Honestly I have not well understood if the program manages to actually compress already compressed data.

No it always expand data, but expanded weak compressed data can be one time compressed smaller with a better compressor, after that this process can not be repeated with the same outcome, so it do not work.

[uglyvan]

... by the way ...

some of them say " it's worth a rough 25 Billion Dollars !! "

- and how much money for little Yvan ??

- hmpf! 4 Billions .

You're in the army now

[uglyvan]

hi folks !

( if you wanna make money with my stuff, I claim the 15% droits d'auteur; if worth 25 Billion then it rounds to approximately 4 Billion )