Needs ready to run C# Arithmetic Coder to simple accept input Symbols

[LawCounsels]

Hi :

any recommendations for a simple basic ready to run C# Arithmetic Coder module/ DLL , to simple accept input Array of symbols ( eg 1000 symbols , 500 symbol A 250 symbol B 125 symbol C 125 symbol D ) to produce Output value ( eg IntX , BigNum ) ?

Thanks in advance,
LawCounsels

[Shelwien]

github search gave me this:
https://github.com/larryk78/SevenZip...oderBitTree.cs

[LawCounsels]

compiling gives 11 errors here :

Error 1 The type or namespace name 'BitEncoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 7 9 ConsoleApplication50


Error 2 The type or namespace name 'Encoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 22 28 ConsoleApplication50


Error 3 The type or namespace name 'Encoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 34 35 ConsoleApplication50


Error 4 The type or namespace name 'BitEncoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 74 46 ConsoleApplication50


Error 5 The type or namespace name 'BitEncoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 89 42 ConsoleApplication50


Error 6 The type or namespace name 'Encoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 90 13 ConsoleApplication50


Error 7 The type or namespace name 'BitDecoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 105 9 ConsoleApplication50


Error 8 The type or namespace name 'Decoder' does not exist in the namespace 'SevenZip.Compression.RangeCoder' (are you missing an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 120 39 ConsoleApplication50


Error 9 The type or namespace name 'Decoder' does not exist in the namespace 'SevenZip.Compression.RangeCoder' (are you missing an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 128 46 ConsoleApplication50


Error 10 The type or namespace name 'BitDecoder' could not be found (are you missing a using directive or an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 142 42 ConsoleApplication50


Error 11 The type or namespace name 'Decoder' does not exist in the namespace 'SevenZip.Compression.RangeCoder' (are you missing an assembly reference?) c:\users\owner\documents\visual studio 2012\Projects\ConsoleApplication50\ConsoleApplicat ion50\Program.cs 143 24 ConsoleApplication50

[Shelwien]

There're other modules around that.
https://github.com/larryk78/SevenZip...ess/RangeCoder

[LawCounsels]

I can on only compile neat simple things

can someone show all included source codes all in all place , ready to simple one-click compile , no complications ?

[Shelwien]

Well, you can try this one i guess.
http://www.codeproject.com/Articles/...ression-With-C
The previous coder was better though - this one looks like "old-school" bitwise arithmetic coder, slow and imprecise.

But overall, java/C# are very rarely used for data compression, due to inherent cpu/memory overhead,
so there's not much choice.

[LawCounsels]

incidental , any other ready to run compact neat simple DLLs ( C# callable ) ?

[Shelwien]

Maybe https://github.com/Janiels/LzmaNet

[LawCounsels]

is there any good fast precise implementations of AC in eg C C++

either port the above to C# callable module ( immediate runs adapt to take int[ ] Array of symbols & # of Array elements evaluate probabilities, OPTIONAL takes fixed preassigned probabilities ... returns IntX output, DECODE takes IntX etc returns Array of Symbols & # of Array elements)

OR

take the larryk78 & make into immediate runs C# module

$70 token of appreciation ( by Western Union) to the 1st correct working solution

LawCounsels

[LawCounsels]

there appears surprising almost no ready to run simple basic fast precise C# Arithmetic Coder out there for a long time. been a real impediment to utilising Arithmetic Coder

hope this helps address the long felt scarcity

[hexagone]

Arithmetic coder in C# ? I guess your compression breakthrough did not work, uh ? Shocking.

[Jarek]

James has fast C implementation of arithmetic coder: https://github.com/jkbonfield/rans_static

[LawCounsels]

James has fast C implementation of arithmetic coder: https://github.com/jkbonfield/rans_static

I appreciate this is a good ANS improvement of Arithmetic Coder , but in C

do you care to quick port to C# callable module ( parameters passing ) & share with forum ?

[LawCounsels]

Arithmetic coder in C# ? I guess your compression breakthrough did not work, uh ? Shocking.

hi it works ! ... now touches up with Arithmetic Coder function

LawCounsels

[LawCounsels]

I appreciate this is a good ANS improvement of Arithmetic Coder , but in C

do you care to quick port to C# callable module ( parameters passing ) & share with forum ?

Jarek : kindly confirm , either port this OR from your other ANS source ... will reserve this for when you confirm

LawCounsels

[Jarek]

Sorry, I haven't worked with C#.
James has implementation of AC, but if you want ANS, there might be some easier to port to C# (I can add yours): http://encode.su/threads/2078-List-o...mplementations

[JamesB]

Sorry, I haven't worked with C#.
James has implementation of AC, but if you want ANS, there might be some easier to port to C# (I can add yours): http://encode.su/threads/2078-List-o...mplementations

Mine is just a restructuring of Shelwien's to make it work on static probabilities rather than an adaptive/dynamic affair. It makes it MUCH faster, but obviously it's not applicable for use with complex models as the model probabilities need storing too.

As for C#, I don't know it, but AC is conceptually pretty short. Can you not simply port over some C code to C#? I know this was done for the ANS C code to Java and it lead to a reasonably quick ANS implementation there too. The tricky bit with these algorithms is getting all the overflow cases nailed, so porting an existing algorithm helps to avoid those pitfalls.

[LawCounsels]

I now have ACEncode part , which takes in screen input string of symbols & prompts for the respective symbols frequency of occurence in input string ... this works

C:\Users\Owner\Documents\ArthimeticCoder2\Arthimet icCoder\ConsoleApplication2\bin\Debug>consoleappli cation2 2122234213
Do you want enter the occurance /*** lowercase y/n only , uppercase Y/N will be ignored ***/
n
Before Compression :2122234213
Compressed (3 bytes) :
62253160 /*** this the output string concatenating ALL bytes from memory-stream ****/
Expanded :2122234213
^C
C:\Users\Owner\Documents\ArthimeticCoder2\Arthimet icCoder\ConsoleApplication2\bi
n\Debug>

Problem is when trying to make standalone ACDecode class , the C# implementation at present takes from memory-stream Array ( sometimes each Array element has 2 digits sometimes has 3 digits ) wont be able take from earlier returned output string concatenated ALL the digits from memory !!!!
mem_test = c.compress(test);
mem_test is the memory stream which is compressed

now, next part is concatenation from the memory stream array
foreach (byte b in mem_test.ToArray())
compressed += b.ToString();

the problem is memory stream is actually an array of streams,
[0] = 32
[1]=53
[2]=201
[3]=217
[4]=172

so compression string is atually 3253201217172, which is just concatenating all the elements of the array,

WHAT IS ( ??? ) the reverse logic for creating stream array out of compression string, like how to know that first index will have 2 digits, 3rd index may have two digits, this is random , and if memory stream changes, the output changes.

( it appear SOMEHOW LIKE Encoder checks 1st 2 digits , if >= eg some fixed 30 threshold it writes to memory stream as 2 digits , WHEREAS if < threshold it waits & write as 3 digits ... something like this ...

how exactly ( ??? ) it decides to write to memory stream as 2 digits , when it decides to wait then write as 3 digits . Knowing this decision logic can then similar correct reconstruct memory stream from input parameter "compression string" , to pass as input parameter to ACDecode )

[LawCounsels]

http://www.codeproject.com/Articles/...ression-With-C

this the original C# codes

[zombie28]

WHAT IS ( ??? ) the reverse logic for creating stream array out of compression string, like how to know that first index will have 2 digits, 3rd index may have two digits, this is random , and if memory stream changes, the output changes.

There is no reverse logic, because you achieved so high compression density that it cannot be reversed.
It's just like crossing event horizon of compression black hole - there is no way back.
BUT you can use antigravity shield of type 2 to protect yourself, just write:

compressed += b.ToString("X2");

[LawCounsels]

WHAT IS ( ??? ) the reverse logic for creating stream array out of compression string, like how to know that first index will have 2 digits, 3rd index may have two digits, this is random , and if memory stream changes, the output changes.

( it appear SOMEHOW LIKE Encoder checks 1st 2 digits , if >= eg some fixed 30 threshold it writes to memory stream as 2 digits , WHEREAS if < threshold it waits & write as 3 digits ... something like this ...

how exactly ( ??? ) it decides to write to memory stream as 2 digits , when it decides to wait then write as 3 digits . Knowing this decision logic can then similar correct reconstruct memory stream from input parameter "compression string" , to pass as input parameter to ACDecode )

attach, Encoder is called 1st then Decoder separately ( passing it the "compression string" as parameter , now works ... apparent this uses 'byte' ( faster ) instead of 'bit' renormalisation so will write out on 'threshold' value 256 )

C:\Users\Owner\Documents\ArthimeticCoder4\Arthimet icCoder\ConsoleApplication2\bi
n\Debug>consoleapplication4 21222342134 11 n
Input : 21222342134
Occurance for 0 is 0
Occurance for 1 is 2
Occurance for 2 is 5
Occurance for 3 is 2
Occurance for 4 is 2
Compressed String : 5413024
String Decoded : 21222342134

string result = ACEncode(inputArray, arrayCount, args[2], largestSymbol, occuranceArray);
string decode = ACDecode(IntX.Parse(result), arrayCount,occurance.ToArray() ); /*** for Decode , at present needs to know pass it "arrayCount" the decoded target length of
symbols Array ***/

its common for Arithmetic Coder to indicate to decoder where to terminate by inserting an extra ending '0' bit .... can we do away with passing "arrayCount" to decoder so it knows when to terminate after this # of symbols produced , AND instead just indicate '0or1' whether to terminate with the 1 shorter or the 1 longer ? ( assume IF without "arrayCount" there could be only 2 ambiguous decode results ? )

WHAT IS THE AMOUNT OF VERY SMALL 'SHORTFALL' FROM 'PERFECT' COMPRESSION EFFICIENCY FROM 'BYTE' RENORMALISATION ?

[LawCounsels]

ALSO just seen here this particular C# implementation not even good at all when trivial small # of symbols , anyone knows a quick fix quick fix to apply ?

C:\Users\Owner\Documents\ArthimeticCoder8\Arthimet icCoder\ConsoleApplication2\bin\Debug>consoleappli cation8 10 2 n
Compressed String : 144
Decode : 10
^C


C:\Users\Owner\Documents\ArthimeticCoder8\Arthimet icCoder\ConsoleApplication2\bin\Debug>consoleappli cation8 01 2 n
Compressed String : 80
ToString len: 1
^C
C:\Users\Owner\Documents\ArthimeticCoder8\Arthimet icCoder\ConsoleApplication2\bin\Debug>consoleappli cation8 0 1 n
Compressed String : 64
Decode : 0
^C


C:\Users\Owner\Documents\ArthimeticCoder8\Arthimet icCoder\ConsoleApplication2\bin\Debug>consoleappli cation8 1 1 n
Compressed String : 64
Decode : 1

[LawCounsels]

Sorry, I haven't worked with C#.
James has implementation of AC, but if you want ANS, there might be some easier to port to C# (I can add yours): http://encode.su/threads/2078-List-o...mplementations

Jarek :

I assume ANS numbers range is identical to range coder's , for the same input symbols set ?

is it true then the final results state number can possible be made adaptable if wishes to be anything from 0 to Max number ( so reduces by 1 bit average ), whereas range coder's is restricted to those final range numbers ?

further can each & everyone of ANS state numbers possible be adaptable such that eg ODD ANS numbers correspond only those ODD input ( if binary ) ?

what is the approx complexity order of ANS vs range coder's ?

[Jarek]

ANS and AC are very different, there are mainly used two different variants of ANS:

- tANS, like FSE ( https://github.com/Cyan4973/FiniteStateEntropy ) - fully tabled, decoding step is just

tmp = decodingTable[X]; useSymbol(tmp.symbol); X = tmp.newX + readBits(tmp.nbBits)

It is rather for static probability distribution, as modifying the tables is costly.

- rANS, uses 1 multiplication per symbol, however it is much easier for adaptation (changing probability distribution) and a bit more memory effective
http://encode.su/threads/1870-Range-...r-range-coding
here is decoding step with 16 bit renormalization and 32 bit state x:

s = symbol ( x & mask )
x = f[s] * (x >> n) + (x & mask) - CDF[s]
if(x < (1 << 16)) x = (x << 16) + read16bits();

where
f[s] is probability of s multiplied by 2^n
mask = 2^n - 1
CDF[s] = sum_{i<s} f[s]
s = symbol(x) such that CDF[s] <= x < CDF[s+1]
symbol() is usually tabled, but there are also different options like alias method or direct search (SIMD, especially for small alphabet)

[LawCounsels]

there is this C# Arithmetic Coder , quite good and simple : http://www.lomont.org/Software/Misc/ArithComp/ArithComp.html

/* Arithmetic compression in C#
A simple, single-file C# arithmetic compressor/decompressor
Source version 0.5, April, 2009 Copyright (C) 2009 Chris Lomont */

However , there is a slight 'bug' . Can someone help recommend a fix :
C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 11111111111111111111111111111111111111111111111111 111111111111
11111111111111111111111111111111111111111111111111 111111111111111111111111111111
11111111111111111111111111111111111111111111111111 111111111111111111111111111111
11111111111111111111111111111111110000000000000000 000000000000000000000000000000
00000000000000000000000000000000000000000000000000 000000000000000000000000000000
00000000000000000000000000000000000000000000000000 000000000000000000000000000000
00000000000000000000000000000000000000000000000000
Encoded String : 11025525525525525525417767015718111413561192031421 2015113314921
30160252416918214792082021015025219564
Decoded String : 11111111111111111111111111111111111111111111111111 1111111111111
11111111111111111111111111111111111111111111111111 111111111111111111111111111111
11111111111111111111111111111111111111111111111111 111111111111111111111111111111
111111111111111111111111111111111000/*** 0s were truncated shortened , otherwise correct ***/

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 000000000001111111111111122222222222
Encoded String : 11100083210489223164
Decoded String : 1111111111111122222222222 /******* 0s were truncated otherwise correct ******/

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 1111111111111100000000022222222222
Encoded String : 11025525525520324517627202
Decoded String : 111111111111110/***0s were trucated shortened otherwise correct ****/22222222222

PLS CHECK HOW '0' CAME TO BE PROCESSED DIFFERENT ( ERRONEOUS ) FROM OTHER SYMBOLS '1' TO '9'
================================================== =====================================

THATS ALL DONE

you can further see here '0's treated different ( erroneously ... can easy trace the logic for '0'), both results in exact same compressed string :

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 0000000000000000111111111111111
Encoded String : 11100063192
Decoded String : 111111111111111

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 111111111111111
Encoded String : 11100063192
Decoded String : 111111111111111

>>you can further see here '0's treated different ( erroneously ... can easy trace the logic for '0'), both results in exact same compressed string :

>>whether this error occurs while Encode , or while Decode , or both ?

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 11110001111
Encoded String : 11024417818263
Decoded String : 1111001111

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 11111111
Encoded String : 110255605
Decoded String : 11111111

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 111101111
Encoded String : 110245177172
Decoded String : 111101111

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 1111000000001111
Encoded String : 11024417818263
Decoded String : 1111001111

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>consoleapplication2 11110010100011
Encoded String : 110244168248251128
Decoded String : 11110010111

C:\Users\Owner\Documents\RangeCoder1\RangeCoder\Co nsoleApplication2\bin\Debug>

[zombie28]

However, there is a slight 'bug'. Can someone help recommend a fix

First of all, your method of converting bytes to string and back to byte array is irreversible in most cases and I already gave you some hint how to fix it.

[Hint #2]: You cannot take individual bytes and concatenate their decimal string representations together, because you won't be able to separate them unambiguously. For example, if you have a sequence: "12345", you cannot tell how to split it into bytes - it can be { 1, 2, 3, 4, 5 } or { 123, 45 } or { 12, 34, 5 } etc.

[LawCounsels]

this the Lomont C# range coder missing '0'/s 'bug' now fixed also ( was sloppy coding )

ATTACH

You run the application as command line parameter for the input value

input string : 0000000000000000111111111111111
Compressed Data string (base64): L////////28/i3gggPoDu6S5/z0zlg==
Original string: 0000000000000000111111111111111

the input number is : 123875965069560
Compressed Data string (base64): MQHNMUwjx8JXWGovtngQgA==
uncompressed string : 123875965069560

ALSO really should use the SimpleModel , uncomment & comment out :

using EncodeModel = Lomont.Compression.FastModel;
using DecodeModel = Lomont.Compression.FastModel;
//using EncodeModel = Lomont.Compression.SimpleModel;
//using DecodeModel = Lomont.Compression.SimpleModel;

present fastModel is (O)1 model

SimpleModel is simplest O(0) ie no dependendies on previous symbol encountered at all ( prefers , for minimum Header length )

WHAT IS THE HEADER FIELDS & FORMAT HERE ? WHAT ARE THE INTERNAL FREQENCIES/OCCURRENCES TABLE & VARIABLES INITIALISED BY DECODER ( reading in the Header fields ) HERE ?

[LawCounsels]

James :

do you have a fast ANS or AC, prefers minimum header overhead OR made to be none at all ( user to input symbols and original frequencies etc, to decode... So compressed file just pure compression output NO header), you can quick cobble together AND made into callable functions from within C#?

Us$150 when done. Any payment method OK

Needs just simplest static, eg 20 distinct symbols of differing frequencies in input

LawCounsels

[LawCounsels]

This offer now open to anyone the first to complete

You may email lawcounsels at all dot com

[LawCounsels]

James :

I have succeeded modify your ANS, but apparently the encoded resulting bitslength EXCEEDS Shannon's optimum length by few tens to few hundred bits...looks like this is order-1 code which adds lots of overheads

Is there a particular folder which uses static order-0 only
So resulting encoded length exceeds optimum SHANNON by only few bits ( on like 16 myltisymbol input)?