GDC Competition: Discussions

[Dresdenboy]

Error 0xC0000005 means only writing to unallocated memory, not reading?

Could be both. Reason could for example be accessing something via a freed pointer or outside of allocated memory arrays.

[lz77]

That result sounds good!

Thanks. If I knew in June where is the right category for my participation, I would make a preprocessor and write these dll's in FASM for Win64...

You might need to build a very small C console app to test it. The free Visual Studio Express might help here.

Unfortunately I haven't practiced in C yet. Maybe someone will send me this C test program for CodeBlocks... Here is my Delphi 7 example. It gives dll contents of file named "bl00.dat" and writes the compressed data to a file named "packed".

P.S. packedsize is a var parameter and passed to dll by pointer. Here's a declaration from the dll:

type pb = ^byte;
...
function encodeRun(inSize: dword; inPtr: pb; var outSize: dword; outPtr: pb; cmprContext: pointer): longint; cdecl;

[Shelwien]

Kennon Conrad just posted his framework in other thread - https://encode.su/threads/1909-Tree-...ll=1#post67549
You can rename your dll to GLZA.dll and see what happens.
Reposting the binaries here, because he missed a mingw dll.

[lz77]

I just saw where the error is: It happened while adapting my subroutine to the conditions of the competition. I confused the name of one variable.

[Dresdenboy]

[lz77]

You can rename your dll to GLZA.dll and see what happens.

Hm, when I run TestGLZAdll.exe in test.bat for my compress dll renamed to GLZA.dll, I get error "TestGLZAdll.exe - application error 0xc000007b"...
The test works correct with original GLZA.dll.

Oh, the original GLZA.dll also contains both decodeInit & decodeRun, but I have separate dll's for encoding and decoding. Perhaps this is the reason for the error...

[lz77]

Did you manage to finish something?

I just yesterday fixed the last hard-to-find bug that occurred while adapting the dll to the conditions of the competition. My dlls are accepted for participation. The result is better than ULZ, but weak. Hope to take 5th place...

My from scratch implementation with hash based LZ77 and some form of my own ANS lands at ~126*10^6 bytes in 6 s for the open part without much tuning and throwing out some more complex ideas.

What is the ratio without using ANS? What did you squeeze with ANS: literals and higher 8 bit offsets?

How does zstd compress 1 GB to 280 MB in rapid test? Does it use any historical buffer for this?

[Ms1]

Hope to take 5th place...

There are only 5 entries in this category to my regret. It's extremely improbable to change because other people still having issues with libraries aim at slower categories.
Thus definitely no lower than 5th.

How does zstd compress 1 GB to 280 MB in rapid test?

That would be too early for me to make comments on such questions, but aren't you surprised by the zlib result? It's not far from Zstd in terms of compression ratio.

[lz77]

but aren't you surprised by the zlib result? It's not far from Zstd in terms of compression ratio.

Not so surprised because zlib compresses it 7.3 sec. longer than zstd.
I'm surprised by pglz's c_time in Test 1, text, Rapid.

Here it is their moral: they are ready to do everything for money!

[Ms1]

Here it is their moral: they are ready to do everything for money!

Yes, they are ready to insert a dictionary/model for our dirty data inside their neat compressors.

Test 4, decompressor_size, bytes:

*pglz	3446477
*k5	202450
*babylz	9014

*pgcm	3145170
*LUNA	23447
*PPMd	1413982

*PPMonstr	1599079
*NINO	85981
*sgcm	3150364

[lz77]

I guess the remaining €2000 should be awarded to the author of babylz for the minimum (de)compressor size.

[lz77]

Suppose one programmer did not adapt zstd but used his own algorithm and won a prize. Next year he will want to win a prize again, but if his binaries will be published, it will be hacked up and his algorithm will be stolen. It seems, in this case something is wrong in the rules of the competition...

[e8c]

algorithm will be stolen

Computer Science - это прежде всего Science, а наука - это конкуренция идей, а не людей: люди заняты общим делом - поиском оптимальных решений. Тот факт, что GDCC не требует исходный код, делает его малополезным с точки зрения научного прогресса (а по деньгам для победителей - это копейки).

[lz77]

14-15 thousand euros for a single participant is it really a penny?

[Shelwien]

1) See Hutter Prize contest - published binaries didn't change anything since 2006.

2) Paying somebody every year for the same code is not interesting at all, even if its incrementally improved.

3) There're technical methods of anti-RE protection

4) We're not going to repeat exactly the same tasks every year. Eg. see dedup contest in Poland

[lz77]

3) There're technical methods of anti-RE protection

Give examples of protection from the study of the program algorithm, so that antihemorrhoid suppositories cost more than the results of this study.

[Shelwien]

> Give examples of protection from the study of the program algorithm

1) Stuff like http://vmpsoft.com/products/vmprotect/

2) Just use any sufficiently complicated unusual code structure
- for example, nobody really decompiled reflate because of coroutines
- I'd expect same issues with RE of MT frameworks
- frameworks with a hierarchy of classes with virtual methods are very annoying for RE
- there're anti-debug tricks that could be explicitly added (like running dynamic copies of functions in heap memory, instead of static ones)
- linux binaries are harder for RE, since public tools are mostly windows-only
- inlined/unrolled AVX+ SIMD code is very annoying already as it is, since even newest IDA doesn't have intrinsics for AVX+

[lz77]

For GDC only these items are suitable:

Just use any sufficiently complicated unusual code structure.
- inlined/unrolled AVX+ SIMD code is very annoying already as it is, since even newest IDA doesn't have intrinsics for AVX+.

[Ms1]

As for source codes. What would be chances of getting a number of strong submissions if source codes would be required? It looks so fishy giving all the circumstances, I would not be a part of it.

As for binaries. If we would not disclose binaries at least for winners, then the entire event would look like a joke. There would be no way to cross-check the results.

[standolf]

With the gdc competition now completed, I was trying to access some of the published algorithms, in order to test their claimed capabilities.
The first one I tried (`pglz`) would not compress anything, instead just returning a copy of input data with an additional 4 bytes header.
Is that expected ?
What's the point of allowing to download the contestant's entries if we can't test them ?

[hexagone]

Did you try with TS40.txt ? It works. My guess is that this compressor is customized to the GDC text data and just creates a copy for any other input.

[standolf]

Do you mean test data from the gdc2020 ?
I did not find a way to download it...

[hexagone]

From the GDC website, http://globalcompetition.compression.ru
You will have to register (form at the bottom).

[algorithm]

pglz needs .txt filename extension for text compression and .dat extension for data compression.

[lz77]

Is a similar competition planned for 2021?

[Shelwien]

We don't know yet, ask again in June.
For now, there's this: https://coronavirus.innar.com

Edit: moved coronavirus compression posts to https://encode.su/threads/3548-SARS-...sion-Benchmark

[e8c]

There is no readme.txt for many compressors from the GDCC site.

UPD: Attached "readme.txt" from top-secret "compressors.7z", which was sent to participants 05.12.2020

[Hakan Abbas]

GDCC was a very serious and excellent organization. I hope it will happen again.
In the process, I conveyed my thanks to the competition committee and Maxim Smirnov. By the way, now I thank you once again from here.
I was able to partially get together what I've been working on (images) for a long time with GDCC. Although I could not be very successful at the point of fast implementation, it was a great experience for me in other respects...

[hexagone]

GDCC was a very serious and excellent organization. I hope it will happen again.
In the process, I conveyed my thanks to the competition committee and Maxim Smirnov. By the way, now I thank you once again from here.
I was able to partially get together what I've been working on (images) for a long time with GDCC. Although I could not be very successful at the point of fast implementation, it was a great experience for me in other respects...

Fully agreed. And Maxim deserves some credit for his no non-sense approach to some questionable entries...