HELP! movie compression

[Lone_Wolf236]

hey all!
over the past months, i've tried a lot of compressors but couldn't find any that was able to compress an .avi movie to an acceptable ratio for the time spent.
i've been unable to gain more than 2% of the file size...

i dont understand why because the codecs are supposed to work only with the frames and pixel blocks, not re-arrange bits so its uncompressable!
why isn't my cpu working at 100% while playing a movie if its already compressed?

or maybe the compressors out there dont have the required filters...?

P.S.: i dont want to reencode my movies to other formats because i will loose quality. (and if i want to keep the same quality it will end up being the same size)


Thanks a lot!!!

[SvenBent]

your data is already compressed witch makes it hard to compress it anymore.

did you also think you could zip a zip file?

depending on cpu codec yuo might find some special tools to enhance the compression on you file og maybe compress it sligtly more. but consider this a rare chance

[Matt Mahoney]

All video formats (MPEG, MPEG2, H.264, etc) are already compressed so you can't compress them further (not much, anyway). It might be possible to write a specialized compressor to squeeze a bit more out, like stuffit or paq8 for JPEG or soundslimmer for MP3. But nobody has done that for video yet. It would likely be much slower than playing the video and I doubt you could squeeze out more than 30%.

[moisesmcardona]

I've managed to compress videos and save at least 10MB using RZM Compression. 10MB is just 10MB of free space!

[SvenBent]

about the cpu usage.
its the simple facts you don decompress AS FAST as possible but only as fast as needs to playback the video.

a 2 hours movie its ok to decompress in only 2 hours thereby not needing to use the entire cpu for the two hours


if you decompressed the movie you would se the cpu usage climb as the decompression now goes as fast as possible and it will be done faster then playback time.

[SvenBent]

I've managed to compress videos and save at least 10MB using RZM Compression. 10MB is just 10MB of free space!

i manage to compress a video with 7-zip and save several gigabytes.
but the input files was in tereabytes

measuring compression gains in bytes without giving af full size doesn't really tell anything

if you save 10mb if a 11mb file thats impressive. but if you save 10mb of a 8gb file, then its not so impressive

[moisesmcardona]

i manage to compress a video with 7-zip and save several gigabytes.
but the input files was in tereabytes

measuring compression gains in bytes without giving af full size doesn't really tell anything

if you save 10mb if a 11mb file thats impressive. but if you save 10mb of a 8gb file, then its not so impressive

I saved 10MB of really low quality Xvid Movies (from 100MB to 400MB each video of 1 hour to 3 hours).

It seems that compressing Slideshows will save a lot of MB since I compressed a 600MB Slideshow and the result was 200MB only.

[Shelwien]

> over the past months, i've tried a lot of compressors but
> couldn't find any that was able to compress an .avi movie
> to an acceptable ratio for the time spent.

Yeah, obviously

> i've been unable to gain more than 2% of the file size...

Now, this is worse than usual

> i dont understand why because the codecs are supposed to
> work only with the frames and pixel blocks, not re-arrange
> bits so its uncompressable!

Still, all these codecs not only do the lossy media processing,
but also apply some (inefficient, but fast) compression algorithms.

> why isn't my cpu working at 100% while playing a movie if its already compressed?

That's unrelated

> or maybe the compressors out there dont have the required filters...?

Yeah, in a way.
The main problem is that to apply better compression we need to
remove the existing one first - and then reapply it
back without any loss.
And then there're containter formats (avi in this case) and multiple video
codecs (most of which are not open-source), and audio codecs.
Anyway, the fashion changes much faster than one could write a working
recompressor for a specific video format.

> P.S.: i dont want to reencode my movies to other formats
> because i will loose quality. (and if i want to keep the
> same quality it will end up being the same size)

You can try this:
http://shelwien.googlepages.com/avip_demo_1.rar
(its an avi parser; its output files should be more compressible
than original file).
Then you can additionally compress the audio track with this:
http://soundslimmer.com/
if its in mp3 though.

[Lone_Wolf236]

i tried the avi parser, it gave me a 500mb file and 5 others that are 0kb
i used my test file that is 25mb

how do you put the files back together now?

(compressing with kgb now... 1,4%)
(winrar gave me an archive larger by a couple Kb)

[Lone_Wolf236]

freearc gave me a file 40kb larger than original

[moisesmcardona]

try RZM.
for simplicity, I would recommend my software, PerfectCompress, which can compress to a lot of formats. That way, you can choose which format compresses the best for you.

http://moises-studios.110mb.com/perfectcompress

[Lone_Wolf236]

nanozip ended 202kb smaller or 0,788% smaller not worth it...
kgb was bigger than nanozip so i stopped it

i guess i will have to leave them as avi...
i dont know if its the right forum for this but...
any good codec i could use to save space without visible loss of quality?
it also depends of the program... any suggestion?

thanks guys!

[Fu Siyuan]

You are on the right forum. But I really think there hardly be software can achieve your goals, especially for the "without loose quality".

[Shelwien]

unfortunately my avi parser only works with valid avi files.
Certainly not 25m parts of avi files or something like that.
Its purpose is to split the avi container into video, audio,
and index streams, which are better compressed when not
interleaved.

As an alternative, you can use this:
http://www.bunkus.org/videotools/mkv...icode-2.9.8.7z
to convert your avi (but only if that was really my avi parser bug,
and the avi is valid) to mkv format without reencoding the media streams.

Mkv files are usually a little more compact than avi files,
so you can expect to shrink the size of your files by a few megabytes.
But that difference is only caused by a more compact index
representation in mkv, so you shouldn't expect much.

[Scientist]

>
You can try this:
http://shelwien.googlepages.com/avip_demo_1.rar
(its an avi parser; its output files should be more compressible
than original file).
Then you can additionally compress the audio track with this:
http://soundslimmer.com/
if its in mp3 though.

Am very much interested in putting back the splitted parts. this avip_demo does perfect withy many avi videos that I tested on, when compressed the splitted parts (.mp3 files) it produced very nice results.

Question: Can you kindly direct how to put back these parts to an AVI file?

Thanks

[Shelwien]

Just read the test.bat there.

@echo off
rmdir /s /q temp
mkdir temp
cd temp
..\avi ..\Semi256.avi 
..\avi -d ..\Semi256.unp
cd ..
fc /b Semi256.unp Semi256.avi 
rar a -m5 -mdg -s 1.rar Semi256.avi 
rar m -m5 -mdg -s 2.rar temp
del Semi256.unp

Specifically, the line

..\avi -d ..\Semi256.unp

merges the 6 streams into single Semi256.unp

(..\ paths there are due to temp subdirectory used
to store the files)

[Fallon]

Avi is still popular in peer2peer, but it's on the way out, because .mkv is also up and coming for music videos.
Mkv is flexible and not tied to old uncle Gates' rules. Microsoft is planning mediaplayer support for .mov files now.
As usual they will have to be dragged to the blessing of supporting an open format.

(edited short - because it's getting off topic)
VLC-player has excellent .mkv support.
http://www.videolan.org/
Another cross platform free and open source player:
http://www.getmiro.com/download/

[Vacon]

Hello everyone,

reading 7-zip's forum, I just stumbled upon that one:
http://ck.kolivas.org/apps/lrzip/README
Anyone tested it? Seems as if there are no windows-binaries. Possibly it can be compiled by one of our gurus

Best regards!

[Bulat Ziganshin]

Vacon, rzip idea lies behind my REP filter you may consider rep as improved version of rzip

from rep docs:

REP is an LZ77-family algorithm, i.e. it founds matches and outputs them as
(len,offset) pairs. It is oriented toward very fast compression and small
memory overhead (1/4 of buffer size), but limited to rather large values of
mimimal match length (say, 32), and don't search for optimum match. It's
intended to preprocess data before using full-fledged compressors. and in
this area it beats RZIP and, to some degree, LZP preprocessors. Small
memory overhead means that RZIP/LZP/REP are capable to find matches at very
long distances and this algorithm does it much better than RZIP and LZP.
The algorithm implemented in functions REPEncode() and REPDecode().

Main differences comparing to RZIP:
1) Sliding window which slides at 1/16 of buffer size each time
2) Almost ideal hash function (see update_hash)
3) Direct hashing without hash chains which 1.5x cuts memory requirements
4) Tags are not saved in hashtable, which again halves memory requirements.
Instead, a few lower bits of hash table entry are used to save a few
bits of tag (see chksum)
5) Hash size is proportional to buffer size (which is equal to the maximum
search distance) and by default limited to 1/4 of buffer size
6) In order to find strings of length >=MinLen, blocks of length L=MinLen/2
are indexed via hash. Of all those possible blocks, only 1/sqrt(L) are
indexed and only 1/sqrt(L) are searched. It is alternative to solution
described in RZIP paper where 1/L of blocks are indexed and each block
searched. This means that logb(sqrt(L)) lower bits of hash entry are
zeroes which allows to use trick 4.


References for RZIP algorithm explanation and implementations:
http://samba.org/~tridge/phd_thesis.pdf
http://rzip.samba.org/ftp/rzip/rzip-2.1.tar.gz
http://ck.kolivas.org/apps/lrzip/lrzip-0.18.tar.bz2
http://www.edcassa-ict.nl/lrzip.zip
http://www.edcassa-ict.nl/rzip21.zip

TAYLOR, R., JANA, R., AND GRIGG, M. 1997. Checksum testing of remote
synchronisation tool. Technical Report 0627 (November), Defence Science and
Technology Organisation, Canberra, Australia. (p.72)


References for LZP algorithm implementations:
http://magicssoft.ru/content/downloa...GRZipIISRC.zip
http://www.compression.ru/ds/lzp.rar

[Bulat Ziganshin]

well, after reading README i see that it uses no memory on decompression. we had such idea, it would be very interesting to see how it's implemented here. i will give it a try

[Bulat Ziganshin]

1. it seems that since lrzip 0.18 nothing important was changed and i just overseen this feature
2. make prints the following errors:

In file included from rzip.c:24:
rzip.h:37:26: sys/resource.h: No such file or directory
rzip.h:38:24: netinet/in.h: No such file or directory
rzip.h:64:22: sys/mman.h: No such file or directory
rzip.c: In function 'init_hash_indexes':
rzip.c:523: warning: implicit declaration of function 'random'
rzip.c: In function 'rzip_chunk':
rzip.c:534: warning: implicit declaration of function 'mmap'
rzip.c:534: error: 'PROT_READ' undeclared (first use in this function)
rzip.c:534: error: (Each undeclared identifier is reported only once
rzip.c:534: error: for each function it appears in.)
rzip.c:534: error: 'MAP_SHARED' undeclared (first use in this function)
rzip.c:547: warning: implicit declaration of function 'munmap'
make: *** [rzip.o] Error 1

so it just needs to support a Windows way of mmapping files in order to compile. afaik, old lrzip was compiled by someone for mfc test

[Skymmer]

I've became interested in compression of already compressed data some time ago, when I was playing with resources from Fallout 3 RPG. I don't mean those BSA containers but the data extracted from them. There are a lot of compressed files there. Basicly OGG files, some MP3s and some movies in BIK format. I've tried to pack them with different archivers\compressors and as far as I remember the best one was Squeez.
Since we talking about movie compression here I decided to conduct a modest test using 2 movies and some compression tools. Both movies in AVI v2.0 (OpenDML) container.
First one is the The Good, The Bad and The Ugly film encoded with Divx 5.2.1 (528 KBit/s) + MP3 48000 Mono VBR (61 KBit/s).
Second one is Lost (TV-series, Season 5 Episode 1) encoded with XviD (1008 KBit/s) + MP3 48000 Stereo CBR (128 KBit/s).
Here are the results and settings used. Results sorted from worst to best.

The Good, The Bad and The Ugly

                                         Original = 734 599 168
                            RAR v3.90 -m5 -md4096 = 712 107 185
   FreeARC v0.60 -mlzma:d512m:h96m:lc3:fb64:mfht4 = 712 022 813       new
  FreeARC v0.60 -mlzma:d384m:h256m:lc3:fb64:mfht4 = 711 729 518       new
  FreeARC v0.60 -mlzma:d112m:h128m:lc3:fb64:mfhc4 = 711 619 169       new
                                7-zip v9.07 Ultra = 710 222 137
                7-zip v9.07 -m0=LZMA2:d=64m:fb=64 = 710 098 012       new
                        NanoZip v0.07a -co -m512m = 708 650 433
                                       RZM v0.07h = 704 331 622
WinRK v3.1.2 ROLZ3, 256MB\512\64\24\fast analysis = 704 056 954
WinRK v3.1.2 ROLZ3, 256MB\512\64\24\full analysis = 704 038 021
        Squeez v5.63 -fmt SQX2 -m5 -MD32768 -uxx9 = 699 899 645
                                     CCM v1.30c 5 = 697 420 919
                                    CCMx v1.30c 5 = 697 329 727
                        NanoZip v0.07a -cO -m512m = 695 996 832
                                  BCM v0.09 -b140 = 687 711 810       new
                        NanoZip v0.07a -cc -m512m = 680 159 944      -7.41%

Lost

                                         Original = 360 011 776
  FreeARC v0.60 -mlzma:d112m:h128m:lc3:fb64:mfhc4 = 357 221 787       new
  FreeARC v0.60 -mlzma:d384m:h256m:lc3:fb64:mfht4 = 357 166 574       new
                                7-zip v9.07 Ultra = 357 071 721
                            RAR v3.90 -m5 -md4096 = 356 699 397
                7-zip v9.07 -m0=LZMA2:d=64m:fb=64 = 356 505 767       new
                        NanoZip v0.07a -co -m512m = 355 764 716
WinRK v3.1.2 ROLZ3, 256MB\512\64\24\full analysis = 354 137 950
                                       RZM v0.07h = 354 097 489
WinRK v3.1.2 ROLZ3, 256MB\512\64\24\fast analysis = 353 769 930
                                     CCM v1.30c 5 = 352 128 887
                                    CCMx v1.30c 5 = 352 122 686
        Squeez v5.63 -fmt SQX2 -m5 -MD32768 -uxx9 = 351 555 517
                        NanoZip v0.07a -cO -m512m = 350 597 110
                                  BCM v0.09 -b140 = 348 240 300       new
                        NanoZip v0.07a -cc -m512m = 344 675 819      -4.26%

Results are talking for itself, isn't it ? But if I were you I would stick with Squeez. The reasons are simple. Squeez is very fast on decoding while NanoZIP is not (especially -cc). CCM is completely symmetric and furthermore its a compressor.

EDIT: Added new results (strings marked with 'new' word).

[Bulat Ziganshin]

try lzma2 too

[Skymmer]

OK, I've added some new results to my previous post (strings marked with 'new' word). Surprisingly good results for BCM.