Does rep support >2GB Dictionary size ?

[SvenBent]

I'm compressing a file which REALLY like big dictionary size. how ever I'm not sure if Rep does actually support 2GB dictionary and above.

the memory utilization in task manager looks fine until i select 2gb or 2048mb. (2047mb is fine) then it drops back to only 2.1xx.xxx K and stays there no matter how much i increase dictionary size

Dict = VM size
512m = 657.688 K
1024m = 1.314.332 K
1536m = 1.839.644 k
1999m = 2.577.336 K
2047m = 2.626.584 K

2048m = 2.126.288 K
2g = 2.126.288 K
3072m = 2.126.288 K
3g = 2.126.288 K
3072m = 2.126.288 K

thats does'n look right.

and i need some LZ based compression which can have more than 2GB dictionary size. but use less then 8GB memory for compression

[Skymmer]

I'm compressing a file which REALLY like big dictionary size. how ever I'm not sure if Rep does actually support 2GB dictionary and above.

I'm not sure about possibility of REP to use 2048m or more but as far as I know you can put two or maybe more REPs in your chain

-mrep:2047m+rep:2047m+...

You can also try to include -lc- to your command line.

and i need some LZ based compression which can have more than 2GB dictionary size. but use less then 8GB memory for compression

Try arc -mlzma:dXXXXm:mfht4
Please let me know about your tests cause I don't have such amount of RAM so can't test for myself.

[SvenBent]

i dont use arc. i just use the standalone rep versioen 1.2

i triede using to reps in a row
second pass did only reduced the file with 0.2%

this is a 8.2GB iso file which by using rep -b2047mb comes down to 3.32GB

[Skymmer]

i dont use arc. i just use the standalone rep versioen 1.2

So why not to try it ? You've asked for LZ based compression and correct me if I wrong, LZMA is LZ based. Furhermore REP included in FreeARC so you can have all-in-one solution. Also. Why you think your version of REP is 1.2 ? I don't see any version neither in console output, neither in sources.

I don't know which command line you're using but you can try:

-b2047 -l512 -h28 -a99

[SvenBent]

So why not to try it ?

Because if rep is the big dictionary preprocesser/compressor in FA and that is not big enough. its probably not going to help to use Arc.
but i will try it

Why you think your version of REP is 1.2 ?

Because this is what it says when you run REP without arguments

REP preprocessor v1.2 alpha (c) Bulat.Ziganshin@gmail.com  2008-11-18

 Usage: rep [options and files in any order]
   -z      -- force compression
   -d      -- force decompression
   -oNAME  -- output filename/directory (default .rep/.unrep)
   -t      -- test (de)compression (redirect output to nul)
   -delete -- delete successfully (de)compressed input files
   -q      -- be quiet. -q[thpr]* disables title/header/progress/results individ
ually
   -v      -- verbosity level (you should recompile program with -DDEBUG to enab
le this feature)
   -cpu    -- compute raw CPU time (for becnhmarking)
   -h      -- display this help
   --      -- stop flags processing
 "-" used as filename means stdin/stdout

 Compression parameters:
   -b#     --  sliding window size (default: 128mb, recommended: half of total R
AM)
   -l#     --  minimal match len
   -d#     --  barrier for smaller matches
   -s#     --  minimal match len after barrier
   -h#     --  hash bits
   -a#     --  coefficient of search "amplification" (1 - default, 99 - max. com
pr.)

[SvenBent]

i tried using arc -mlzma:dXXXXm:mfht4 seems to not go above 1024M Dictionary size

settings = Mem usage
-d1024m = 1.877.044K
-d2048m = 1.877.044K
-d3072m = 1.877.044K

I hardly think that if you triple the dict size that you remain within 1kilybytes of mem usage.

Anyway going to let both 1024 and 3072 continue an have a look at the end file.

[Bulat Ziganshin]

lzma by itself doesn't support dicts >1gb

rep is limited by largest memory block you can alloc. try it on linux! also rep+rep usually helps in these cases, not always though

larger rep dicts may be avaliable only with 64-bit version, but i don't think that rep sources now are 64-bit ready

[SvenBent]

Thanx for the input Bulat. Just found out myself that "using" 3gb or 1GB dictionaries with lzma give exactly the same file (same size and same md5sum)


Regarding rep i don't get the out of memory message. it actually starts but just uses less memory then what my logic would say. and since it limits at the 2gb point it was thinking it was a 32bit/windows limits in rep. (I'm using WinXP 64)

Would it be hard to recompile a 64bit version which support larger dicts ?

Org size = 7,88gb
Using lzma 1gb dict = 6.52GB
Using rep 2047mb dict = 3.35 gb

is a rather odds case as i have never seen something (real life data) that likes these kind of big dictionaries

i believe the reason for is it that the iso file contains a rerelease of several game CD's whicch contains the same install files, same acrobat reader, same DirectX, etc etc.

[Bulat Ziganshin]

i don't think that rep sources now are 64-bit ready

[pat357]

Because this is what it says when you run REP without arguments

REP preprocessor v1.2 alpha (c) Bulat.Ziganshin@gmail.com  2008-11-18

Is this REP version available for download ?
Do you maybe have a change-log ?

BTW: IIRC, current 7z (4.65 and 9.04) both support dictionaries up to 1 GB. (-md30)
You'll need lot's of memory for this though!

[SvenBent]

I got it from bulat because the version did use prior had a bug.
I'm not sure if I'm allowed to reditribute it.


an LZMA with 1gb dictionarys just doesn't cut it on this file


-- edit ---
Look here
http://encode.su/forum/showthread.ph....2alpha&page=4

Bulat posts the link for the 1.2 alpha


@Bulat.
must have been blind when i asked you about the 64 bit version

[pat357]

Bulat posts the link for REP 1.2 alpha

Thanks, got it.

If someone else wants it, the full link is :
http://encode.su/forum/attachment.ph...8&d=1232632247

[Skymmer]

Bulat posts the link for the 1.2 alpha

REP preprocessor v1.2 alpha (c) Bulat.Ziganshin@gmail.com 2008-11-18

Completely but pleasantly unexpected and fascinating ! I wonder how many pearls are in the depths of encode.su