FastBackup: yet another deduplication engine

[Bulat Ziganshin]

i've started development of one more zpaq/exdupe clone. the very first version that just deduplicates data in memory and doesn't write any archive, made available as http://freearc.org/download/research/fb001.zip

but i need your help. i'm poor on good program names. can you suggest me a good one? generally speaking, i plan to make all-in-one incremental backup tool, with dedupe, compression, encryption, recovery record and so and so

[FatBit]

Dear Mr. Ziganshin,

Data Squeezer is suitable?

FatBit

[joerg]

the program is really very fast - congratulation!

- as a name for the program i would suggest "fastback" or "fastbak"
- recovery record as in rar sounds wonderful
- maybe a special variant of a wellproofed archive-format like 7z - but only 1 or 2 compression methods?
best regards

[Bulat Ziganshin]

Incremental backups (at least in the zpaq/exdupe style) requires new archive format. It's why i can't just add new features to freearc/7-zip. Also, "Data Squeezer" name looks more appropriate for archiver rather than backup tool

Now i have names fastback and sciback (both "scientific" and "sky"-like, meaning "cloud backup"). more ideas please!

[Sportman]

more ideas please!

First part of your family name, zigzag files together with fast speed and name is not in use accept one gmail account.

Zigzagfast

[Bloax]

Zigzagfast

ZippyDeDup ~ Use it for backup :^)

[nburns]

How would you explain the difference between deduplication and compression?

[Bulat Ziganshin]

dedup engines splits input data into content-defined chunks and stores SHA hash of every chunk in the archive. then new data are checked against existing chunks and only new chunks are added to the archive

[encode]

FreeArc ... FreeBackup? Or BulletBackup (Backup from Bulat)

[Bulat Ziganshin]

BulatBackup looks interesting..

[Paul W.]

ZippyDeDup ~ Use it for backup :^)

I kinda like this one, but a lot of non-native English speakers probably won't get it. (A punny reference to "Zip ah dee do dah", made famous by a Disney movie.) I don't know if kids these days watch "Song of the South," either.

https://www.youtube.com/watch?v=LcxYwwIL5zQ

The reference would be clearer if it was "ZipADeDup," but probably still not recognizable enough.

[Matt Mahoney]

I would just call it freearc and add it as a new format.

[fcorbelli]

sr (sleepright)
---
I suggest some ideas for "definitive" backup: create two files.
One index, one (or more) data.
Data file always grow, but does not change (aka: append only new data).
This is great for rsync (with or without --append)

Index file for very fast extracting of files (the main "limitation" of zpaq).
Into datafile insert a recovery header (for the chunks) just in case index is lost
(or you don't need fast extraction).
---
Eventually split (basing on SHA's) on more than one datafile.

Example: when adding
file1.txt hash aba12345...
file2.txt hash aa0...
file3.txt hash aa3...

add the first in
ab_backup.sr

second and third in
aa_backup.sr

If you have a very big backup, you can easily split (sharding) without need to "choose" where to add.
Using 0,1,2 or 3 chars\level (in HEX) you will get 1, 16, 256 or 4096 set.

[zyzzle]

I also think you should add it to freearc as a new format, independent of .arc, even compile it as an option into the arc.exe file.

[fcorbelli]

Here's the very early test on my source-tree
[code]
F:\zarc>z:\fb64.exe -t4 f:\zarc
4+2 threads * 8mb buffers, 48b..4kb.. chunks
Scanning: 6,238,754,776 bytes in 32,976 files
100.0% deduplicated: 6,238,754,776 => 3,952,103,894 bytes

F:\zarc>zpaq64 a r:\temp\kao f:\zarc\*.* -method 0
zpaq v6.51 journaling archiver, compiled May 7 2014
....
0 + (6238754776 -> 4164812809) = 4164812809
93.907 seconds (all OK)[/quote]
As you can see deduplication seems good (better than zpaq's)

[Piotr Tarsa]

How would you explain the difference between deduplication and compression?

Deduplication only looks for repeated sequences which are above some quite big threshold in length. Compression usually also involves making statistics and/ or looking for short repeated patterns.

[Matt Mahoney]

zpaq uses 64 KB average fragment size by default. To test 4 KB fragments use option -fragment 2

Also, there is already a compressor named sr (symbol ranking).

[nburns]

Also, there is already a compressor named sr (symbol ranking).

If sr is taken, the logical second choice is jr.

[nburns]

dedup engines splits input data into content-defined chunks and stores SHA hash of every chunk in the archive. then new data are checked against existing chunks and only new chunks are added to the archive

That's pretty much what I thought, but that's just a kind of dictionary compression.

Are you familiar with git's object database? The applications probably don't overlap perfectly, but it's probably worth studying git's design decisions and how they turned out. Backup and revision control are not thought of as the same, but there is a fair amount of common ground IMO.

[Bulat Ziganshin]

As you can see deduplication seems good (better than zpaq's)

they are exactly the same since i'm using zpaq's chunking algo the difference is due to using smaller chunk size by default and lack of archive index

i plan to beat zpaq on the speed side as well as flexibility, but deduplication efficiency probably will remain the same

[thometal]

Backup ULtimATive

[Bulat Ziganshin]

why not just Backup ULtimATe?

[thometal]

Do you also want define a decompression language like zpaql??? So its possible to create an own custom compression algorithnms? So its possible to decode for everyone?

[thometal]

why not just Backup ULtimATe?

if its more correct in english then ok, i thought ultimative would sound nicer

[Bloax]

The least cheesy solution would be to integrate it into FreeArc, yes.

[Matt Mahoney]

Using a fixed format like tornado should be faster than a decompression language like zpaq. The disadvantage is losing compatibility when you want to improve the compression. I think this is more of a problem for high end compression like CM where you are more likely to make changes.

[Nania Francesco]

For me ..
FLASHDUP
or
STORMDUP

[fcorbelli]

FatBackup?

[nburns]

bzig2 -- from your first initial and first three of last name, and "2" because it's your second backup program. Do you think it sounds original enough?

[Piotr Tarsa]

Maybe something Hollywood sounding, like: The Synchronizer.