Results 1 to 7 of 7

Thread: Software file-RAID

  1. #1
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts

    Software file-RAID

    I have 6 USB 2.0 disks of 1TB that is my backup-stack.
    1,2,3 is the same and 4,5,6 is the same. All info on these disks is a copy from another disk of 1.5TB.

    Today I use Total Commander´s "synchronize dir" to copy 1 folder from 1.5TB to 1,2,3 and then 2 other folders to 4,5,6. This is much work but I want my data backup up.

    I would like someone to make a simple program that reads all files (including subfolders) from one disk (A:\FOLDER\) and write a fake backupfile to all 6 disks.
    Something like this:
    Read 64KB into buffer 1,2,3,4,5 (=320KB)
    xor buffer 1,2,3,4,5 to 6
    write buffer 1 to output file on disk 1 ( N:\DESTINAT.ION )
    write buffer 2 to output file on disk 2 ( O:\DESTINAT.ION )
    write buffer 3 to output file on disk 3 ( P:\DESTINAT.ION )
    write buffer 4 to output file on disk 4 ( Q:\DESTINAT.ION )
    write buffer 5 to output file on disk 5 ( R:\DESTINAT.ION )
    write buffer 6 to output file on disk 6 ( S:\DESTINAT.ION )
    continue reading until all files has been processed.

    Why you may ask! Disks 1,2,3,4,5,6 is USB 2.0 and as you know usb2.0 is slow, and I would like so see if this utilize more speed using all disks at the same time. I know that write speed of usb2.0 can be 40MB/s, but it´s more like 20-25MB/s in real life. So 6 disks may give a boost, i hope.

    There is no need for checksum, keep filenames, make any sort of reverse process to restore files afterwards.
    I am using Win7x64.

  2. #2
    The Founder encode's Avatar
    Join Date
    May 2006
    Location
    Moscow, Russia
    Posts
    4,012
    Thanks
    399
    Thanked 398 Times in 152 Posts
    Why not? However, potentially dangerous idea if something got wrong.

    As a note, you can buy an HDD enclosure with RAID support and connect it via eSATA, USB3(or even 3.1), ... Well upgrade your PC - nowadays even network routers got USB3 support...

  3. #3
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    I intended this to be a test of speed using multiple usb2.0 disks simultanios. Thereof no use for restoring.

  4. #4
    Member
    Join Date
    Jul 2013
    Location
    Germany
    Posts
    24
    Thanks
    4
    Thanked 4 Times in 4 Posts
    Yes, in theory and in practice the file transfer rates of your drives add up.

    64 kB might be too small for best performance. You can benchmark different 2^n-blocksizes.

    Splitting files into blocks can get you into trouble. In the worst case, each HDD contains at least one block of each file. So one crashed HDD can corrupt every file. If you use XOR, two crashed HDDs could corrupt all your files.

    If you use something smarter, like RS, LDPC, TPC, you can lose more HDDs before corrupting your data.

  5. #5
    Member
    Join Date
    Dec 2013
    Location
    Italy
    Posts
    342
    Thanks
    12
    Thanked 34 Times in 28 Posts
    I do not suggest a RAID-0 with 6 disks, neither 2.
    Apart from that, if you want to academic try, use zfs and format with raidz2 (6 disk-raid6-like), or stripe (raid-0).
    With raidz2 you should get about 2x in r and 2x w.
    With 4 striped ~3x r, 2x w

    But I am not sure that a USB controller can write IN PARALLEL via the same bus on different devices.

  6. #6
    Member
    Join Date
    Jun 2013
    Location
    Sweden
    Posts
    150
    Thanks
    9
    Thanked 25 Times in 23 Posts
    I chose 64kb just because I'm still in the 80386-age with com-files and 64kb is the maximum memory. Sure, I managed to make some exe with multiple segments, but then I copied other people's asm files and cleaned out their code and then add my own (create a clean slate), Because I could not then, as now, make my own asm-template to make my own programs. Today i cant even compile an asm-file even if someone else has done it, can not teach an old dog to do new tricks.

    That said, this is just a speed test. I've been through some disk crashes in the past and lost a lot of data. The best way is to have loose files on the disk and then make regular copy to multiple disks. And I use iceecc to get "some" protection (less than 1%). Therefore, A (single USB3.0 1.5 TB) is a copy of D (Windows Software RAID of 3x1TB to boost sequential speed), and then N, O, P, Q, R, S (6x single USB2 1TB .0) is a copy of A.

    And USB 2.0 alone goes up to 40MB / s, therefore, I have placed my external USB2.0 in separate USB connector so that only the two of them share one USB channel.

    Regarding RS, LDPC, TPC and ZFS, it's little outside of what I can manage to understand. I'm probably not ready to start with Solaris or Linux. I'm familiar with windows, so it's likely that is what I'll be using for as long as possible.

    Add: If TrueCrypt could be used to assign multiple container files or disks in own raid.... then nobody could recover anything without all disks.

  7. #7
    Member
    Join Date
    May 2016
    Location
    Waverly, KS, USA, Earth, Sol System, Milky Way
    Posts
    5
    Thanks
    3
    Thanked 2 Times in 2 Posts
    Well, if you're running Windows Vista or newer give StableBit DrivePool a shot. It works on top of NTFS and does read striping.
    NVM this is probably irrelevant for your use case.
    Last edited by OneDeltaTenTango; 23rd May 2016 at 19:25.

  8. Thanks:

    a902cd23 (29th May 2016)

Similar Threads

  1. PerfectCompress, a new file compression software.
    By moisesmcardona in forum Data Compression
    Replies: 148
    Last Post: 21st May 2018, 03:16
  2. Replies: 0
    Last Post: 29th October 2014, 13:44
  3. ARJ Software dead?
    By nikkho in forum Data Compression
    Replies: 11
    Last Post: 19th October 2013, 08:18
  4. Software protection
    By Shelwien in forum Data Compression
    Replies: 11
    Last Post: 10th April 2011, 15:13
  5. RAMDisk software
    By Surfer in forum The Off-Topic Lounge
    Replies: 9
    Last Post: 30th March 2010, 22:44

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •