Page 1 of 3 123 LastLast
Results 1 to 30 of 72

Thread: mp3dump

  1. #1
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts

    mp3dump

    In a discussion yesterday I said that it would be really easy
    to write a dumper similar to jpg2dct for mp3.
    ...And appears that I was right ;)
    http://shelwien.googlepages.com/mp3dump_v0.rar
    Its far from perfect (has problems with some tags and buggy mp3s)
    but still should be usable.

    But then, another unexpected problem appeared.
    I don't think that there's any significant amount of
    redundant information in the dump file, but still it
    has a fixed record size of 5610 which is kinda too
    much for universal compressors.

    And I'm too lazy to try and compress a 45M file with paq8,
    so here's a question: can anybody compress that dump better
    than original mp3, and how much?

  2. #2
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    Why tags and buggy mp3s are problem? It's really easy to parse both ID3v1 and ID3v2 (second one is a bit complex than the first one). For finding valid frames, firstly I search sync word and then check validation for 2 or 3 consecutive valid frames (frame size is calculated from sync word). If you are interested in, I can send a delphi code about it which i wrote long time ago. IIRC, I have some small problems with VBR streams though.
    BIT Archiver homepage: www.osmanturan.com

  3. #3
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I tried LPAQ8, LPAQ9k, LPAQ9l, PAQ9a, DULRILCA.
    The last one won with 169078 B. I guess that the data needs some preprocessing.

  4. #4
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    I just tried a dozen of own mp3s and not even 1 was reconstructed correctly.

  5. #5
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    I just skipped most of "garbage processing" there - only mp3 frames were supposed to be encoded properly.

    Anyway, what about this:
    http://shelwien.googlepages.com/mp3dump_v1.rar
    Last edited by Shelwien; 3rd February 2009 at 20:29.

  6. #6
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quick test: OK.
    Will look deeper later.

  7. #7
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    Blizzard 0.24b only with .dmp - 165460 (I hope m^2's tests were processing cat.mp3 too)
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  8. #8
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Surely the same, but for programs that couldn't handle multiple files, I tared both output files.
    So Blizzard gets 166828. Still no. 1, still worse than original.

  9. #9
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    Tried specifying the record size for the programs which allow that
    like rar,slim and rkc
    but it didn't help.
    Btw .dmp record size for v1 is 5616.
    And cat.mp3 is compressed to 139736 by mp3zip/soundslimmer...

  10. #10
    Programmer schnaader's Avatar
    Join Date
    May 2008
    Location
    Hessen, Germany
    Posts
    566
    Thanks
    216
    Thanked 200 Times in 93 Posts
    Result for paq8p -4 is 164379 (.dmp) + 868 (.pad) = 165247 bytes, so still not better than the original.

    Some questions:
    • If record size is 5616, why is the size of the .dmp file not divisible by 5616? Is there some other data at the beginning or the end of the file or is it just the last record not being complete?
    • Why does the .dmp file look so interleaved (especially the text in the first record looks like Unicode there) - are these stereo left/right channels? The odd bytes are 0 more often than the even. Anyway, when trying to split the "channels", results get worse.
    http://schnaader.info
    Damn kids. They're all alike.

  11. #11
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    The ReadFrame function returns either mp3 frame (0), or a single byte
    of "garbage" (c-256), or EOF (1). And this status value is written into
    the output file as int16. So to be precise, the record size is either
    2 bytes or 5616. I don't think that would affect the compression,
    but I can move these status words into the .pad file, to make the filesize
    divisible by record size .

    Also, these records are unpacked mp3 frame structures, there're many
    short fields stored as ints, and scalefactors and DCT coefficents are
    stored as arrays of 16-bit "words".

  12. #12
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    complete tar in PAQ8o10 - 165052...
    Could adding BMP header help with some specific algorithms? Also in each 5616 bytes long record there are usually over 30 values out of 256 used and (as far as I have checked) always less than 64 values. Could that be in some way used for preprocessing?

    EDIT: where could I find mp3zip?
    EDIT2: cat.mpz is tiny bit smaller without mp3zip - 139641
    Last edited by Black_Fox; 4th February 2009 at 17:37.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  13. #13
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    BBB 161221
    I put it in my testing script and threw everything that I have on it.
    I'll post the logs when it's finished.

  14. #14
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    And the winner is.....

    BBB.

    YBS and BCM beat PAQ too.
    Attached Files Attached Files

  15. #15
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    BWTS on the file and the script again:
    Nanozip cc 160547. 91 bytes more than mp3.
    Attached Files Attached Files

  16. #16
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    159303 with BBB and data preprocessed with own dumpsplitter = 1153 bytes less than original. Will try to improve a bit and provide some more generic dumpsplitter, this one works only on cat.dmp.

    EDIT: forgot to add up size of compressed .pad file.
    Last edited by Black_Fox; 5th February 2009 at 00:37.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  17. #17
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    159161 (which is 1295 B smaller than original) in 3 .bbb files, they can be zipped or paq8o10-ed together to shave a couple more hundred bytes off, but note that this doesn't apply for every mp3.
    I also tried "Entering The Cage", which can be downloaded from this site, and "Life By Proxy" from Jamendo's Dreamer's Paradox. Both work, so it seems the splitter should work on any file dumpable by mp3_rd I also tried BCM, which is faster, but weaker.

    cat.mp3
    original size: 160,456
    BBB size: 159,161 (99.2%)
    BBB time: ~57s
    BCM size: 162,602 (101.3%)
    BCM time: ~21s

    Entering The Cage
    original size: 1,542,144
    BBB size: 1,494,362 (96.9%)
    BBB time: ~77s
    BCM size: 1,529,931 (99.2%)
    BCM time: ~43s

    Life By Proxy
    original size: 10,341,818
    BBB size: 10,013,227 (96.8%)
    BBB time: ~~400s
    BCM size: 10,296,391 (99.5%)
    BCM time: ~244s

    Uploaded whole tryout pack:
    Attached Files Attached Files
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

  18. #18
    Programmer osmanturan's Avatar
    Join Date
    May 2008
    Location
    Mersin, Turkiye
    Posts
    651
    Thanks
    0
    Thanked 0 Times in 0 Posts
    BIT 0.8's new experimental single model (without dumber):
    cat.mp3 -> 157,705 bytes (<1 seconds)
    BIT Archiver homepage: www.osmanturan.com

  19. #19
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    http://shelwien.googlepages.com/mp3dump_v2.rar

    Wonder if this helps...
    Code:
    .hdr: ???               // record size = 378
    .scf: short[2][2][78]   // record size = 624
    .dct: short[2][2][576]  // record size = 4608
    .pad: padding + garbage // n/a

  20. #20
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Quote Originally Posted by Shelwien View Post
    http://shelwien.googlepages.com/mp3dump_v2.rar

    Wonder if this helps...
    Code:
    .hdr: ???               // record size = 378
    .scf: short[2][2][78]   // record size = 624
    .dct: short[2][2][576]  // record size = 4608
    .pad: padding + garbage // n/a
    Why is the output twice as big?

  21. #21
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    sorry, fixed, v2 reuploaded.

  22. #22
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    v2+tar+bwts+nanozip: 160372.
    I'll try bruteforce with Black_Fox's splitter now.

  23. #23
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Taring hurts a lot.
    Tried nz and bwts+nz on individual files:
    d = .dmp
    dn = .dmp.b (wts)
    etc.
    Code:
    d.nz	144435 
    db.nz	140597
    h.nz	10566
    hb.nz	8005
    p.nz	1263
    pb.nz	1337
    s.nz	8570
    sb.nz	7637
    All but .pad benefit from bwt.
    So I tried all to compress when all are treated with bwts and all but .pad.
    All:
    157101
    All but .pad:
    157113

    I'll check how it is with another file.

  24. #24
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    What about image headers + paq8?

  25. #25
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Entering The Cage:

    Code:
    d.nz	1469091
    db.nz	1435412
    h.nz	39241
    hb.nz	38525
    p.nz	2941
    pb.nz	2994
    s.nz	117
    sb.nz	125
    All treated with bwts and packed together :1476824
    BWTS only where it helps: 1476823
    Packed individually: 1476995

  26. #26
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    http://shelwien.googlepages.com/bin2tga_v0.rar

    Code:
    bin2tga 378 cat.hdr cat_h.tga
    bin2tga 624 cat.scf cat_s.tga
    bin2tga 4608 cat.dct cat_d.tga
    convert table to images with this, then to bmp
    with some converter (.tga is much easier to create).
    Then compress with paq8p1.

    Also, rotating and flipping the images would be interesting.

  27. #27
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    You'll be more successful by doing it yourself than trying to convince me to do so.

    I banned paq8, deleted all copies from my HDD and added it to my antivirus blacklist, marked as a pure waste of time.

  28. #28
    Administrator Shelwien's Avatar
    Join Date
    May 2008
    Location
    Kharkov, Ukraine
    Posts
    3,366
    Thanks
    212
    Thanked 1,018 Times in 540 Posts
    I don't insist on using paq8 here, you can try anything else which has 2D models.
    Also, I already made a full mp3 recompressor once.
    Now the question is, whether its possible to do something another way,
    to make it open-source or something.

  29. #29
    Member m^2's Avatar
    Join Date
    Sep 2008
    Location
    Ślůnsk, PL
    Posts
    1,611
    Thanks
    30
    Thanked 65 Times in 47 Posts
    Rings: it hurts. Arc, CCM - no difference.

    We know there's a way. It's only a matter of finding it.

  30. #30
    Tester
    Black_Fox's Avatar
    Join Date
    May 2008
    Location
    [CZE] Czechia
    Posts
    471
    Thanks
    26
    Thanked 9 Times in 8 Posts
    My splitter shouldn't work on v2 output... will look at it tonight.
    I am... Black_Fox... my discontinued benchmark
    "No one involved in computers would ever say that a certain amount of memory is enough for all time? I keep bumping into that silly quotation attributed to me that says 640K of memory is enough. There's never a citation; the quotation just floats like a rumor, repeated again and again." -- Bill Gates

Page 1 of 3 123 LastLast

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •