File "Type" identification tool

[soor]

Hi
I have developed a tool to identify file types (or rather similar structure), so it may detect
for example ppm files as bmp , map or raw (all images) , or zip and rar, (or compressed exe) as jpeg (all compressed), in order to build optimized models by structure. Because i have rather limited training data, i would like to hear about your results identifying some of your test data sets.

I hope someone finds this usefull ( and i hope it works on your data , my test data is rather "skewed", mostly software projects and resources from video games)

[VoLT]

this method by Luigi Auriemma ...

[m^2]

http://mark0.net/soft-trid-e.html

[VoLT]

m^2 in method by Luigi Auriemma informations was retrieved from TrID, and from http://toorcon.techpathways.com/uploads/headersig.txt

by Luigi Auriemma

informations retrieved from the following resources:
- http://mark0.net/soft-trid-e.html (a very stripped-down version of it)
- http://toorcon.techpathways.com/uploads/headersig.txt

consider that I need only a quick check based on the first bytes because
this function is used only rarely and needs to be fast and simple

[Matt Mahoney]

http://mark0.net/soft-trid-e.html

Sometimes...

C:\res\maxcomp>trid *

TrID/32 - File Identifier v2.10 - (C) 2003-11 By M.Pontello
Definitions found:  4320
Analyzing...

File: a10.jpg
 50.0% (.JPG) JFIF JPEG Bitmap (4003/3)

File: acrord32.exe
 71.8% (.OCX) Windows OCX File (123521/4/18)

File: english.dic
       Unknown!

File: FlashMX.pdf
100.0% (.PDF) Adobe Portable Document Format (5000/1)

File: fp.log
       Unknown!

File: mso97.dll
 46.6% (.AX) DirectShow filter (201555/2/20)

File: ohs.doc
 36.2% (.FLO) iGrafx FlowCharter document (33500/1/5)

File: rafale.bmp
100.0% (.BMP) Windows Bitmap (2000/1)

File: vcfiu.hlp
       Unknown!

File: world95.txt
       Unknown!

C:\res\maxcomp>cd ..\calgary

C:\res\calgary>trid *

TrID/32 - File Identifier v2.10 - (C) 2003-11 By M.Pontello
Definitions found:  4320
Analyzing...

File: BIB
       Unknown!

File: BOOK1
100.0% (.AIML) Artificial Intelligence Markup Language (14500/1/4)

File: BOOK2
       Unknown!

File: GEO
       Unknown!

File: NEWS
100.0% (.PL) Perl script (4000/1/1)

File: OBJ1
 44.3% (.CEL) Lumena CEL bitmap (63/63)

File: OBJ2
100.0% (.MPG/MPEG) MPEG Video (3000/1)

File: PAPER1
       Unknown!

File: PAPER2
       Unknown!

File: PIC
 23.0% (.WK*) Lotus 123 Worksheet (generic) (2005/4)

File: PROGC
       Unknown!

File: PROGL
100.0% (.DII) Summation Document Image Information Load File (4000/1/1)

File: PROGP
100.0% (.DPR) Delphi project source (16000/1/3)

File: TRANS
       Unknown!

C:\res\calgary>cd ..

C:\res>trid enwik8

TrID/32 - File Identifier v2.10 - (C) 2003-11 By M.Pontello
Definitions found:  4320
Analyzing...

Collecting data from file: enwik8

Warning: file seems to be plain text/ASCII
         TrID is best suited to analyze binary files!

 39.0% (.XML/ATOM) Atom web feed (35500/1/16)
 21.4% (.KML) Google Earth placemark (19500/1/6)
 20.3% (.SVG) Scalable Vector Graphics (18500/1/6)
 15.9% (.AIML) Artificial Intelligence Markup Language (14500/1/4)
  3.2% (.HTML) HyperText Markup Language (3000/1/1)

C:\res>trid enwik9

TrID/32 - File Identifier v2.10 - (C) 2003-11 By M.Pontello
Definitions found:  4320
Analyzing...

Collecting data from file: enwik9

Warning: file seems to be plain text/ASCII
         TrID is best suited to analyze binary files!

 49.6% (.XML/ATOM) Atom web feed (35500/1/16)
 25.8% (.SVG) Scalable Vector Graphics (18500/1/6)
 20.2% (.AIML) Artificial Intelligence Markup Language (14500/1/4)
  4.1% (.HTML) HyperText Markup Language (3000/1/1)

C:\res>