Re:File Content Histograms
by wrwetzel on Wednesday July 06, 2011 @10:22AM
(#2734008)
I did something similar many years ago for identifying bit order and sense in mu-law encoded audio files as part of a larger audio-file display program. There were four combinations possible (big-endian, little-endian combined with bit-normal, bit-inverted) and no consistent standard was adhered to by the various organizations supplying data.
Although developed specifically for mu-law data I found it quite helpful for identifying other file types as well, just as the author here described.
Re:File Content Histograms
by jannes on Wednesday July 06, 2011 @10:18AM
(#2733998)
All binary file formats should be randomly distributed along the spectrum if their file size is large enough.
Also, I am not sure that this sort of diagram would be suitable to tell certain file types apart, there is probably always a better way to do that.
You should notice that the histogram of a zip file, for example, should level out (like the truecrypt histogram does) if the file were as big as the truecrypt file.
The look of these histograms is largely influenced by the file size. So, I think only the peaks are worth to interpret.
Re:File Content Histograms
by jannes on Thursday July 07, 2011 @02:33AM
(#2737473)
When I wrote that previous comment I assumed that most binary data would probably use an encoding that is longer/shorter than 8 bit (that's what was examined here). But thinking about it, that might not be true. I don't know.
Although developed specifically for mu-law data I found it quite helpful for identifying other file types as well, just as the author here described.
Bill
http://imgur.com/a/vwoIF
Hmm, may not get any work done this afternoon.
Also, I am not sure that this sort of diagram would be suitable to tell certain file types apart, there is probably always a better way to do that.
You should notice that the histogram of a zip file, for example, should level out (like the truecrypt histogram does) if the file were as big as the truecrypt file.
The look of these histograms is largely influenced by the file size. So, I think only the peaks are worth to interpret.
Why would that be the case?
- Executables: some opcodes are more common than other, and usually contain text/messages
- Zip files / compressed images / mp3s: compression might have a distinct "signature" that has peaks, for example block headers
- Database files: not all values are equally common in records (probably peaks around zero, because of fixed-size integers and string padding)
And so on...
http://www.cutawaysecurity.com.nyud.net/blog/file-content-hi...
perhaps relative distributions would be close enough to the "general pattern" for a particular file type to be guessed?