News for hackers. Stuff that matters.
Re:File Format Posters by chillingeffect on Friday April 07, 2017 @04:35PM (#14063295)
back before 2000, it really was important to know file formats. we didn't use libraries. we looked up the formats in books and implemented fresh code every time. I prided myself on having memorized most of the .wav header, enough that i didn't need a reference. Then, I learned .fig. Then, I worked on understanding .jpg.

Nowadays, with widespread APIs, the file formats' significance is almost irrelevant! In theory, only a single person in the world needs to know any file format. Everyone else can use a library they've written.

my how the world changes :)

Re:File Format Posters by alexott on Friday April 07, 2017 @04:47PM (#14063375)
There are different tasks, not everywhere the libraries could be applicable - because of different constraints, like, memory consumption, performance, etc.

For example, For many years I developed and maintained several libraries for high performance media type detection - libraries aren't applicable there. Similarly is for Dara extraction - sometimes you need to have tradeoffs between performance and other parameters, while libraries are optimized for average case.

Re:File Format Posters by questerzen on Friday April 07, 2017 @09:11PM (#14064671)
I'm constantly surprised by how often I still need to look up file formats, and having a resource like this is a great starting point. Some examples from my recent experience for which using a library would be overkill / more effort: examining a .bmp file to figure out why the alpha channel got dropped during resizing; extracting meta-data (top-left pixel alpha value) from a .png file; fixing a failed library-call attempt to change the aspect ratio of an image; working out the version of an ancient document file so I could find an application to read it. We shouldn't become so dependent on libraries that this kind of simple manipulation forces us to install huge libraries and learn complex APIs for simple tasks, or else we collapse hopelessly into despair. Especially when a simple Python script and a one-page file description can allow us to do the job in a few seconds.
Re:File Format Posters by sdenton4 on Friday April 07, 2017 @10:15PM (#14064901)
And that is why you, sir, are the guy who knows the file formats. I look forward to repurposing your scripts for manipulating .pwn files.
Re:File Format Posters by nradov on Friday April 07, 2017 @06:25PM (#14063958)
If only that were true. Yet as far as I can determine there is no available Java library which allows for full reading and writing of the JPEG/EXIF file format. There are a number of Java libraries which allow for reading and encoding JPEG image data but none of them offer complete support for the metadata.

The JPEG/EXIF format itself is a mess, just a completely horrible design. It's one of those old-school file formats like MS Word .doc which isn't so much of a "format" as modern programmers commonly understand the term but more like a straight dump of memory chunks into a file. That's not so bad if you're programming in C but surprisingly difficult in higher-level safe languages like Java.

Re:File Format Posters by cvwright on Friday April 07, 2017 @09:59PM (#14064836)
It was surprising to me how few unique implementations there are of JPEG at all. Dig deep enough into most open source image libraries for almost any language, and eventually you'll find IJG libjpeg or one of its descendants.
Re:File Format Posters by alexott on Friday April 07, 2017 @06:30PM (#14063985)
For metadata, did you check Tika?
Re:File Format Posters by nradov on Friday April 07, 2017 @11:08PM (#14065074)
I tried that but it's pretty buggy. Writing some metadata just fails silently with no indication of what went wrong.
Re:File Format Posters by CyberDildonics on Friday April 07, 2017 @07:53PM (#14064369)
It isn't a format's fault if a language can't manipulate memory properly.
Re:File Format Posters by nradov on Friday April 07, 2017 @11:02PM (#14065057)
I think you missed the point. First, we should have learned by now that using an unsafe language like C to read data from an untrusted source directly into memory is really stupid. Just look at the security vulnerabilities that have been found in applications like ImageMagick.

Second, the JPEG/EXIF format is just needlessly complex regardless of language. Instead of marking metadata locations using something sensible like delimiters it's essentially a whole file system within the file with directories and directory entries. The values of some directory entries aren't storied there directly but instead pointed to by a byte offset value. So if you want to insert another entry you have to recalculate all the offsets! Even using C it's easy to screw that up and corrupt the whole file.

Re:File Format Posters by tdeck on Friday April 07, 2017 @09:54PM (#14064822)
The only image format I can read is XPM :/

https://en.wikipedia.org/wiki/X_PixMap