News for hackers. Stuff that matters.
Re:File format wiki by scrollaway on Wednesday December 09, 2015 @08:33AM (#10703445)
I worked in video games reverse engineering for several years, so I'll take the occasion to quote a recent tweet from a friend/colleague of mine:

> At HandmadeCon @mike_acton brought up the game data format problem. I've reverse engineered hundreds of proprietary formats from games, and he's right that all I would need to make things simple is a basic description of the file data. Just struct declarations are enough. No encryption has stopped me, I always find the data I need, and your data all has the same general architecture, so there's no reason a game developer should be afraid of the public knowing how their game's data is laid out. This zeitgeist of closely guarding that information is doing you more harm than good — moddability is a key feature of many successful games, and all modders want is docs.

--

There's a lot of other industries that could stand to learn from this. Reinventing the wheel sucks. Doing your own, proprietary thing in your corner, taking on extra maintenance work, extra work to modify existing tools to make them compatible with your own image/archive/whatever format... it all sucks and nobody wins.

Re:File format wiki by TheCapn on Wednesday December 09, 2015 @09:47AM (#10703877)
Any good sources you'd use to get off the ground running with file format reverse engineering?

I have a couple small projects on the backburner at work that require me to reverse engineer file formats. I've been able to decode a few basic details out of the files by looking for patterns in a hex editor but the finer details are still escaping me.

Re:File format wiki by scrollaway on Wednesday December 09, 2015 @10:52AM (#10704312)
I'm self-taught so I won't be much help there. The usual path starts with learning assembly. If you have skills in reverse-engineering the actual executables, reversing the file formats is as simple as tracing back what happens when the files are read.

It gets tougher when there are anti-debugging protections in place but that generally happens only in the video games industry.

If the format is simple enough though, you will be fine with a good hex editor and some pattern recognition.

Re:File format wiki by a_t48 on Wednesday December 09, 2015 @01:42PM (#10705769)
Along with learning assembly, find an active project looking for help. I cut my teeth on deconstructing\modding Oni (see http://wiki.oni2.net/ONCC for example) and it was great to have other people to bounce ideas off of.
Re:File format wiki by voltagex_ on Wednesday December 09, 2015 @10:23AM (#10704153)
The only way I've been able to do it is start writing a program that reads the header and prints details.

Your other option is something like QuickBMS - on mobile, can't link.

Re:File format wiki by voltagex_ on Wednesday December 09, 2015 @08:48AM (#10703520)
Correct me if I'm wrong, but haven't the various game modding and emulation scenes been notoriously anti-open-source? Even the mighty Xentax had political issues which resulted in the creation of ZenHax
Re:File format wiki by scrollaway on Wednesday December 09, 2015 @09:02AM (#10703594)
That wildly depends on the community. Often enough, they are mostly composed of teenagers who got into scripting their favourite game. Which is awesome but unfortunately it means that they lack a lot of experience, end up in bad terms with the developers etc.

As a counterpoint, the community I founded around Hearthstone (http://hearthsim.info/) is extremely open source friendly and on excellent terms with the developers. It's really about how you decide to manage it.

Re:File format wiki by Shish2k on Thursday December 10, 2015 @01:35PM (#10712389)
> Just struct declarations are enough

To echo this with code, I've open sourced my collection of game archive unpackers. Given how similar they all are, this is a typical example of how to add a new decoder:

https://github.com/shish/pyge/blob/master/archive/PackDat3.p...

(Note that two of those 6 short strings are user-friendliness metadata and can be omitted, and one of the others is redundant -- "Magic Bytes", "Header Format" and "Directory Entry Format" are all you really need in most cases)

Re:File format wiki by voltagex_ on Wednesday December 09, 2015 @08:49AM (#10703527)
When you say you worked in video game reverse engineering, can you elaborate at all? That sounds incredibly interesting.
Re:File format wiki by scrollaway on Wednesday December 09, 2015 @09:12AM (#10703646)
Most of my work was on World of Warcraft. I wrote an early Django version of what is now WoWDB (http://www.wowdb.com/).

The work consisted mainly in reverse engineering WoW's file formats, protocols etc in order to understand, index and organize data, and to be able to produce "datamined" patch notes as soon as new game files were available.

I got to work with and learn from people a thousand times smarter than me, it was by far my favourite job and it's why I still do various Blizzard-related work today.

It also creates an interesting relationship with the game's developers. I think on that front, many studios could learn a lot about working with the people interested in the technical aspects of their game, rather than working against them. Most get it horribly wrong, few of them get it right at all - Riot is one of the few that do, I wish I enjoyed LoL. :)

Re:File format wiki by voltagex_ on Wednesday December 09, 2015 @09:20AM (#10703697)
For an interesting yet sad comparison, see Rockstar vs Avalanche - GTAx vs Just Cause 2 and 3. While the modding future of JC3 is precarious due to Denuvo DRM (blame Square Enix), JC2 is pretty much the gold standard.

http://www.eurogamer.net/articles/2013-12-06-avalanche-gives...

vs

http://www.eurogamer.net/articles/2015-08-10-rockstar-bans-g...

Re:File format wiki by scrollaway on Wednesday December 09, 2015 @09:26AM (#10703742)
Yeah. This happens a lot. It's infuriating.

Studios that get too large for their own sake and get completely disconnected from their userbase. They no longer see the players, the fun or even the game - they just see the numbers. Business guys thinking a video game company can be run like a bank.

The devs get blamed for it, too; as if they had anything to do with it. These decisions are made high above them.

Re:File format wiki by chei0aiV on Wednesday December 09, 2015 @08:45AM (#10703498)
You might want to talk him into putting that info on the fileformats wiki.
Re:File format wiki by scrollaway on Wednesday December 09, 2015 @08:55AM (#10703548)
I already contribute to these types of project whenever I can, and I keep a close eye on the archive team.
Re:File format wiki by johnlbevan2 on Wednesday December 09, 2015 @08:22AM (#10703402)
FileFormat.info probably deserves a mention, given that has pretty much the same objective: http://www.fileformat.info/format/a.htm
Re:File format wiki by callesgg on Wednesday December 09, 2015 @08:59AM (#10703576)
Wow that site actually had a ton of stuff in a good structured way.

just go to: http://www.fileformat.info/format/${file_extention}/internal...

Re:File format wiki by hackuser on Wednesday December 09, 2015 @12:59PM (#10705389)
Doesn't FileFormat.info already 'solve the problem'? What does the new wiki offer that FileFormat.info lacks, and which couldn't be added to FileFormat.info?
Re:File format wiki by mynewtb on Wednesday December 09, 2015 @02:47PM (#10706205)
You may not publish, copy, display, distribute, transmit, perform, modify, create derivative works from, or sell any Materials, information, products, or services obtained from this Site, except as otherwise expressly permitted under applicable law and as described in these Terms of Use. FileFormat.Info retains all right, title, and interest to the Materials.
Re:File format wiki by ddorian43 on Wednesday December 09, 2015 @08:10AM (#10703355)
You can also help the Archive Team by running the warrior which archives websites that are closing: http://www.archiveteam.org/index.php?title=ArchiveTeam_Warri...
Re:File format wiki by xomateix on Wednesday December 09, 2015 @09:21AM (#10703702)
A bit off topic, but I would like to know why did they choose mediawiki as a platform?

Under my limited experience working with wikipedia and wikidata I've seen it's not the best option to a) store structured data and b) edit the pages (markdown is, imho, much better for that).

Re:File format wiki by scrollaway on Wednesday December 09, 2015 @09:38AM (#10703824)
Mediawiki is honestly the only option when it comes to user-friendly wiki platforms.

[Last time I said that, someone recommended Moin. I hope this won't happen this time - Moin is a UX joke]

Re:File format wiki by hackuser on Wednesday December 09, 2015 @01:02PM (#10705424)
Dokuwiki's UI is fine. Also, what about TWiki/FOSSwiki?
Re:File format wiki by db48x on Thursday December 10, 2015 @06:20AM (#10710070)
I think it's hosted somewhere that has automated tools for installing common software, and Mediawiki is what you get.
Re:File format wiki by gobusto on Wednesday December 09, 2015 @08:14AM (#10703367)
This reminds me of the Xentax wiki: http://wiki.xentax.com/
Re:File format wiki by ChrisArchitect on Wednesday December 09, 2015 @10:04AM (#10703995)
Good share, but why is this Show HN? Did you create it? It's not new...... etc
Re:File format wiki by yoodenvranx on Wednesday December 09, 2015 @08:50AM (#10703530)
Has anyone an idea what happened to wotsit.org?
Re:File format wiki by numberwhun on Wednesday December 09, 2015 @11:36AM (#10704698)
To be honest, I expected a bit more from the site, considering its name. It is good, don't get me wrong, covering a lot of file formats, but you have limited it to things on your computer. For me, if it were an all inclusive wiki, you should expand it to include things like financial file formats. Banks tend to use formats like ach and edi (for example). Those formats are extensive and can get quite complicated, but their file specs are published and available. The site is a great start, but I would love to see it expanded to include anything file format related. Even having the most obscure formats will make the site more appealing to people as a reference. Sort of 'the place to go' for file formatting. Just my .02.
Re:File format wiki by textfiles on Wednesday December 09, 2015 @02:03PM (#10705921)
Then get editing.
Re:File format wiki by g1n016399 on Wednesday December 09, 2015 @08:59AM (#10703569)
Like many ambitious wiki-based projects, this site seems to suffer from lack of focus. Look at these pages:

http://fileformats.archiveteam.org/wiki/Dendrochronology

http://fileformats.archiveteam.org/wiki/Quantum_computer

http://fileformats.archiveteam.org/wiki/TLD_.mobi

http://fileformats.archiveteam.org/wiki/Endianness

How is any of these things a file format? Because the definition of a file format in the FAQ (<http://fileformats.archiveteam.org/wiki/FAQ:File_Format>) is so broad that it can be made to encompass basically everything. The manifesto that started this site is a rambling sermon that doesn't clarify anything in this respect: it just repeats "let's solve the problem" without even properly defining what the problem is.

There are also other issues. The classification scheme is often Procrustean and confused. Error messages were until recently mixed up with error detection codes, which conflates two different meanings of "error". Similarly FUSE shares a category with HFS+, even though the former is an API, and the latter a disk format; distinct things which just happen to share the name of "file system". The pages are rather short and consist mostly of lists of links. Given the above-mentioned lack of clearly defined scope, I suspect many pages seem to be created about topics just because they're in the news and/or just to have a place to put a link to a "neat" blog post: see for example <http://fileformats.archiveteam.org/wiki/Facebook#Links>.

Last but not least, the whole site is rather ugly, and the logo is awfully non-descriptive of what it's supposed to contain; what am I supposed to do with this thumb, stick it up my arse?

It's a shame, really, because documenting file formats is a hard and valuable endeavour. But I don't think these people are going to do a good job of it.

Re:File format wiki by TillE on Wednesday December 09, 2015 @09:21AM (#10703703)
Endianness is extremely relevant to file formats. The others certainly seem like irrelevant nonsense; "organic file formats"?
Re:File format wiki by db48x on Thursday December 10, 2015 @06:21AM (#10710074)
Archive Team is backing up DNA, just in case.
Re:File format wiki by TuringTest on Wednesday December 09, 2015 @11:14AM (#10704503)
Looking at the "what links here" page, the "quantum computer" is probably there as a stop-gap definition because the term is used at "Quantum compressed archive".[1]

Any successful wiki will have a problem of demarcation; the initial signs being that people add whatever they want, including ramblings like Two cows. [2]

If the project goes along, taking Wikipedia as precedent there will develop two "inclusionist" and "deletionist" cultures that will fight over the definition of what the project should cover, but which would get rid of these extreme cases.

[1] http://fileformats.archiveteam.org/wiki/Quantum_compressed_a...

[2] http://fileformats.archiveteam.org/wiki/Two_cows

Re:File format wiki by textfiles on Wednesday December 09, 2015 @01:59PM (#10705891)
Except from the very first day, the declared final authority falls at my desk, and I am a hardcore inclusionist. So no, it won't be making that mistake.
Re:File format wiki by jrgoj on Wednesday December 09, 2015 @09:13AM (#10703653)
Thanks for this. I write file parsers for language translation at my day job, so this could be a nice resource to have on hand.
Re:File format wiki by scrollaway on Wednesday December 09, 2015 @09:16AM (#10703668)
Out of curiosity, are you familiar with Translate Toolkit? https://github.com/translate/

If you're not and are interested by it or in contributing to it, you should email me - I'll put you in touch.

Re:File format wiki by jrgoj on Wednesday December 09, 2015 @11:23AM (#10704579)
I have not worked with the Translate Toolkit yet, but I'm a little familiar with a few of the tools. I'm always interested in learning or contributing where I can.

I currently work on a proprietary l10n/i18n system, so a lot of what I do is under an NDA. However, I'm slowly gaining traction in moving us, at least in part, to a more open model.

If you let me know your email, I will reach out.

Re:File format wiki by scrollaway on Wednesday December 09, 2015 @11:50AM (#10704828)
My email is on https://leclan.ch - Translate was my previous job.
Re:File format wiki by trampi on Wednesday December 09, 2015 @09:19AM (#10703695)
I know the probability tends to be zero, but is it Telelingua?
Re:File format wiki by jrgoj on Wednesday December 09, 2015 @11:25AM (#10704591)
Not Telelingua here. I currently work on a proprietary system. Why do you ask?
Re:File format wiki by trampi on Wednesday December 09, 2015 @11:57AM (#10704887)
Oh, I just had the joy to connect an application of the company I am working at with Telelingua. Everything went really well.
Re:File format wiki by hackuser on Wednesday December 09, 2015 @12:59PM (#10705399)
What is the name of your wiki?