News for hackers. Stuff that matters.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by bbanyc on Friday February 14, 2014 @06:24PM (#7241760)
The punchline to this story is that a few years ago the Times stopped scrambling their .puz files, making all this reverse-engineering work largely irrelevant.

(At the time I was using a modified version of the "xword" program in Debian's repo, which didn't detect whether the file was scrambled. In other words, it treated every letter as wrong because it didn't match the enciphered grid. I ended up hacking in some code to detect these files and disable the check/reveal features when playing them.)

Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by breadbox on Friday February 14, 2014 @07:13PM (#7241984)
Very true! I mentioned that fact the first time I gave this presentation, but it wound up being an anticlimactic ending, so I chose to omit it from the written essay. (And at this point, the focus is more about the process of reverse-engineering anyway.)

EDIT: To be precise, there were still a few other crossword publishers using the scrambling feature. None as important as the New York Times, though, of course.

Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by StavrosK on Saturday February 15, 2014 @12:49PM (#7244526)
Problem solving is its own reward!
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by davepeck on Saturday February 15, 2014 @01:41AM (#7242953)
This is a great and entertaining read about reverse engineering.

It's such a good read that this is almost beside the point... but, as it happens, I worked on and reverse-engineered this same "encryption" scheme (I hesitate to use the word) for an iOS app that never shipped. I just dumped the code (which seems to have been written in late 2008) up on github... it's old, and messy, but hey, maybe it's fun for someone:

https://github.com/davepeck/puzfile

Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by danielpunkass on Friday February 14, 2014 @02:20PM (#7240432)
Very cool rundown of the approach to trying to decode this. FWIW there is also a significant archive of information about the format here, including information about the scrambling: https://code.google.com/p/puz/wiki/FileFormat
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by snori74 on Friday February 14, 2014 @03:22PM (#7240768)
Indeed, staggering how patient and determined some people can be. I love this line when asked by his friend if he would be able to reverse-engineer this scrambling algorithm:

"My response was: maybe. Hard to say, but I'm willing to try. Privately, though, my reaction was THIS IS MY DREAM PROJECT AND THERE IS NO WAY I'M NOT SPENDING ALL AVAILABLE FREE TIME ON THIS."

Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by zem on Friday February 14, 2014 @03:17PM (#7240750)
yeah, i found that extremely helpful when writing the acrosslite module for a crossword format converter. i decided not to bother with scrambled grids for the moment, though, since they are a very acrosslite-specific feature; once the project progresses a bit further i'll go back and add them in.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by TrainedMonkey on Friday February 14, 2014 @03:12PM (#7240718)
Extremely methodical and determined approach. Especially analysis of the errors that partially successful approaches encountered. Well done.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by mistercow on Saturday February 15, 2014 @12:39AM (#7242800)
>In a way this is just a restatement of Occam's Razor, but I like it because it clarifies why Occam's Razor is a good idea. It's not because simpler solutions are actually more likely to be true; they usually aren't. It's because it's almost always easier to improve a simple solution by adding complexity, than it is to improve a complicated solution by digging out a simple solution buried within it.

While the second part of that is an interesting observation, the first part is simply false. It basically comes down to prior probabilities and conjunctions. Every bit of new information implied by a hypothesis is another "and". It is a simple fact that P(X) ≥ P(X and Y), so the more conjunctions your hypothesis implies (the more complex it is) the lower its prior probability.

Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by bluedino on Friday February 14, 2014 @03:43PM (#7240876)
Wouldn't it have been easier to disassemble the program that works with these files, and analyze the code?
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by odin1415 on Friday February 14, 2014 @03:49PM (#7240906)
Maybe, but the author mentioned that he wanted to reverse engineer it as a black box, out of legal concerns. It makes a more interesting challenge this way too.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by voltagex_ on Friday February 14, 2014 @11:33PM (#7242678)
I particularly liked the approach to automation of the application, even through WINE. Setting the time with LD_PRELOAD is a really neat trick.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by bluedino on Friday February 14, 2014 @04:43PM (#7241173)
Ah, I skimmed the article and didn't see that part - makes for a much more interesting writeup anyway.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by mistercow on Friday February 14, 2014 @09:53PM (#7242442)
Man, reading little-endian binary formats makes my head hurt. I get why it's done that way, but what a nightmare for comprehending what you're reading.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by userbinator on Friday February 14, 2014 @10:22PM (#7242513)
After enough time spent reading hexdumps you get used to it, and then all of a sudden big-endian feels really backwards.
Re:Cryptic Crossword: Amateur Crypto and Reverse Engineering by mistercow on Friday February 14, 2014 @11:20PM (#7242651)
Well, assuming you're reading little-endian hexdumps. I cut my teeth on OS X back in the PPC days, so to me it is just the opposite.