News for hackers. Stuff that matters.
Re:TSA redaction fail: hidden text easily readable via copy ... by cduan on Sunday December 06, 2009 @09:09PM (#980547)
It's helpful to understand a little bit of how a PDF file works.

A PDF file stores a sequence of instructions for what do draw on a page. The instructions are commands, such as "write this text here" or "draw a circle there." When you draw black redaction boxes, you are just appending an instruction to the end of the list ("draw a box here"), but most of the time the previous instructions are all still there.

Really, the only way to redact a PDF with certainty, short of manually reading the file itself (yes, PDF files are text-readable; you just have to decompress them) is to rasterize the file to a large image. Otherwise, the instructions for writing the text are all still in the file, and there are any number of ways to extract them (for example, simply by deleting the instruction to draw the redaction box).

Re:TSA redaction fail: hidden text easily readable via copy ... by mschwar99 on Sunday December 06, 2009 @10:37PM (#980652)
A government document can't (shouldn't) be rasterized because it would loose all the tagging information making it non-accessible in a 508 sense.

It is however trivial for even a non-technically inclined person to remove the text content through Acrobat's content editing pane.

Just as you suggest, things like this seem to be an example of someone in a large organization not quite understanding how the "magic" behind the tools they use work and winding up with results they didn't expect.

Re:TSA redaction fail: hidden text easily readable via copy ... by weaksauce on Sunday December 06, 2009 @11:07PM (#980690)
I understand that but why would they not just replace the area with text that says [ redacted ] instead of the black box?

Edit: you can pad the redacted box with spaces to keep the formatting consistent.

Re:TSA redaction fail: hidden text easily readable via copy ... by mschwar99 on Sunday December 06, 2009 @11:15PM (#980697)
From what I have seen in working with the government and the web I think that anigbrowl's comment is spot on. Many people use tools that are designed for the web or the desktop in the same way that they used to use tools designed for print. In this case just slapping a black line over it works in print, and it does not occur to many people that don't fully understand the technology they use that it might not be directly analogous.

A significant portion of my day job involves working with people in government that have a mental model of content distribution that was constructed during the era of print. Its a fun challenge to help people adjust those models to make things more efficient / more informative / less dangerous.

Re:TSA redaction fail: hidden text easily readable via copy ... by derefr on Sunday December 06, 2009 @11:14PM (#980695)
Perhaps they want to keep the formatting, and don't want pages to reflow after they've already been layed out? The Unicode character "█" works well for this purpose.
Re:TSA redaction fail: hidden text easily readable via copy ... by lt on Monday December 07, 2009 @06:01AM (#981028)
I remember a similar story from a few years ago where redacted names where discovered from the length of the redacted characters. Turned out it wasn't too difficult to brute force names and see which ones matched the exact size of the redaction.
Re:TSA redaction fail: hidden text easily readable via copy ... by eru on Monday December 07, 2009 @10:42AM (#981326)
They should have used a fixed width font.
Re:TSA redaction fail: hidden text easily readable via copy ... by brown9-2 on Sunday December 06, 2009 @11:14PM (#980696)
Occum's Razor: because "they" did not understand PDF and thought that they were making the text unreadable simply by drawing boxes over it.
Re:TSA redaction fail: hidden text easily readable via copy ... by oiuytgfhn on Sunday December 06, 2009 @11:45PM (#980727)
Black boxes look much more secret agent?
Re:TSA redaction fail: hidden text easily readable via copy ... by anigbrowl on Sunday December 06, 2009 @09:26PM (#980573)
Mrs Browl works at a e-discovery company where documents are often redacted depending on which bits are admissible in court; I gather they use TIFF files for exactly the reasons you describe.

I suppose the (extremely limited) redaction features in Acrobat stem from the belief that the document is ultimately destined for printing rather than computer reading.

Re:TSA redaction fail: hidden text easily readable via copy ... by metachor on Monday December 07, 2009 @09:29AM (#981209)
I also work at an e-discovery company and can confirm the use of TIFF files as the court-admissible format for producing any sort of document (especially if it has redactions).

Here is an example of the process to apply redactions to a PDF file (or any sort of electronic document). The document is opened in a third-party native viewer with redaction capabilities (and not in the original application like Arcobat or Office). The end-user draws a black box over the appropriate text using the third-party native viewer. This redaction is saved as a separate layer that can be rendered on top of the native document when viewed in the software, as well as printed out. Finally, when documents need to be produced for court, they are printed out to a TIFF-printer driver, effectively converting them to TIFF and "burning in" the redaction layer. The TIFFs are delivered to the courts and opposing council.

Re:TSA redaction fail: hidden text easily readable via copy ... by pasbesoin on Monday December 07, 2009 @01:12AM (#980820)
I have no personal experience and at best perhaps a vague recollection of having read or heard something, but would document images be or in the past have been used also to make discovery as inefficient as permitted for the opposition? Images require a person to analyze, or pre-processing before electronic search and analysis can be performed against them.
Re:TSA redaction fail: hidden text easily readable via copy ... by anigbrowl on Monday December 07, 2009 @01:36AM (#980836)
OCR is very good nowadays, so that's not so big a problem - and for big lawsuits (eg patent fights) there are literally hundreds of lawyers hired to comb through the vast array of documents deciding whether they're responsive or not.

The pre-processing is largely automated but there's certainly a portion (maybe 5%?) of documents that need to be hand-classified in the database before they get to the lawyers. It an interesting field - lots of money to be made, intense competition for it, relatively simple technology requirements but a legal industry which has been resistant to technology for quite a long time. Autonomy seems to be the leading software company in this space.

Re:TSA redaction fail: hidden text easily readable via copy ... by pmorici on Monday December 07, 2009 @05:18AM (#980987)
I can't speak for their other products but in my experience their enterprise search product is a useless piece of crap.
Re:TSA redaction fail: hidden text easily readable via copy ... by tedunangst on Monday December 07, 2009 @02:11AM (#980866)
When you're billing the client $300 an hour to do a job they can't easily take elsewhere, you don't have a lot of incentive to improve efficiency.
Re:TSA redaction fail: hidden text easily readable via copy ... by eli on Monday December 07, 2009 @09:19AM (#981185)
I heard once that NSA guidelines are to print out PDFs, redact portions with a marker and then rescan them into new files.
Re:TSA redaction fail: hidden text easily readable via copy ... by bia on Monday December 07, 2009 @12:38AM (#980785)
If you open the file in Adobe Acrobat Pro, there's a tool called "TouchUp Object" that lets you literally just click and delete the boxes.
Re:TSA redaction fail: hidden text easily readable via copy ... by wgj on Sunday December 06, 2009 @09:04PM (#980536)
Want to know which twelve passports will instantly get you shunted over for secondary screening, simply by showing them to the ID-checking agent?

Cuba, Iran, North Korea, Libya, Syria, Sudan, Afghanistan, Lebanon, Somalia, Iraq, Yemen, or Algeria

Re:TSA redaction fail: hidden text easily readable via copy ... by staunch on Sunday December 06, 2009 @10:19PM (#980631)
That's such an obvious list I'm not sure if it's from the document itself, or if you just wrote down what it would probably be.
Re:TSA redaction fail: hidden text easily readable via copy ... by sage_joch on Sunday December 06, 2009 @10:59PM (#980681)
The absence of Saudi Arabia is pretty interesting.
Re:TSA redaction fail: hidden text easily readable via copy ... by naz on Sunday December 06, 2009 @11:14PM (#980694)
Right, didn't half the 9/11 hijackers have Saudi passports? They wouldn't even get secondary.
Re:TSA redaction fail: hidden text easily readable via copy ... by rfreytag on Monday December 07, 2009 @03:19AM (#980907)
None of the 9/11 hijackers' nationalities are represented (Saudi Arabia, Egypt, UAE), excepting Lebanon.

http://en.wikipedia.org/wiki/Organizers_of_the_September_11,...

Re:TSA redaction fail: hidden text easily readable via copy ... by jcromartie on Monday December 07, 2009 @10:18AM (#981285)
Yes, but the Saudis are our rich friends so we wouldn't want to bother them with anything like pursuing real justice.
Re:TSA redaction fail: hidden text easily readable via copy ... by endtime on Sunday December 06, 2009 @10:39PM (#980654)
I was a little surprised about Cuba. They're not our favorite government, but I don't see them as a security risk.
Re:TSA redaction fail: hidden text easily readable via copy ... by JimmyL on Sunday December 06, 2009 @11:06PM (#980688)
Cuba (along with Syria, Sudan, and Iran) compromise the State Department's list of State Sponsors of Terrorism - which I would assume is the origin of their being on that list.
Re:TSA redaction fail: hidden text easily readable via copy ... by Dilpil on Monday December 07, 2009 @09:46AM (#981238)
The countries on the list are there more for "Screw you" reasons than actual security. That is why Cuba is on the list, and possibly first.
Re:TSA redaction fail: hidden text easily readable via copy ... by oiuytgfhn on Sunday December 06, 2009 @11:30PM (#980710)
Are you serious!!!!

America has been living under the shadow of imminent Cuban aggression for 50years. There have been Cuban attempts to invade America and numerous assassination attempts on the US president by Cuban agents.

And their chemical weapons kill 1000s of Americans after being secretly delivered from Canada.

Re:TSA redaction fail: hidden text easily readable via copy ... by anigbrowl on Monday December 07, 2009 @12:30AM (#980775)
Apparently your sense of humor is too subtle for some other HNers!
Re:TSA redaction fail: hidden text easily readable via copy ... by Beanblabber on Monday December 07, 2009 @02:31AM (#980879)
Or perhaps a bit too blunt.
Re:TSA redaction fail: hidden text easily readable via copy ... by wgj on Sunday December 06, 2009 @10:40PM (#980656)
It's from the doc.
Re:TSA redaction fail: hidden text easily readable via copy ... by RyanMcGreal on Sunday December 06, 2009 @11:06PM (#980686)
Sounds like George Carlin's seven words you can't say on TV.
Re:TSA redaction fail: hidden text easily readable via copy ... by chaosmachine on Sunday December 06, 2009 @08:57PM (#980529)
This isn't the first time this has happened. Several years back, another government agency made the same mistake. On a slow computer, the text would render first, and could be seen for several seconds before the black boxes would appear.

Googling around, it seems to be a fairly common mistake, going back to at least 2000.

http://www.securityfocus.com/news/7272

http://cryptome.org/iran-cia/cia-iran-pdf.htm

http://blogs.zdnet.com/BTL/?p=12907

Re:TSA redaction fail: hidden text easily readable via copy ... by pasbesoin on Sunday December 06, 2009 @10:34PM (#980645)
In the same vein, if not the same format: Distributing a Word document from which you've not removed change tracking (and/or other metadata). (Or whatever Word calls it; it's been a while.) I had to correct my teammates on this one, a while back (and yes, the documents were going to external clients). Nothing novel; the problem's been in the news repeatedly. Nonetheless, people -- even "technical" -- still don't get it right.

(I encouraged them to go further and switch to PDF format for the distribution, but they wouldn't.)

Re:TSA redaction fail: hidden text easily readable via copy ... by zitterbewegung on Sunday December 06, 2009 @09:49PM (#980596)
This should be required reading on redaction in the government. http://www.fas.org/sgp/othergov/dod/nsa-redact.pdf
Re:TSA redaction fail: hidden text easily readable via copy ... by anigbrowl on Sunday December 06, 2009 @10:48PM (#980667)
Nice one. I had to giggle at the ubiquity of the MS Office assistant down in the corner... It looks like you're trying to conceal some information from prying eyes. Would you like help with that?
Re:TSA redaction fail: hidden text easily readable via copy ... by grinich on Monday December 07, 2009 @12:43AM (#980794)
Here's a version with the redactions lifted and the areas highlighted. http://cryptome.org/tsa-screening.zip
Re:TSA redaction fail: hidden text easily readable via copy ... by clistctrl on Monday December 07, 2009 @08:33AM (#981128)
it would be really interesting if the government attempts to go after you for hosting that document.
Re:TSA redaction fail: hidden text easily readable via copy ... by carbon8 on Monday December 07, 2009 @10:03AM (#981259)
That's actually hosted by cryptome: http://en.wikipedia.org/wiki/Cryptome
Re:TSA redaction fail: hidden text easily readable via copy ... by rms on Sunday December 06, 2009 @09:26PM (#980572)
Re:TSA redaction fail: hidden text easily readable via copy ... by sage_joch on Sunday December 06, 2009 @10:43PM (#980659)
It's a lengthy document. Luckily, they've highlighted the areas of interest.
Re:TSA redaction fail: hidden text easily readable via copy ... by pmorici on Monday December 07, 2009 @05:02AM (#980971)
They aren't all that remarkable.
Re:TSA redaction fail: hidden text easily readable via copy ... by keenerd on Monday December 07, 2009 @06:06AM (#981030)
A responsible citizen would not google the following:

"SENSITIVE SECURITY INFORMATION" site:.gov filetype:pdf

Less than 2000 such PDFs. A very determined and very irresponsible citizen could have already mirrored all of them.

Re:TSA redaction fail: hidden text easily readable via copy ... by skorgu on Monday December 07, 2009 @09:33AM (#981215)
Well, not from their home IP they wouldn't anyway.
Re:TSA redaction fail: hidden text easily readable via copy ... by noonespecial on Monday December 07, 2009 @12:11AM (#980750)
I like watching Security Theater as much as the next guy, but this season's plot is so thin, I'm having trouble suspending disbelief.
Re:TSA redaction fail: hidden text easily readable via copy ... by zacku235 on Monday December 07, 2009 @12:37AM (#980783)
I downloaded it earlier but it seems to have been taken down now. Very interesting. I bet that will lose some people their jobs - but I think it definitely highlights an underlying issue within the government bureaucracy that shows why people getting paid 30k with degrees from the University of Phoenix shouldn't be entrusted with national security... I wonder how many security people that passed through...
Re:TSA redaction fail: hidden text easily readable via copy ... by pmorici on Monday December 07, 2009 @05:10AM (#980978)
"bet that will lose some people their jobs"

I doubt it. the more likely outcome is they mandate all their employees attend document redaction training developed and administered at great tax payer cost by a government contracting company. It's also quite likely that they use the incident to justify spending millions to have a government contractor write document redaction software.

Re:TSA redaction fail: hidden text easily readable via copy ... by ryanwaggoner on Monday December 07, 2009 @02:02AM (#980856)
Most interesting thing I learned from the document: members of Congress get special ID cards. It makes sense, I just never thought about it before.
Re:TSA redaction fail: hidden text easily readable via copy ... by sorbus on Sunday December 06, 2009 @11:22PM (#980707)
Using the link in the article, it appears that it's already been taken down (link leads to a 404 page). However, from the comments in the article, it's been mirrored at http://cryptome.org (at the top of the list, in fact, filename is tsa-screening.zip, in case that makes it easier to find it).
Re:TSA redaction fail: hidden text easily readable via copy ... by anigbrowl on Monday December 07, 2009 @12:35AM (#980781)
Looking at the dates on the document, it was last revised in May 2008, so it seems like the horse left the stable some 18 months ago as far as actual bad guys and spies (who presumably seek out this information more actively than your or I) are concerned.
Re:TSA redaction fail: hidden text easily readable via copy ... by julio_the_squid on Sunday December 06, 2009 @10:07PM (#980622)
Pretty ridiculous. Why not just make one version for internal distribution, and a completely separate document for the public?
Re:TSA redaction fail: hidden text easily readable via copy ... by jrockway on Monday December 07, 2009 @04:45AM (#980959)
Because that would cut into your Solitaire time.
Re:TSA redaction fail: hidden text easily readable via copy ... by milkshakes on Sunday December 06, 2009 @09:24PM (#980569)
searchable too. are we feeling safer yet?
Re:TSA redaction fail: hidden text easily readable via copy ... by dhyasama on Monday December 07, 2009 @10:19AM (#981286)
My sister is an attorney and asked me about software that can catch things like this, remove metadata, etc. Does anyone have experience with something along these lines?
Re:TSA redaction fail: hidden text easily readable via copy ... by tlrobinson on Monday December 07, 2009 @02:44AM (#980888)
Schneier is going to have a field day with this one.