The DoJ filed a motion on Wednesday to delay the release of the Clinton Foundation emails for 27 months. The reason cited was that there were errors in how the search was conducted.

Given that, it sounds like there is also a fundamental problem in how information is stored and classified. The FOIA requests were for emails, so I would think that the retrieval would be simple. They must be vetting each email for sensitive (secret) information, otherwise I can't imagine how it would take 2.25 years to fulfill that request.

I can't tell if this is a stonewalling effort or a technology shortfall in the FOIA process.

I'm vetting FIOA-like request emails right now and yes each email is vetted for sensitive information. And not just secret info, there are a bunch of exemptions you have to watch out for. If anything the list of exemptions is too short. A lot of stuff that seems like it's private isn't exempted.

It's incredibly time consuming and very very expensive. I'm working one person's emails from a 4 month span and we have ~dozen people working on it for a week.

The more specific request the easier it is. The ones asking for "everything" are just hard as hell.

Cry me a river. It's hard only because there is too much utterly worthless, indeed counterproductive, secrecy. A more open government could not do some things. Good. It shouldn't.

The constitution of the United States + the Bill of Rights were all designed and argued about in secrecy. This was so the politicians in the room could actually be honest and negotiate rather than pander to their states short sighted interests.

No, the framer published articles and books putting forth there arguments, then briefly privately discussed a proposal to vote on.

You may recall something called the Three-Fifths compromise, which pandered to the states' short-sighted interests.

"Briefly" is an interesting way to describe four months.

How much that's exempt from FOIA is so high minded? It is 99.999% ass-coverage through secrecy and you know it.

Surprisingly, most people in government, as elsewhere, spend a majority of their time simply trying to get their work done.

Is there malfeasance and corruption? Certainly. Is the solution to go to 100% open records in a hyper-politicized environment? I'm less sure.

I know I wouldn't want Fox News / MSNBC combing over every email I've ever sent in the course of doing my job.

Working for the Government isn't "just a job" though. The Government holds a huge amount of power over people - and not just the people who pass the laws, but every department that has to come up with its own policies and processes.

Everything the Government does affects somebody in a way that few businesses can. A bad process can literally result in death in some cases (although the third sector attempts to help when Government fails).

People already get into Kafkaesque situations on a regular basis due to either the department's employees not understanding its own policies, or not knowing the magic words to use. The only way to protect against that is to allow people to force the Government to give them information. Personally, I've used FOIA responses in the UK to great effect in forcing the NHS to follow their own procedures (not medical ones, admin ones).

Few business, but gradually more and more. The decisions that Google, Monsanto, MSNBC, Fox News, Dow, or even Walmart make are starting to have more impact than US government decisions.

And I absolutely agree with your comment, but per my comment below, I think 100% transparency carries its own risks in making government less effective.

If the UN and large bureaucratic organizations have taught us anything it's that position jockeying gets ruthless when promotions are a zero sum game because of no organizational growth.

I'm not talking about malfeasance and corruption. I'm talking about embarrassment and enablement of dubious practices. FOIA should rightly make that painful.

On one hand you have dubious practices. On the other hand you have political football / back-stabbing.

My point is that I believe the public interest is best served by a balance between "the public knows how this business was conducted" vs "any rival who wants my job can dig through everything I've done to find some dirt of me."

Politicians, functionaries, and executive staff are people too. And I think HN can be a bit schizophrenic decrying "people in government are so ineffectual and unwilling to take risks" while simultaneously designing a purity litmus test where only the most boring, unadventurous apparatchiks are able to keep their jobs.

> My point is that I believe the public interest is best served by a balance between "the public knows how this business was conducted" vs "any rival who wants my job can dig through everything I've done to find some dirt of me."

Was that ever a standard for exemption from FOIA? Much less the intent of anyone writing any part of FOIA? Stupidity should hurt.

Your original point was that "99.999% [of FOIA exemptions are] ass-coverage through secrecy" and that's a bad thing. I disagree for the reasons I specified.

If stupidity were cause for dismissal, we'd have a hard time staffing most of the jobs in the world.

Why not?

So if someone asks for emails, no one should care if some of them are employees asking for medical leave? HIPAA just shouldn't apply to government employees? Emails containing complaints about harassment or assaults, the victims who complained should be just named to anyone who asks the government? Employees have to provide SSNs and banking info to HR, should that be publicized for scammers?

"The Government" is also a workplace, and deals with all sorts of personal information on a daily basis. Their privacy is hardly "worthless," their emails deserve at least a quick sanity check before being released potentially to the entire world.

> They must be vetting each email for sensitive (secret) information, otherwise I can't imagine how it would take 2.25 years to fulfill that request.

Yes, that's the gist of it. From the filing in question:

"Furthermore, in this case in particular, the substantive review process is challenging given the wide-ranging subject matters covered by the documents at issue in this case. Id. ¶ 25. The documents implicate subject areas governed by multiple different bureaus within the State Department, all of which must be consulted before materials may be released. Id. State must also consult with other agencies within the U.S. Government. Id. Moreover, because Plaintiffs’ requests cover communications with outside organizations, State must consult with such outside organizations if the organizations’ confidential business information is at issue."

Though there is certainly tech incompetence involved as well:

"Specifically, it found that one office had not conducted the search by keyword, but rather had only searched the ‘To’ and ‘From’ line of the latest e-mail in an e-mail communication, a method that would not capture responsive e-mails found only in older communications that were part of an e-mail chain."

"In order to do so, as explained in Mr. Stein’s Declaration, IPS must manually open and review every e-mail with an attachment amongst the 34,116 potentially responsive emails in order to determine whether any responsive attachments had inadvertently been omitted."

("Responsive" in this case means "sent or received by four current or former State employees – Michael Fuchs, Melanne Verveer, Cheryl Mills, and Huma Abedin – to or from individuals associated with the Clinton Foundation and Teneo Holdings".)

This is just anecdotal, but it speaks to the technology problem. I live in DC; a few weeks ago, I found myself in a queue with some rather loud young women who all work/worked in a FOIA department of some federal agency (they didn't say which). They spent maybe 15 minutes complaining about various coworkers and their general incompetency. I especially remember their remarks about one man who absolutely could not learn about the redaction feature in Adobe Acrobat. He always printed documents, manually redacted them with a marker, and scanned them again. Not out of any kind of malice--he just couldn't/wouldn't learn a better way.

Actually, from a computer security standpoint his way may be safer.

See this for some food for thought on some of the ways digital alteration can fail you: http://security.stackexchange.com/questions/126932/how-secur...

This is an excellent example of fud created by security pundits wanting to say "actually, that's not secure". If you read that link carefully you will find that the problems discussed are not unique to digital redaction (size/length of the redaction, height of the redaction, etc). The accepted answer also clearly stated that if you use pixel-replacement (this is what DoD calls it) you're fine. Further, there have been plenty of times I've seen a redaction-by-marker break down when the underlying ink bleeds through in photocopy.

NSA published guidelines in 2005 for how to safely redact PDF (as well as word) documents. The corresponding physical redaction guidance begins by suggesting that the inspector physically remove the information to be redacted with an xacto knife. They redact and release thousands of documents, if nobody has reversed their redaction process yet then it's good enough.

Note the may part of my answer.

Also note the answer that points out underhanded ways to leak data.

You then go ahead to recommend just the thing that gp was making fun of: printing and physically altering the document instead of using a software tool.

Basically violent agreement: )

You should re-read my post. I said that there were standards for digital redaction which are considered sufficient and equivalent to physical redaction. Break. If you still want to manually redact... iow, manual redaction is a waste of time, but if you have time to waste here's an effective way to waste it.

May is a pretty clear clue that the security "advice" which follows is fud. Your house may flood even if it's on top of a high hill, buy my insurance product.

I'm from the Literal Police. Do you know why I stopped you?

Uh-huh...

No, it's about how I, and a vast majority of people, actually can't just make up a reason to deny FOIA requests.

Try to be more careful next time, OK?

Seriously. Anyone who has ever done FOIA requests knows how laughably incorrect that is. If it's not an exemption and it's not classified, you can't just deny it.

Furthermore, if it is denied, you can appeal that decision. It's not going to go back to the same person either, it'll go a lot higher on the foodchain. Do you think a commander/director is going to put up with a FOIA officer who is constantly creating work for them by improperly denying requests? Here's what a response to a FOIA appeal looks like[1]. It's comprehensive and needs to give its reasoning.

1: http://www.lsc.gov/sites/default/files/LSC/pdfs/FOIA%202014-...