How do you propose the mayor's office identify senders and recipients from e.g. that law firm without combing through every message? What if their email domain has changed? What if someone used their personal email address? If they left out any records, you could sue them.

The remaining reasons, mainly an inability to filter emails by department, sound normal. If you're designing a database, can you build an efficient index for every possible query? No. The mayor's office has not optimized their records for random people making FOIA requests. Nor should they. I'd rather my tax dollars go to potholes, rather than axe-grinders.

You'll have better luck with more legwork on your end.

I'm not asking for a complicated query system, really. An upgraded version of outlook would probably do the job. Or an email to their IT department who could do it. That's what I ended up doing last time which ended up turning into a year and a half battle to prove that the numbers were releasable under FOIA.

The judge ended up telling them to JFGI [0], since not googling it was costing the city thousands in court fees. That should address your concerns. It probably didn't get me every releasable number, but I'm sure as hell not going to sue and waste another six months. It takes too long to sue, they know it, and it's a huge factor in why this is all so difficult to do.

Please realize that paving the way for others to obtain this data is a huge part of the end goal. In some ways, I'm trying to fix the potholes of FOIA.

[0] "Next, DoIT and its counsel conducted online research on each remaining responsive phone number — in other words,they "googled" each number to determine whether that number was publically listed, and, if so, to whom it belonged."

Edit: Cleaned up (a couple times).

NB: I'm not expecting they'd actually do the below. The political downsides of being efficient in your FOI requests are too great to seriously expect somewhere like the Mayor's office would dedicate effort in to achieving it. But document/record management is not some mystical never before done thing.

Somewhere in the 9.5 million dollar budget of the Mayor of Chicago [1] they could squeeze out $20k for a HP Records Manager setup (or any of the dozen competing software packages) and just blindly catalogue all emails in to it. Even without setting up ANY of the business classification features it makes this kind of search trivial.

Perhaps it could come out of the 1.4 million dollars allocated to the "Innovation delivery team" who "Creates and implements new solutions to improve the efficiency of City government.".

[1] Page 59 Program and Budget Summaries by Department Finance and Administration Program Summary OFFICE OF THE MAYOR (http://www.cityofchicago.org/content/dam/city/depts/obm/supp...)

Obscurity by bureaucracy at its finest. We all do this to some extent in our day to day jobs. Remember that behind these emails are real human beings.

Keep pressuring, keep pursuing, for you, for us. Observe, pressure, and report. Rinse, repeat. There are plenty of us that appreciate your hard work in holding the system accountable. Keep up the good fight ++1

Totally agreed on the points in your first paragraph. I try my best to help them out and stay as friendly as possible. For example, the Department of Finance senior FOIA officer and I are on a "hey man" basis now. :)

Thanks! This all started in November of 2014 and I doubt there'll be an end for a while.

Not to derail too much, but this sort of work is one of the main reasons why I'm in favor of a basic income. It sucks that it's impossible to live freely while doing this stuff, without being a journalist (bad at writing, so my reliance is on other journalists) or without having a stable job. Not having either right now makes me think to go to patreon for "basic income", but that just feels dirty.

A comment from me from below:

"What is the new becomes the old in no time at all. Something will, and realistically has to, fill the void of high level domestic security. And unless the people that make up it are fundamentally different, or unless the structure itself and its system of rewards and punishments are different, we will only repeat history.

Laws and education are our only hope. Right now these two things are time consuming and are full of hardship so much so that the average person has a hard time taking on responsibility in these domains in their daily lives. If we could lesson these burdens for the average person in these areas then I think, I know, that we are really on to something."

If we could limit the burden for people like you then it would be a huge net positive. If we all figure out that basic income is our best bet then I am all for it. I remain skeptical though and think that there are better options in the medium and long term.

You should have given them an explicit list of domains to query for in the headers.

Based on history from previous and successful requests, I shouldn't have needed to. They've fulfilled requests as vague as that in the past.

Submitted two requests two days ago that might be a bit better than providing the domains, though. One for DNS resolution logs of the mayor's office and another for all domains and times for all emails sent from their office. If just one of those are successful, then I don't need to play any more FOIA games with mayor's office.

Sure, I could just submit the same request to their IT department like those two... but, no. I'm getting that information from the mayor's office directly, dammit. :)

DNS resolution logs...

Brilliant. I love you.

Thanks. :) & <3

Do a FOIA request for the staff directory?

That's actually all available at data.cityofchicago.org. My first thought was along that line. Turned out to be about 60 different people, so it was still a bit much.