Also hearsay personal experience but a friend of a friend had his house raided by the FBI and all his computer equipment impounded. When he got it all back they told him his hard drives were empty or unformatted. He had them all formatted as ZFS...

I'm unsure as to how to read this; are you saying because they were ZFS, the FBI couldn't read them and because of their lack of expertise, assumed they were empty?

Or that they formatted them before returning them?

I read it as his drives were formatted as ZFS and the FBI couldn't read them because of their incompetence (probably only thinking to try and read them with Windows as FAT32 or NTFS.

FBI assumed them to be empty, or wanted the owner to believe that they had assumed them empty...

Judging by how Windows, when presented with an unknown FS, acts as if the disk were unformatted and immediately offers to format it, my guess would be both...

(And by "unknown" I mean of course "anything that is not FAT* or NTFS"...)

Johnny clubfingers 5-0 clicks Ok, and destroys "evidence" for being a total idiot... I take it he/she was hungover during that 2 hours of computer forensics in the academy. For crooks, this is good news; for stopping potential future victims, this isn't good, and misapplied to innocent/MPAA/RIAA enforcement, it's destructive and lowers LE credibility.

The moral of the story is for individuals, whom should implictly fear government overreach no matter whom is in office, one has to back their shit up and make it SWAT-proof, even if that means running several TahoeLAFS boxes in countries like Switzerland, because running a server (physical or Linode) or just replicating data to a friend's server just doesn't cut it and never did.

If you plug in a non-FAT/exFat/NTFS drive to a Windows machine, Windows will act almost as if the drive is empty and needs formatting in My Computer.

> his house raided by the FBI and all his computer equipment impounded.

Thanks for reminding me to set up Amazon glacier for my FreeNas today. Jesus Christ, the world is a messed up place.

Just a note, read up on glacier pricing and issues. Standard S3 might be better

Or go GCS Nearline, same great price, no terrible gotchas. https://cloud.google.com/storage/docs/nearline

Disclaimer: I work on GCP

I wouldn't read too much into this. SOP for most police forces is a tiered approach to seized drives. They plug it into a windows machine and see what's what. If that doesn't get the immediate results they have to hand it to the professional data recovery people. Even a cursory forensic examination of a drive starts at 1,000$ and may yield nothing if encrypted. With a small pile of drives to look at, they probably didn't bother trying after spotting it wasn't going to be plug-and-play.

That's weird, it looks like they didn't even try something like photorec.

I wonder how would they handle full disk encryption.

Sounds like they call customer support and order a software modification for the entire system...

> When he got it all back they told him his hard drives were empty or unformatted.

Maybe that's what they wanted him to think. I refuse to believe the alternative...

Why? It's people following a forensics script/program which looks for Windows partitions. There aren't a huge number of people who understand this kind of thing and most of them are able to work at places where they're paid much better than the FBI.

the forensics programs (EnCase, FTK) have a lot of problems but they don't assume that the drive is "windows partitions" though it's possible that they don't deal with ZFS.

you would think that national level stuff would have people take more care, but if you're dealing with state or local police, they will have someone who has taken an 8 hours Encase or FTK class driving a GUI to gather evidence and if the tool doesn't support it, there's effectively no evidence to gather.

Yeah, I once had to explain to an officer what a known_hosts file is so they could send MLAT requests and release me from jail without risking that I would wipe (the already automatically wiped) servers myself when I got out.

You're going to go through the trouble to get a warrant for a raid and then apply such gross incompetence to the seized evidence? When was the last time you pulled a hard drive out of a functioning machine, found it to be empty and didn't immediately think, "huh, that's odd" before throwing it in the trash.

I guess it depends on whether the hard drives were the target of the raid or just a collateral of "grab anything that may be useful" mindset. They could already have gathered enough evidence without needing to waste time/money on digital forensics. Hard to say without specifics.

The fact that they went out of their way to assure him that his own hard drives were empty reeks of manipulation.

Color me highly skeptical.

I'll color you trusting: you are choosing to be fixedly _un_-skeptical of the FBI's competence.

Slightly off topic, but it really isn't hard to find partitions. Just run the various software to scan and that's it. I've done it at least twice when accidentally erasing a windows partition (once with fdisk when I misread the output, and at that time I didn't even know what a partition is).

> Also hearsay personal experience but a friend of a friend had his house raided by the FBI and all his computer equipment impounded. When he got it all back they told him his hard drives were empty or unformatted. He had them all formatted as ZFS...

I had imagined that computer forensics "experts" would make this mistake somewhere at some point, but I did not think I would actually hear that it had happened. Thanks for the affirmation, even if it is just hearsay.

It makes me curious about what goofs were made with the IRS hard drives that had "crashed" according to forensics "experts".

This is so true. It has been a number of years since I worked with either agency doing computer forensic work, but my experience was pretty much the exact same. While SS consistently made a concerted effort to investigate crimes and build a solid case, FBI seemed often to be interested primarily in raising the public profile of the cases they worked on to get more visibility in the press, often at the expense of (what appeared to me at least) due process.

Whereas SS had a number of excellent specialists who understood acquisition, the FBI seemed to be wearing clown-shoes most of the time. I'm not surprised at all that they botched this case so badly.

It's clear as day that Apple is on the right side of this argument. It's not their job to bail out the FBI for yet another colossal screwup. Especially not when it damages their product so severely.

<<The FBI is the opposite in every way, mostly because of budget constraints and the subsequent lack of training. I hope that this is a good learning opportunity for them and a chance for them to increase their training budget in this area.>>

Besides the legal precedents and other associated drama, I think is one of Apple's major concerns, and one of the reasons they implemented the "we don't have the keys" approach to their encryption. If the FBI can always just call on Apple (or Google) to fix whatever mistakes they made, there is little motivation for training / getting better on this front, effectively making Apple the computer forensics arm of the government.

The request they submitted to Apple was clearly written by competent people. They knew exactly what and why they wanted to do, how Apple can help and why only Apple can help.

I think part of that is Apple is/was actively working with the FBI to find alternative solutions. I would bet that the engineers described what would need to happen, i.e. the new OS. As is often the case, the Apple engineers probably documented alternative solutions. The FBI took that "solution" and ran with what they described. It's the "well Apple told us this is the only way to do this, but they won't do it for us" scenario.

> I would bet that the engineers described what would need to happen. ... The FBI took that "solution" and ran with what they described.

I think you absolutely nailed it!

For a high-profile investigation like this, Apple would have given the FBI access to the key developers in the security group. The developers are smart guys trying to be helpful. They are not thinking about Apple policy, or constitutional law, or the big picture of world liberty and privacy. They are tasked with finding the solution to a technical problem: How to get access to protected data.

What likely happened--exactly as you already suggested--is that the FBI asked the developers to explain how the security system could have been designed so as to permit easy government access in cases like this. The FBI was asking "hypothetically" of course. The developers happily gave a blueprint of how the system could have been designed.

The FBI now demands that blueprint be implemented.

Apple should have talked to the FBI through lawyers only.

> For a high-profile investigation like this, Apple would have given the FBI access to the key developers in the security group. > Apple should have talked to the FBI through lawyers only.

You went from "would have" to "should have", turning your hypothesis into a certainty...

Hopefully they will be talking to the FBI through lawyers only from now on. :)

Why wouldn't the developers in the security group think about constitutional law? Have you ever seen an internet forum that talked about computer security regularly, yet did not talk about constitutional law regularly? If not, how would those developers have possibly avoided regular reminders about the 4th amendment?

They didn't have to avoid any reminders. They were most likely just asked "how" it could be done, not to do it. The law comes into effect now, where the FBI is trying to get the courts to order them to comply. Simply telling someone how to potentially do something illegal is not illegal itself, and really doesn't cross any boundaries in my opinion. A white hat hacker uses many of the same techniques that a black hat hacker uses, but in one instance it is legal and in the other it is illegal.

Well, that's the overhead of selling closed-source devices.

If you think about it, consulting vendors is probably a better use of taxpayer money then RE-ing every stupid crypto system on the market.

They contacted Apple, did their homework and came up with specific and generally sane demands. They even went as far as suggesting to perform the hacking at Apple site to ensure that insecure firmware doesn't leak outside.

BTW, this last part looks very much like a response to concerns voiced by Apple, which means that the official statements from both sides are just a tip of the iceberg.

>> Well, that's the overhead of selling closed-source devices.

Closed-source may be a pet-peeve of yours, but it has nothing to do with any of this.

Sure it does. If all the hardware and software associated with iPhones was open-source Apple could tell the FBI to fuck off and write their own firmware. Then the only thing they would need Apple for is signing it once it's complete. And if each user could sign their own firmware updates with a key based on their password or provide their own key then it's game over.

They've put themselves in a weird legal situation because they've made it so that they are the only ones who can actually write and sign the firmware the FBI is demanding. A judge would laugh them out of the courtroom if the FBI was technically capable of writing the firmware and demanded Apple's help because it was too hard.

> Sure it does. If all the hardware and software associated with iPhones was open-source Apple could tell the FBI to fuck off and write their own firmware. Then the only thing they would need Apple for is signing it once it's complete.

This is an example of a non-free software feature. Why are the keys baked in and can't be disabled. And "write your own firmware" doesn't solve this problem -- they could just pay a developer to do it $X an hour. A better security model should've been used -- where updates have to be confirmed (read: signed) by the user before they are applied.

> Spivak 1 hour ago Sure it does. If all the hardware and software associated with iPhones was open-source Apple could tell the FBI to off and write their own firmware.

No, not based on the interpretation of the all writs act that the FBI is attempting to use. As far as the FBI is concerned, they could force my Grandma to write a backdoor if they deemed her the best person to do so. Given that she can't answer the phone most days it'd be a lon wait, but I wouldn't put t past them.

--edit to correct auto-correct

Closed-source doesn't change the central issue, which is that the FBI does not have Apple's code signing key.

Poor choice of words, I meant general "closedness" of the platform - from undocumented design, through lack of source code up to centralized code signing.

The only reasonable way for law enforcement to deal with even a single one of those factors is to request help from device vendor.

There is one thing the FBI is very good at, and that's writing a compelling narrative. It's possible that there are highly competently people who know everything, but it's also possible there are moderately but not dazzlingly competent people who are really good at writing a story that feels complete and keeps one from asking questions outside the narrative.

You don't just make up deep technical knowledge with a compelling narrative.

Though, on second thought, I have to add that we don't know how many back-and-forth mail exchanges happened before they were able to come up with the officially published request.

Maybe they were just competent at working around excuses from Apple.

I think like the police in UK once a FBI officer gets experienced in computer security the leave for the better paid private sector.

Exactly. Yet another reason to fight the court order. We should expect FBI to be competent, embarrassed if they aren't, and fix the problem. It's not a good state of affairs when a company is more trusted to do forensics.

>I think is one of Apple's major concerns, and one of the reasons they implemented the "we don't have the keys" approach to their encryption.

They do have the keys. A 4-digit pin is useless if you have Apple's private key.

Completely false, as has been endlessly detailed by nearly everyone at this point.

Software update signing keys, which can't be disabled by the end user. This is what most people would consider "a flawed security model". Even UEFI lets you change the trusted booting keys.

Please enlighten me. Is this not exactly what the FBI is asking for? For Apple to flash a custom version of iOS that doesn't have the software rate-limiting and auto-wipe, which only someone with Apple's private key can do. A four-digit PIN is only secure in combination with those features. Having Apple's code-signing key is in fact "having the keys", except in the most pedantic literal sense.

Then they should apologize to Apple for tarnishing their name all over the media for "helping terrorists" to the point where the front-runner of a party, who has a chance of becoming the next president, called for their boycott because of it - when all along it was FBI's dumb mistake for not getting that data and Apple has no responsibility to fix FBI's stupidity. I think it's only fair.

That's assuming it wasn't all staged to take advantage of this situation to pass some backdoor law or set a precedent here, in which case, I don't expect the FBI to retract anything, because then their goal isn't to unlock this phone, but to set that precedent.

Just curious, and sorry if this is intrusive, but your blog is very up front about your employment history... reddit, netflix, paypal/ebay, sendmail...

At which job were you working so closely with the FBI and SS that you were able to ascertain their technical abilities?

In the mid-90s I had several encounters with the FBI while running an ISP in Oklahoma City. They asked me to write code to pull out specific entries in our massive amount of dialup logs and when I told them they had to pay me, they refused based on budget constraints.

I can well imagine jedberg has had to deal with them just being a systems admin for such large deployments as reddit and paypal.

Just curious - which one?

I started out teaching "Welcome to the Internet" classes once a week, and then was the first tech support guy at ioNet / Internet Oklahoma.

Worked my way up to assistant sysadmin before I moved to Austin to be a sysadmin for Texas.net. April 95 to September 96.

And my first day on the job was April 19, 1995.

Talk about a hell of a first day... We pretty much switched into full "get news about the bombing online in some form or fashion, however possible" mode for about 48 hours. This was before CNN had a huge online presence, so we had stuff going like a RealVideo stream of a webcam pointed at local TV news, a couple of people went down to take pictures, etc, etc...

The Internet Shop: https://web.archive.org/web/19990423075248/http://www.thesho...

Started it in mid to late '96.

Those early ISP days both sucked and were fun in their own special way. Trumpet Winsock for Win3.x users! Getting OS/2 online and older Macs! Trash-talking in the local Usenet groups!

Where you kept a copy of the Solaris Operating system for all of your coworkers to peruse. We use to track you back then as you were one of the larger pirates of our software. :P

What? That's a bit libelous, I'd appreciate an explanation.

If it happened at ioNET after September '96, I wasn't involved.

All of Texas.Net's systems ran legit copies of Solaris; a large amount of them were brand new and came with OS entitlements. I remember going to the post office one day to pick up our brand new copies of Solaris 2.6.

Are you talking about me running SUNHELP.ORG, which started in '97? That was (and is) a third-party user community and resources, along with mailing lists. It did not provide Solaris downloads.

I happen to have a (personal) archive of Solaris releases, but it's not public access.

The closest thing to "piracy" I could ever be accused of was reposting design documents for an unreleased system that I found on Sun's own publicly-accessible website in 2000.

That whole scenario is explained in detail here: http://www.sunhelp.org/letters/

If anyone at Sun held me in bad regard, it was never mentioned to me by anyone, and I had a lot of contacts at the company.

- I was one of 250 people to be picked as an external pre-release beta tester / community liason for the release of OpenSolaris.

- In 2005, Sun donated a fully loaded T1000 server (8 cores, 8G RAM) for use by me in running the site. I'd think that if I was a huge software pirate the company wouldn't encourage it by giving me expensive hardware for free.

Pictures: https://www.flickr.com/photos/mrbill/sets/72057594124431136/

Thanks.

Texas.net is what we're talking about. And by software, I mean Solaris source code.

This is the first time I've ever been accused of being a pirate! There's a first time for everything I guess.

To be perfectly honest: I vaguely recall source of an older Solaris version being passed around the user/hobbyist community, but I wasn't the originator. I'm sure I downloaded it at one point, but don't remember ever doing anything with it OR putting it up for coworkers to browse.

Like the (released and quickly discontinued) version of Solaris 2.6 for PPC that ran on certain RS/6000s, it was "out there" and easily obtainable by anyone in the hobbyist community if you asked enough people.

No idea what was done at TN after I left; I had issues with how management treated the technical staff and resigned in late '98.

It's been 18 years, my memory is a bit fuzzy.

Anyone who has ever met you has heard you mention this. :P

Nice karma you have there.

MrBill is famous for more than just solaris.

Dunno if I want to be called "famous". Tried my best to do what i could for the Sun / hardware rescue communities, but I'm still a relative nobody in the grand scheme of things.

What's really weird is being called a "peer" by people who I looked up to when learning and just getting started. Massive case of impostor syndrome...

There are a ton of people out there, who are better talented and have contributed more to UNIX / Linux, open source, and the hobbyist/maker community in general that deserve recognition and fame.

I'm just a fat old fart sysadmin who's had a good run and was lucky to be able to enjoy most of it. The best I can hope for is to be thought well of by others.

Ebay and PayPal. The secret service doesn't like it when foreigners steal from Americans (their main job is to protect the currency, protecting the President is an add on).

I would expect Paypal/ebay are involved in umpteen investigations at any given time, from money laundering to fraud to drug dealing and so on.

Reddit is also a cesspool of activity from all sorts of people, so there might be some monitoring involved there.

Probably via criminals that were posting on reddit

I've worked with vice-principals at elementary schools investigating porn on library machines who had a better grasp of forensics.

Really, I teach a course in a local forensics program and have helped a couple local schools over the years. Any teenager with an iPhone could have explained that resetting the AppleID was a bad idea. There are plenty of very intelligent kids out there who need jobs. For the FBI to act in such a manner is inexcusable. Ditch some of these hack cops and hire some proper technology experts.

I worked with multiple agents (FBI and Secret Service) in 1999 - they were not competent back then. I'm glad they have improved.

i suspect just like every other large organisation, you have small islands of excellence, spread across oceans of mediocrity.