I'm speculating here, but surely by this argument Apple has a backdoor to unlock the phone if it physically has it.

Of sorts. Apple (and unless they have been compromised, only Apple) could replace the firmware of a locked phone with a new firmware that will accept millions of unlock attempts without erasing the phone, at which point the password could be found by brute force.

The FBI asked Apple to create this firmware, and Apple refused.

And presumably Apple could have designed it to not allow firmware updates without being unlocked. Or require overwriting of "secured" secrets if an update is applied anyways. It probably wasn't worth the potential user hassles?

My loose understanding is that this is in the works (not an apple employee).

No.

What Apple had was the option of updating the phone with a compromised (or compromisable) security implementation.

Apple refused to do that.

The FBI did find an exploit (which AFAIR they've refused to disclose, in controvention of previous policy if not regulation and/or law), but that represents a flaw in implementation which Apple can then remedy.

http://mashable.com/2016/02/16/apple-hack-san-bernardino-pho...

Rather than asking Apple to obtain and provide the device’s passcode, Tuesday's ruling orders Apple to provide the FBI with special software that can act as a workaround for the iPhone's built-in security. Court documents suggest Apple should either provide software that bypasses the need for a passcode entirely or software that allows the FBI to unlock the phone by running an endless combination of passcodes until they are able to unlock the device. The latter method would also require software that disables the iPhone's auto-erase feature, which wipes a device after a certain number of unsuccessful attempts at unlocking.

http://fortune.com/2016/04/27/fbi-apple-iphone-crack/

If Apple can take the phone and unlock it with the tools and knowledge at their disposal, then they have a backdoor for it. The distinction that you're drawing is meaningless.

A backdoor is typically reserved for solutions that bypass the existing security. As I understood it, the only thing Apple could do was provide a firmware without rate limiting. This would have allowed the FBI to perform an exhaustive search for the key. We already know that brute-forcing a key is always possible, that's not a new way to bypass security.

No, brute forcing a pin is not supposed to be possible. And yes, it was always possible, but only if you have Apple's secrets. A short PIN is only secure because of rate-limiting. Loading a firmware without rate-limiting is effectively hacking the phone. Apple possessed backdoor to phones that have a short PIN.

The FBI was simply asking Apple to walk through a door that Apple created (perhaps unwittingly) for themselves. The case had nothing to do with forcing Apple to backdoor their products, which would mean mandated vulnerabilities that Apple is forced to ship phones with.

Apple can, and apparently is, designing future phones to not be vulnerable to this flaw. TPM chips, widely used in PCs, are not vulnerable to such an attack because the rate-limiting is done in the TPM hardware itself.

The statement assumes the attacker knows what they're doing when they have access to the phone.

To be fair, the FBI did get into the phone, and could have much sooner had they not wasted their time fighting Apple.

They got into an iPhone5C. If that phone was a 5S (and the FBI screwed up the MDM process just like they did) they would be out of luck.