The issue is that it's just a token that would simply say Passed/Fail. So it's trusting the client/browser.

It's more than that -- from what I'm seeing, it looks like it's a cryptographic token, presumably signed by a certificate that's embedded somewhere inaccessible in the device (probably in the secure element).

If the token is signed you could validate it with Apple (or the vendor that implemented the face recognition on the device, eg Samsung, Nokia, pinephone etc).

You just need an open standard, you could even embed the url of the validating api in the token, so anyone could create their own Face ID provider.

That's precisely what the attestation section of the talk describes. This is all part of the WebAuthentication standard.