Note that browsers pass downloadable fonts through a sanitizer before they even consider handing them off to anything else that might need to parse the font. And browser security teams have spent years now fuzzing both those sanitizers and various font libraries...

There's still a lot of attack surface here, but "only just begun to experience life with hostile inputs" isn't quite true either.