The problem wasn't storing raw memory offsets, it was not having some way to validate the data at runtime.

Yes you still need to validate the keys to avoid logic errors, but you can avoid the memory errors.

The addresses aren't being generated internal to the program, so there are no "handles". They are referencing external data by design.

That's like saying "you shouldn't use a hard-coded volatile pointer to reference a hardware device". No, you literally need to do that sometimes; especially in embedded software.

Of course it doesn't have to be either/or; you can have fast + secure, but it costs a lot more to design, develop, maintain and validate. What you can't have is a "why don't they just" simple and obvious solution that makes it cheap without making it either less secure, less performant, or both.

Given all the other mishaps in this story, it is very well possible that the software is insecure (we know that), slow and also still very expensive. There's a limit to how high you can push the triangle, but there's not bottom to how bad it can get.