Raw memory access is something you normally need to create vulnerabilities, not to exploit them :)
It's an ergonomics thing, not a "can't" issue. There is a reason I called out Swift in particular—their "unsafe" APIs are so horrid to use that they make you regret doing unsafe things in the first place. Plus, throw in FFI and now you've got an even worse problem because not only are you forced to use the unsafe types, but often a lot of the critical APIs you need to interface with (in *OS exploitation, mach is the worst offender) have such funky types due to their generic nature that you have to go through half a dozen different conversions to get access to the underlying data.
https://github.com/hellman/libformatstr
You can do something like this, no need to work with raw memory.