Is there any open source library that does that? I know in theory many things are possible but it would be great to actually test it/see it in action. I never thought of Swift as a language with zero cost abstractions and deterministic performance. If it works I think it really gives it a leg up.

It's pretty uncommon, so I'm not sure. If I may toot my own horn a bit, you may find the code to my memory dumper interesting:

https://github.com/mikeash/memorydumper2

I also gave a talk on this subject a couple of years ago. It's out of date by now, but the basics are still more or less intact:

https://vimeo.com/107707576

Or peruse the standard library documentation for UnsafePointer, UnsafeRawPointer, UnsafeMutablePointer, and UnsafeRawMutablePointer. These are built-in types that the compiler knows about, and code that uses them should compile down to essentially the same stuff as C pointer code would.

This is especially true in Swift 3, which has many improvements around this type of programming.