Yeah, it bothered me a lot, especially when toggling between architectures and noticing all those errors when It Should Just Work.
At least the only casting you need to do is when doing
myDouble = myPoint.x // This works on 64 bit out of the box but not 32bit.
For now I try to avoid setting like that.
Everything else including myDouble += myPoint.x should work fine with the library.
My first workaround was to go down the conditional compilation path by using the "#if arch([...])" directive and writing specific code for 32-bit/64-bit. Of course, that works, but it just doesn't feel right, so I went with a casting solution instead.
Hopefully this will be ironed out in subsequent beta versions of Xcode, so that we can feel a little more confident about this issue in advance of the final production version of Xcode.