Yeah, all of the manual casting (which I feel like sometimes takes more trial and error than it should to get to compile) and the awkward null checking are the two most painful points. The debugger being basically useless is a problem, but it's something that will get fixed eventually. I have less faith in the first two points though because initially they were less painful but in later betas they became more painful.
Probably the biggest issue with Swift right now though is that it has to accommodate all the older Cocoa APIs so in practice it's often just a slightly nicer syntax for the same code. Cocoa functions tend to return untyped values or things like NSArray so you have to do a lot of manual casting, etc.