In practice I agree it's easy to write code that's slow, but you can't knock the language itself for user error. They are still working on improving the user experience, such as how 4.2 will remove redundant generated retain/release calls. And hopefully they make the compiler smarter at figuring out which specialization functions to output for generics/protocols.
There is a subset of Swift that is fast. But if you tell people that Swift is faster than Obj C, they will think that they can rewrite code in Swift and it will be faster. That may be the case at some point in the future, but right now that's just not true yet.
In my experience, the opposite is the case. Swift is a great language, it's safe, and nice to work with. But those bits where you are processing data in a tight loop -- you still may have to do that in Obj C.
Obviously this depends a lot on your problem, but I have seen 2x speedups and more for things like string processing (rewriting things from Swift in Obj C)
And as I said, that's without using the fancy parts in Swift (things like type erasure absolutely kill performance)
Since it is missing mutable arrays and strings, your code often ends up copying a lot of memory around if you're not extremely careful, and you quickly get accidentally quadratic complexity.
A lot of APIs are just wrappers around ObjC APIs, with lots of memory management calls required for bridging. You really need to be careful what calls you put in a tight loop.
And that's before you even start with the good stuff. Protocols with associated types are so slow that they are unusable for anything performance sensitive. Theoretically the compiler should be able to optimize it incredibly well since it's all statically typed, but in practice the compiler seems to never be smart enough. Call stacks get very deep very quickly (protocol witnesses, closures, ...).
In my experience the easiest way to make Swift code fast is to rewrite hotspots in ObjC.