It wouldn't be especially surprising to me.
JSON's parsing performance is difficult, whether or not the language doing the parsing is fast or not. There's a reason that there are almost always libraries claiming faster JSON performance, regardless of the language.
Go is marketed as a fast language, and they're still trying to build a high performance JSON parser [0].
Parsing JSON will always be slower than most of the alternatives.
Swift’s slowness is kind of unfortunate, although I hear that there is work being gone to remove a lot of the lifetime cruft that was going in behind the scenes to make it slow. And unfortunately I think the primary motivation for Swift’s performance is driven by its primary application, which is UI development, so “good enough” performance usually works…
WWDC has a couple of talks regarding runtime improvements, and low level coding including unsafe style coding with Swift.
I agree with you, it's weird that Swift JSON is so slow and binary isn't necessarily faster.
My exasperation is you were surprised by JSON becoming an issue, when this was the most controversial design decision about the project. To me, it's completely unsurprising.
Also, Swift is marketed as a fast language (also based on LLVM), yet in my measurements it's 20x to 50x slower than Rust for JSON processing. I found that surprising. Would you not?