If IB's performance and stability issues could be fixed, I think its biggest problem is the near-unintelligible spaghetti XML that it generates for XIBs and storyboards, which is a huge pain for VCS and frequently cause of merge conflicts (which must be manually corrected). If they just cleaned up the XML output or switched to something more friendly to manual editing that'd help a lot.
xibs and storyboards were also XML but IB would insert attributes that had no visual impact, move lines of XML around simply upon opening the file, or randomly change x/y values of the viewport upon opening the file, or update float values to use 15 decimal places instead of 2, and so on. Apple's insistence on making the visual editor have priority over text led to a worse developer experience as teams grew.
Most developers I know in the ecosystem tried and tried to like the visual editor, but ended up falling back to code. Why? Source code is pretty great. It’s easy to read, write, share, diff, and organize.
SwiftUI has, again, a very modern and interesting take on visual editing. The code is the truth, but there’s a rich, realtime preview canvas that makes rountripping possible. For reasons, it’s unfortunately all but unusable for a lot of folks with complex Xcode projects, but it’s a very good idea.
In summary: Visual layout editors and text are both pretty great. The devil’s in the details.