While I don't know the Rust code, this isn't necessarily solely a Unity problem. Unity's C# API provides a lot of footguns that game developers tend to not think about in the near term. Lots of code bases I've seen in Unity don't do a good job of caching component instances, or just never opened the profiler to see their GC is off the charts. Unity can run really well if the proper thought and care is put into it.

Loading into a new world takes a long time, but once I'm in there, I get great performance.

That said I'm in the "dozens of hours" bracket and not the "thousands of hours" one, so maybe I just haven't hit that yet.