Yep, I abandoned Unity after encountering the same issues as you. My experience is that every Unity GC run takes 10+ ms on my machine, even when there is no memory to clean up.

I tried MonoGame once and the results were much better--garbage collection was fast when there wasn't much to do. However, I'd still choose a non-garbage-collected engine any day because I dislike lag spikes.