As far as I'm concerned, the primes aren't particularly larger or heavier for MF than they are for FF, it's more that the higher resolution and larger sensor size creates physical limits that makes everything slower on MF. No 20fps burst shots in RAW at full resolution with a Fuji GFX100 II, it's only 5fps. That lack of speed is reflected everywhere, because it's doing it on a sensor that's twice the size and twice the resolution, but otherwise it seemed like a pretty amazing camera to me. Speed for me is important though, so I stuck with a Nikon Z8.
It sounds like an MF setup with say a 28mm and a 100mm equivalent lens would require more work but possible (though maybe not enjoyable) to casually carry around.
When shooting raw stills I don’t particularly care about speed. I do enjoy shooting raw CinemaDNG video, there are reasonably priced FF bodies that support it, but if I get a camera for high-quality stills FPS wouldn’t really be a deal-breaker for my uses (I guess it might for bird watching or for sports).