If you didn't have the "random" buffers, you'd complain how slow it is. Syntax highlighting? Needs a boatload of caching to be efficient. Code search? Hey, you want a cached code index. Plugins? Gotta run your python code somewhere.

Run vi/nano/micro/joe - they're optimizing for memory to some extent. vi clocks in at under 8 MB. You're giving up a lot of "nice" things to get there.