Little special hardware on those old macs, memory was 16-bits, framebuffer was part of memory (memory was expensive), no blitting hardware, no hardware cursors (software removed the cursor, rendered to the screen, copied the data under the cursor and restored it). Screen refresh also refreshed memory, during horizontal blanking it fetched audio data
There's no really separate video memory on the original Mac, just the 21kb buffer that's updated by software. When it needs to, the video circuitry grabs the bus and reads the pixels it needs and outputs them. (There's probably some caching in the controller, but not a lot - a single screen line tops, I'd wager)
Modern machines have huge overheads because of the indirection, virtualisation, and protection mechanisms we desire. On a webpage, you obviously don't have direct access to the framebuffer, but are drawing into some in-memory canvas. Drawing a single pixel in Javascript is going to go through many function calls, with probably orders of magnitude more instructions than you would have on an old Mac. There are still ways of getting (near) direct graphics access that is screaming fast, since people do write games and play videos and such, but there is much more ceremony to it now.