In X11 it is just as expensive to change the foreground color as to draw one dot (since all the overhead is sending the command), but you could efficiently draw a lot of dots of the same color with XDrawPoints, so it was worth it to sort the pixels by color and then draw them in a few same-colored batches, instead of switching colors between each pixel you draw.
I disabled that crazy code for the OLPC port since it was always running locally and XPutImage would always always use the shared memory extension, but in the days when you wanted to run remotely over a slow network, or the local server didn't support the shared memory extension, it was well worth doing all the work of comparing and sorting individual pixels to incrementally update the map!
https://github.com/SimHacker/micropolis/blob/master/micropol...
(I'm sure that code used to work on old 8 bit displays, but for some reason I never bothered to figure out, it was buggy on the OLPC's 24 bit visual, so I #if'ed it out of its misery. That's X11 for you.)
https://github.com/SimHacker/micropolis/blob/master/micropol...
You can do that on various Linux distros with xpra [1].
Rightly or wrongly, modern GUIs tend to be sufficiently graphical that X turns into effectively sending bitmaps anyway, so in my experience a lightly compressed VNC is more responsive.
It is nice that in X individual windows can live independently, but adding that to your windowing manager feels like less work, and is how for example windows works nowadays (don't know other windowing systems well enough).