You almost hit upon a real property of the set, namely that if you can find a closed contour of the interior, all the enclosed points are in the set...
A common variant of your approach, then, was to quad tree the area of interest and evaluate the boundary of each square, if it was consistent you just fill in the block, otherwise subdivide. This is a massive speed up for locations like the canonical initial view with large areas of the set shown...
... Of course the problem with this is you are sampling discretely, so inferring that the boundary is closed because a sampling of it is doesn't really work. Your filling in of the white blocks like this is particularly problematic, as that can easily cut off "children" in the set.
You can do this sort of approach properly, but it is a bit more complicated.
I wish I'd had similarly inclined peers at the time. We could have totally geeked out. But I was the only one I knew.