Last time I implemented something like this was in uni about 10 years ago now, but I recall that not being the case – you should be able to render just the visible area, as (at least in the case of Mandelbrot) the data is independent.