Right, forgot ST cant do 640×512 4bit, so it would have to fake it at 4x smaller picture, that shrinks requirements considerably. What is ST memory bandwidth? something like almost 40KB per frame?
Sure, you can use pre rendered screens, at that point you could argue C64 could do raytracing because this: https://www.youtube.com/watch?v=yxZ7Idi2Bi4 displayed on real C64 with 16MB ram expansion, every frame of animation is 2 frame buffers in 320x200 ~16color NUFLI mode.
Regarding the planes: you can modify each independently. Just set up your palette in the right way! For 4 planes, something like:
0000 - light grey (background)
0001 - purple (lines)
0010 - dark grey (shadow)
0011 - dark purple (shadowed line)
0100 - white (ball)
0101 - white
0110 - white
0111 - white
1000 - red (ball)
1001 - red
1010 - red
1011 - red
1100 - pink (ball)
1101 - pink
1110 - pink
1111 - pink
Now you can manipulate each plane separately, as appropriate, ignoring the rest, and you'll get the right result. Of course, this limits the number of colours you can have at once, the limit being 1 layer per plane, and then N+1 colours in total for N planes.With the ball demo, you have 2 x 1-bit layers (2 colours each, 1 shared), and 1 x 2-bit layer (4 colours, 1 shared), for a total of 7.
All of this applies to the Amiga just as it does to the ST, because both operate using the same principles in this respect. (The Amiga's dual-playfield mode existed, I suspect, only because it was- cheaper than having more palette entries! If you had 64 palette entries, there'd be no need for it.)
(As for the resolution, I think the ball demo was 320x256 pixels? - certainly seems to have been the case judging by the screen grabs I could find. So I think my calculations stand. Don't ask me why they didn't go for 640x256x4bpp, because I have no idea.)
The ball looks like it's about 128 pixels wide and 100 pixels high, so 8 words x 100 rows. Note how the screen appears to be laid out in a bitplane-minded fashion: you have a 1 plane backdrop, a 1 plane shadow, and a 2-plane bouncing ball. So, you don't have to mask in your background when writing. Load source, write dest.
You'd pre-shift your ball lots of times to make this go quickly. (The shifted copies would have the ball's rotation applied, so you'd get a two-for-one with this.) One ball = 8 words * 2 bitplanes * 100 rows = 3,200 bytes. Plenty of room, even with 512K.
So your loop for one row will go something like this, I guess. Assume A0 points to your 2-plane sprite data, arranged bitplane 0, bitplane 1, bitplane 0, bitplane 1, and so on, for 8 words, then your 1-plane shadow data, again 8 words. Assume A1 points to bitplane 0.
First draw the ball:
So, 200 cycles per line.Then draw the shadow in the same sort of way into bitplane 2 (this is the reason for the (A1)+ and the -4s everywhere above - it lets you start off with (A1) here, rather than 4(A1), saving you 4 cycles).
So, for that, 136 cycles per line.Cycles per frame = 8,000,000/50 = 160,000; cycles per ball = (200+136)*100 = 33,600.
Taking into account that you'd have to erase the old ball and its shadow, and you'd have to advance your pointer from row to row, and whatnot - maybe 25% in total?