Regarding the planes: you can modify each independently. Just set up your palette in the right way! For 4 planes, something like:
0000 - light grey (background)
0001 - purple (lines)
0010 - dark grey (shadow)
0011 - dark purple (shadowed line)
0100 - white (ball)
0101 - white
0110 - white
0111 - white
1000 - red (ball)
1001 - red
1010 - red
1011 - red
1100 - pink (ball)
1101 - pink
1110 - pink
1111 - pink
Now you can manipulate each plane separately, as appropriate, ignoring the rest, and you'll get the right result. Of course, this limits the number of colours you can have at once, the limit being 1 layer per plane, and then N+1 colours in total for N planes.With the ball demo, you have 2 x 1-bit layers (2 colours each, 1 shared), and 1 x 2-bit layer (4 colours, 1 shared), for a total of 7.
All of this applies to the Amiga just as it does to the ST, because both operate using the same principles in this respect. (The Amiga's dual-playfield mode existed, I suspect, only because it was- cheaper than having more palette entries! If you had 64 palette entries, there'd be no need for it.)
(As for the resolution, I think the ball demo was 320x256 pixels? - certainly seems to have been the case judging by the screen grabs I could find. So I think my calculations stand. Don't ask me why they didn't go for 640x256x4bpp, because I have no idea.)
Right, forgot ST cant do 640×512 4bit, so it would have to fake it at 4x smaller picture, that shrinks requirements considerably. What is ST memory bandwidth? something like almost 40KB per frame?
Sure, you can use pre rendered screens, at that point you could argue C64 could do raytracing because this: https://www.youtube.com/watch?v=yxZ7Idi2Bi4 displayed on real C64 with 16MB ram expansion, every frame of animation is 2 frame buffers in 320x200 ~16color NUFLI mode.