Well, point cloud renderers are a real thing. There are open source libraries that let you explore millions of points in your browser: http://potree.org/. I even saw a demo (http://nurulize.com/) showcasing animations but I think the main problem with point clouds is that there is no good solution for physics/lighting yet.