k is right that d3 has some helper functions for drawing on canvas, so most of the methods here will work:

https://wattenberger.com/blog/d3#drawing-svg-shapes

Canvas is way more performant than SVG for a lot of shapes, since they don't have the overhead of a DOM node.

But in general, you want to work outside of the browser if you have that much data. What I'll usually do is either:

- write a python script to process my data and use d3 for the final visualization, or

- have an API that returns a sliced portion of the data and create an interface that will request different sliced datasets when interacting with filters, controls, etc.

This election dashboard is a good example:

https://currents.parsely.com/election

it hits two endpoints, which return basically the maximum amount of data you can expect a browser to handle (the massive dataset is processed elsewhere). You could create an endpoint to return different views when you click on, eg, a different date, although then the user would have to wait for the data to load.

As far as I know, D3 can also render into a canvas, which should eliminate the DOM/SVG nodes overhead.

Yes, in my experience this can get to considerable scale (hundreds of thousands and beyond) but at a price. You need to do all your own management for things DOM/SVG were doing (e.g. hit detection, styling). You still need to send the data to the client and probably store it locally (customers will enjoy learning how to start Chrome from the command line with flags like --args --js-flags="--max_old_space_size=8192"). The client environment still doesn't have much support for data management or processing unless you bring in some other JS libraries. I'm a big fan of JS development but the criticism is on target.

True story.

There is no endless RAM, even for browsers :)

DOM-nodes are rather heavy compared to simple JS objects, so getting rid of them can bring some visualizations that are too heavy for the browser back into the realm of doable.

After that you have to optimize your other data-structures. If you have objects with many boolean fields or strings that are essentially used as enums, you can try to replace them with bit-fields, I read some library creators (BlueBird promise) got much performance gains with them.

This is where you want to make POST requests to an API you have / have built to get new data on the go! Of course, d3 is definitely still limited in scale, but just want to mention that technique bc it does allow you to slice & dice pretty huge datasets as part of the interactive browser presentation.

One of my colleagues, for his PhD research, wrote a jQuery-based tool to do something similar to that (still under development): https://pdfs.semanticscholar.org/a5b4/5bb7b0091963781390557e...

That's where DC and crossfilter might come in handy. One of their demos uses 250k rows of airline transit data and handles it well.

bokeh then