News for hackers. Stuff that matters.
Re:Datashader: turns even the largest data into images, accu... by simplyinfinity on Monday February 11, 2019 @11:49AM (#19135803)
Is there anything similar for network graphs?
Re:Datashader: turns even the largest data into images, accu... by lmeyerov on Monday February 11, 2019 @01:21PM (#19136763)
Yep -- at https://github.com/graphistry/pygraphistry, we started by making millions of nodes/edges interactive. If you use notebooks, can signup on our site and get going. The trick is we connect GPUs in the browser to GPUs in the cloud, and encapsulate it enough that you can stick to writing standard SQL/pandas/etc.

We've been curious about server-side static tile rendering for larger graphs, but has been on the back-burner. (We already connect to GPUs on the server, so not rocket science.) Currently, we're actively increasing how much can be ingested + computed on, such as for finding influencers, communities & rings, etc. However, visualizing that hasn't been an operational priority for our users. More useful to generate the communities, and then either inspect individual ones, or see how communities stitch together: quickly run out of pixels otherwise due to too many edges. Likewise, we're building connectors to gigascale-petascale graph DBs: titan, janus, aws neptune, tigergraph, spark graphx, etc.

We still are interested, but more for when we start supporting geographic maps: you can see that is the primary use for datashader. Also, because data art is fun :)

Re:Datashader: turns even the largest data into images, accu... by jbednar on Monday February 11, 2019 @02:35PM (#19137514)
Geographic maps aren't the primary use for datashader; those are just easy examples that people can appreciate without a lot of explanation. In practice we use it for any large datasets that we don't want to subsample before visualizing them.
Re:Datashader: turns even the largest data into images, accu... by lmeyerov on Monday February 11, 2019 @03:09PM (#19137786)
Yes, definitely more capable. I've seen them primarily used in scatterplots (x, y, maybe z) + maps. Curious where else you're seeing the 80/20 breakdown if not there...
Re:Datashader: turns even the largest data into images, accu... by jbednar on Monday February 11, 2019 @02:20PM (#19137402)
Datashader itself renders networks: http://datashader.org/user_guide/7_Networks.html
Re:Datashader: turns even the largest data into images, accu... by jointpdf on Monday February 11, 2019 @01:06PM (#19136594)
Perhaps Gephi? It was used for those Graphcore neural network visualizations (millions of edges and nodes) and they look stunning.

[1] https://gephi.org/features/

[2] https://www.graphcore.ai/posts/what-does-machine-learning-lo...

Re:Datashader: turns even the largest data into images, accu... by simplyinfinity on Tuesday February 12, 2019 @04:20AM (#19142143)
I've used both gephi and cytoscape, i have about 2 million nodes with about 3 million edges, it's hard to visualise at once. however i've reduced the graphs to only sub networks with more than 5 connections, the remainder is about 600k, which is still painfully slow to visualise
Re:Datashader: turns even the largest data into images, accu... by maliker on Monday February 11, 2019 @01:20PM (#19136748)
My team has had luck rendering an SVG from the graph and sending it to a browser. It works well for about 10k vertices and edges. Above that scale we use datashader, and we're investigating a potential to move to QGIS. We tried Gephi a few years ago and it had trouble at these scales.
Re:Datashader: turns even the largest data into images, accu... by IanCal on Monday February 11, 2019 @01:43PM (#19137036)
You can render networks in datashader, there's a line primitive.

I added edge bundling (probably the slowest thing in datashader!) but I know there's examples of flight path rendering in the video I linked in another comment.