justtoni

So where do you exactly store data inside the cluster, and can I send it to a self-hosted datastore like Clickhouse or S3?

Everyone was/is a beginner at some point. Even if you think about other areas in life, you need to begin in order to improve, e.g. driving a car, riding a bicycle, using any tool, learning to cook, etc. That's why I recommended to do a tutorial first. A tutorial that is building a small project in a tech stack you are looking for.

For the text based, blogs, books or video courses, it is all up to you. Some people will do better with blogs, some with video courses - there is no general rule. I like text based tutorials because I usually want to try something on my own before continuing with the tutorial. That's why video courses don't work for me because my pace needs to be aligned with the pace from the video which is almost never the case.

Practice. Just do it. Start doing it. Something. Anything. Start with any project that is meaningful to you. It doesn't need to be large or have business value.

You can start with a tutorial that covers building up a small project just to get that first introduction with the technology stack, but don't get into the tutorial loophole because it is usually one way (step by step) learning.

vis-network is a great library. We actually started with it because we liked the API, the styling capabilities, easy to use event handlers. But we experienced performance issues when we wanted to simulate and render larger graphs because it was done in the main thread, the whole UI was blocked by it. We tried to fix it in vis, but simulation was using DOM so it was super hard to split simulation and renderer.

Actually that was the main reason (along with the note that main authors are not contributing to visjs any more [1]) for a creation of the Orb where we fixed the blocking UI issue with graph simulation. Orb engine has two parts now:

* Simulator that doesn't depend on the DOM so we can move its heavy calculation to the web worker - we use d3-force for it [2]

* Renderer is pretty much influenced by vis-network, using similar style mechanism and canvas drawing capabilities (we credited vis-network in our code for those sections)

[1] https://github.com/almende/vis/issues/4259#issue-412107497

[2] https://github.com/d3/d3-force

Maybe they hired a CFO who said on his first day: "I am sorry, but you are fucked".

SVG excels at drawing a small number of large elements, where it sometimes performs even better than the HTML5 Canvas or WebGL.

D3 is a great product. We really like using it. D3 has a great integration with svg rendering, but we needed a canvas rendering which resulted in a merge of vis.js (guys did a great job on canvas rendering) and D3 (just the simulator: d3-force).

Great destruct of the existing title to get to the better title!

Even though "end all engines" might not be 100% correct, because the initial idea of the Orb is to make a single interface where the background (simulation and rendering engine) can be changed.

To add to your title, "you shouldn't build an engine because we just built the engine to unify all engines". :)

What would you expect from a display to be good enough?

Just as a note, Orb is not there to compete with high volume graph visualizations like Cosmograph, Graphistry, Linkurious. It is more as a child from d3 and vis.js, which are great libraries, that uses d3 simulation and vis-like canvas rendering. We really liked what vis.js team did with the styling of the graph and how you can customize it - this is often a limitation for high volume graph visualizations.

We could also discuss about the analytics usability of seeing a graph with 1 billion nodes. It is definitely awesome, but it is too much data to grasp on as a user seeing it. Clustering or other graph algorithms would help. I think the question is: What is the maximum graph size (number of nodes/edges) when it becomes hard to get any useful visual information expect the graph global state? (e.g. seeing a bar chart with 365 columns (days) is harder to read than a bar chart with a smaller sampling, e.g. per week or month).

I don't know the answer to this, but maybe you will have due to your experience with graph visualizations.