News for hackers. Stuff that matters.
Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @05:57AM (#33704381)
Hi HN, I'm Alex, at Terrastruct, where we've been making D2. This actually popped up on HN a couple months back, though it wasn't ready, e.g. not open source yet. It is now!

We also put up a site for you to compare D2 with MermaidJS, Graphviz, and PlantUML: https://text-to-diagram.com.

Full disclosure, we're a for-profit company. The open-core part is that we make an alternative layout engine which we sell (Jetbrains model, i.e. your copy is your's forever if you've paid for 12+ months). It's not packaged with D2, so you won't see it if you don't want it. D2 is perfectly usable without it, and integrates with multiple free open source layout engines (e.g. the one that Mermaid uses, "dagre", is D2's default). If you want to read more about our plans for D2: https://d2lang.com/tour/future.

Hope you can check it out! It's got an easy install (and uninstall) process.

Re:D2 is now open source – a new, modern language that turns... by wg0 on Tuesday November 22, 2022 @06:06AM (#33704456)
Beautiful! And then in golang. Thank you for open sourcing the core part.
Re:D2 is now open source – a new, modern language that turns... by mooneater on Tuesday November 22, 2022 @07:25AM (#33705007)
20+ year fan of graphviz here, love seeing what you are doing with D2!

Any examples of larger diagrams (1000+s of nodes) and how does that perform in layout?

Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @07:35AM (#33705073)
thank you! There isn't right now, we'd need to add a simpler layout algorithm for those, e.g. force-directed, radial, etc. That use case is not something D2 is trying to tackle though, we're just focused on software architecture diagrams right now, and I've yet to see one span that many nodes
Re:D2 is now open source – a new, modern language that turns... by nonrandomstring on Tuesday November 22, 2022 @08:30AM (#33705536)
Picking a set of domain specific targets and honing a tool to service that seems a good way to start building - rather than a general solver.

I also love graphviz but had a project to create an app for legible quality management, and it was a headache to shoehorn some concepts into "nodes in an edge connected graph" way of thinking (ended up programatically scripting LaTeX).

With so many visualisation concepts there's room for niche approaches (same in plotting; Matlab/Octave vs Gnuplot vs Pyplot vs R and so on)

Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @08:39AM (#33705616)
agreed! that reminds me, there's a company that specializes in this niche of visualizing large data: https://www.graphistry.com/ (haven't used it myself, looks well-made).
Re:D2 is now open source – a new, modern language that turns... by lmeyerov on Tuesday November 22, 2022 @07:44PM (#33713617)
100% agreement, spot on observations! Indeed, we end up referring folks to cool diagramming tools all the time at Graphistry as d2/mermaid/diagrams.net are well-optimized for quick & beautiful diagram presentation tasks on manageably sized & fairly static datasets. Using Graphistry to do a quick markdown of how a 4 node cluster might work is a bit like driving a tank to pick up some milk :) Teams go to us more for gnarly investigation & splunking tasks that need a visual power tool, like looking at alert logs or big systems, so we optimize for making that scale interactive & easy.

The overlap is real however, so a lot of room for teams to learn. Ex: We're the easiest graph tool for jupyter/databricks/streamlit etc teams who use dataframes, and I can imagine those tools learning from us here. In the reverse direction, our work more in terms of quickly configurable global data<>viz data bindings ("using the UI or API, bind each event's score to a hot-and-cold coloring palette and use a warning icon on all type=alert events"), but we have a ways to go to support the more manual artisinal effects of diagramming tools like Figma, where each element might have a super fancy & unique border style.

Re:D2 is now open source – a new, modern language that turns... by deathanatos on Tuesday November 22, 2022 @12:06PM (#33708034)
IDK if it's quite the use case you're looking for, but at my company, we turn our CI YAML for our monorepo into Dot (Graphviz), and graph it, to visualize the dependencies between steps in our CI, & to highlight the critical path.

It's an ~100 node DAG. Graphviz struggles a bit with it, particularly with edge layout.

(Though really, the layout I want for it is a Gnatt chart style layout.)

Re:D2 is now open source – a new, modern language that turns... by philsnow on Tuesday November 22, 2022 @12:54PM (#33708658)
I have made software architecture diagrams, but more "descriptive" than "prescriptive" ones, by generating graphviz from source code analysis tooling. (The objective here was getting oriented in a large codebase where there wasn't really a rigorous diagram of anything written down anywhere, and trying to find sort of a min-cut where putting a line through the graph of imports would cross the fewest edges)

In my experience graphviz handles up to thousands of nodes and tens of thousands of edges without really breaking a sweat.

edit: I've done the same thing with extracting AWS diagrams from aws describe* calls, with good results.

Re:D2 is now open source – a new, modern language that turns... by riedel on Tuesday November 22, 2022 @12:54PM (#33708653)
I am always quite astonished how bad the default layouters for graphs perform. When I was still doing compiler optimization in the beginning of the 2000s we did not struggle with quite big graphs thanks to cool graph visualizers such as vcg [1] . Two weeks ago I was tempted to try it again after nearly 20 years because I was frustrated even visualising a relatively small graph in python (cytoscape seemed to be the only working software in the end but it was quite a pain to get it to just render what I wanted)

[1] https://www.rw.cdl.uni-saarland.de/people/sander/private/htm...

Re:D2 is now open source – a new, modern language that turns... by ygra on Wednesday November 23, 2022 @04:15AM (#33716880)
Laying out larger graphs is tricky, often because the size simply stands in the way of generating anything that's useful to the viewer. To add to that, most layout algorithms prioritize optimizing certain criteria, while other parts of the visualization emerge from that; and if the human viewer chooses a layout algorithm because of one of the latter properties, they are often surprised that the result doesn't look like they envisioned (because, hey, the algorithm optimized something entirely different). We see this disconnect fairly often in our own customer support, but I haven't really found a good way of putting an explanation in writing.

Then there's generally the problem of larger graphs, which tend to devolve into a tangled mess and hairballs, simply because they often tend to be well-connected. If there's no way of pruning them beforehand, or perhaps grouping, aggregating or clustering (in a way that makes sense to the viewer, not necessarily only structurally), then it can be hard to get good result.

Re:D2 is now open source – a new, modern language that turns... by JonChesterfield on Tuesday November 22, 2022 @07:48AM (#33705177)
Is the language described anywhere? Couldn't find an example other than the one on text-to-diagram and can't find anything beyond that.

edit: guessing URL found https://d2lang.com/tour/intro/ which covers it, perhaps worth linking to there from the GitHub page

edit2: ok it is linked, just above the section I was looking at:

Reference

A more formal reference specification is in the works.

Re:D2 is now open source – a new, modern language that turns... by localhost on Tuesday November 22, 2022 @10:33AM (#33706852)
This is a great idea; my go-to approach is the "I'll use PowerPoint to do that and take a screenshot" mindset since I don't have a need for publication quality output.

I really like the comparison site that you have created - it's great for learning these languages. I would love to see a plugin for Obsidian for the times where I need to add a custom drawing to my notes. It looks like there already are plugins for PlantUML[1] and graphviz[2] today.

[1] https://github.com/QAMichaelPeng/obsidian-graphviz

[2] https://github.com/joethei/obsidian-plantuml

Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @10:39AM (#33706911)
thanks! i don't use Obsidian personally and don't know how the plugins work, but if anyone sees this and wants to do a plugin for it, please reach out or open a Github issue and we'll support whatever API changes are needed (if any) to D2's plugin system to make integration easy.
Re:D2 is now open source – a new, modern language that turns... by bitcurious on Tuesday November 22, 2022 @11:04AM (#33707203)
Is the name inspired by d3.js?
Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @01:24PM (#33709056)
no
Re:D2 is now open source – a new, modern language that turns... by jmyeet on Tuesday November 22, 2022 @11:08AM (#33707269)
This looks really interesting. It made me curious about something.

As an example your tree is laid or left-to-right. The GraphViz one in comparison is top down. It’s not clear to me as a user if that’s forced or a choice (in either case).

Are you able to clarify that sort of difference by either making the diagrams similarly laid out or adding a note if something isn’t possible?

Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @01:24PM (#33709052)
Good observation! Those are just defaults, we don't change the defaults for the comparisons to match (e.g. font sizes, shape dimensions, etc).
Re:D2 is now open source – a new, modern language that turns... by ritzaco on Tuesday November 22, 2022 @11:19AM (#33707420)
wow thank you for the transparency - you literally answered all the questions up front that I care about re open core, for profit, killswitches, etc etc that usually take hours to find from a marketing landing page.
Re:D2 is now open source – a new, modern language that turns... by billconan on Tuesday November 22, 2022 @11:50AM (#33707821)
I really want to learn the algorithms behind automatic layout engines. Are there any books on this topic?
Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @12:05PM (#33708016)
not that i know of, but there's a lot of research papers, under the topic of "graph visualization". we read a ton of these and combined and modified ideas for TALA.

The folks at ELK are doing fantastic work in public as an academic team out of Germany.

https://www.eclipse.org/elk/documentation/algorithmdeveloper... https://www.rtsys.informatik.uni-kiel.de/en/team/soren-domro...

Graphviz's papers are also must-reads (for hierarchical layouts): https://graphviz.org/theory/

In general they are surprisingly readable even without an academic background. Sometimes they go into proofs of correctness like method X really results in minimal edge crossings, and those are gnarly math, but skippable. The bad part is that a lot of these research papers have code that's just broken or wrong, or is NP-hard.

Re:D2 is now open source – a new, modern language that turns... by ejuliobatista2 on Wednesday November 23, 2022 @01:55PM (#33723152)
There is this open book https://cs.brown.edu/people/rtamassi/gdhandbook/ We've drawn some inspiration from there and followed to some references too
Re:D2 is now open source – a new, modern language that turns... by escot on Tuesday November 22, 2022 @08:03PM (#33713744)
dagrejs has a section on "Recommended Reading" which lists which algorithms from which papers they implement.

https://github.com/dagrejs/dagre/wiki#recommended-reading

Re:D2 is now open source – a new, modern language that turns... by trey-jones on Tuesday November 22, 2022 @11:59AM (#33707927)
I'm excited to see this. It's the first repository I've starred in a long time. I'm asked for these types of diagrams from time to time and I've bounced between many different tools, none of which I've found to be satisfactory. I will try this next. Thank you for your work on it.
Re:D2 is now open source – a new, modern language that turns... by laomai on Tuesday November 22, 2022 @02:06PM (#33709740)
Looks awesome! Very easy to read!

Can't wait for an Obsidian plugin! :D

Re:D2 is now open source – a new, modern language that turns... by linuxdude314 on Tuesday November 22, 2022 @02:16PM (#33709926)
Do you maintain a dagr fork internally?

The project looks to be deprecated: https://github.com/dagrejs/dagre

Re:D2 is now open source – a new, modern language that turns... by rahimnathwani on Tuesday November 22, 2022 @02:25PM (#33710056)
Mermaid itself uses this fork (at least its npm link appears in one of the package.json files), which is still active;

https://github.com/tbo47/dagre-es

Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @02:21PM (#33709996)
we maintain a wrapper around dagre, which adds functionality like supporting container to container connection. dagre is unmaintained, but it's also "complete", in that it implements the theoretical papers it set out to.
Re:D2 is now open source – a new, modern language that turns... by tunesmith on Tuesday November 22, 2022 @06:19PM (#33712937)
I'm wondering why you don't use elkjs? I recently moved my own side project from dagre to elkjs for exactly this reason.
Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @06:26PM (#33713010)
elkjs is an alternative! https://github.com/terrastruct/d2#plugins. we haven't gotten the defaults to look as good though
Re:D2 is now open source – a new, modern language that turns... by tunesmith on Tuesday November 22, 2022 @06:43PM (#33713154)
ah! I understand, I've spent a little too much time playing with way too many combinations of elk's configs.
Re:D2 is now open source – a new, modern language that turns... by alixanderwang on Tuesday November 22, 2022 @06:54PM (#33713220)
if you'd like to help, we're trying to figure some of these configs here in case something stands out to you! https://github.com/terrastruct/d2/pull/95
Re:D2 is now open source – a new, modern language that turns... by Abishek_Muthian on Wednesday November 23, 2022 @12:42PM (#33722218)
Congratulations on the launch!

The comparison website is neat. I don't like to draw the architectural diagrams which graphical tools for my Open-source projects, I'll try D2 for my next project.

Re:D2 is now open source – a new, modern language that turns... by t0k0l0sh on Thursday December 01, 2022 @01:53PM (#33820801)
Can you include https://diagrams.mingrammer.com/ in the comparisons? :-D