Yes, the same applies to many niche syntaxes: Influx' flux language (I was able to Design my dream Grafana dashboards now!) or Auto Hot Key (AHK) for Windows automation are only two examples.

I love using LLMs to generate plantUML diagrams

What other tool do you use? I guess you want an svg or png output at some point?

If you mean "I have a .puml and need .svg or .png", the tool does that natively, as do most online versions (this one is their example https://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4... and there are PNG and SVG links under the textarea). It also works offline https://plantuml.com/command-line#:~:text=Types%20of%20Outpu...

I didn’t know plantuml.com existed. Thanks!

I had an absolutely terrible time today trying to get chatgpt to write me a very simple awk oneliner. It “thought” I was specifying a much more complicated requirement than I actually was.

"ChatGPT" can mean many things... If you meant the free 4o-mini model, then yes, this outcome is not surprising, since it is bad at basically anything related to code.

If you meant the more powerful o1 or o3-mini models (great naming, openai...), then that outcome would be surprising.

O1 Reasoning. You know, the feature you used to have to pay $200/mo for.

o1 never cost $200/mo. That was o1-pro, which still isn’t available to Plus or Free users.

whatever it is it’s the latest model and the hype is just not real.

Over and over again I have witnessed people just drive themselves in circles downstream of a refusal to just step out of the “make the LLM fix the issue for me” loop.

At one point the syntax and the specifics matter! * and + have different meanings in the regex. Overly-specified LLM output is worth trimming down to what your problem actually needed.

I appreciate LLM output able to draft out sketches of results (and yes, a lot of the time, getting exactly the right result). And it’s great as a learning tool (especially if you’re diligent in the trust + verify department). But I worry that people are not taking opportunities to sit down and actually use the output as the sketch, and to insert the sort of precision that comes from the “infinite context” of the human working on the problem. Devs can’t just decide to opt out of getting into the details IMO

> Over and over again I have witnessed people just drive themselves in circles downstream of a refusal to just step out of the “make the LLM fix the issue for me” loop.

You're reminding me of a 1997 talk by an American Airlines trainer about automation dependency and when dropping down a level of automation reduces workload: <https://www.youtube.com/watch?v=WITLR_qSPXk>

cool talk, respect him making the point in 20 minutes. There's a lot of levels of applying this when programming, not just with LLM usage.

> Over and over again I have witnessed people just drive themselves in circles downstream of a refusal to just step out of the “make the LLM fix the issue for me” loop.

I think that's an amazing encapsulation of my experience with LLMs.

They can be great tools, but you need to extremely willing to throw away their output and start over instead of trying to incrementally fix what they give you.

Exactly the same for me: the major breakout success was having ChatGPT teach me how to use pandas. It really shone as an interactive manual with worked examples.

Taking the training wheels off isn’t something I’ve really nailed though: for example, I keep coming back with the same questions about how to melt and pivot. I can self diagnose as this showing I didn’t really spend enough time understanding the answers the first time around.

To be fair, that also could be because the pandas API is…not great. When I used to write lots of python and pandas I basically always had the pandas docs open intuitive and memorable it is not.

I kind of agree. I’ve had very mixed experiences with LLMs and DSLs.

I was writing an NRQL query (New Relic’s log query language) and wanted to essentially do a GROUP BY date_trunc. It kept giving me options that I was eager for, and then the functions it gave me just didn’t exist. After like four back and forths of me telling it that the functions it was giving me didn’t exist - it worked.

Then I needed it to split on the second forward slash of a string and just give me the first piece. It gave me the foundation to fill in the gaps of the function, but the LLM never got it.

In that case, I assume it’s a lack of training data since NRQL is pretty niche.

I catch myself swinging from “holy shit this is impressive” to “wow this sucks” and back regularly for code.

This is similar to my experience with LLMs and DSLs. They tend to hallucinate functions that will magically work in the situation you are describing. My pet theory here is that they are fooled by many forum posts/issues "why doesn't a function exist in this DSLs called ABC that does this?"

Did you try asking the LLM to writ the missing functions? It might work in some cases.

In that case putting NRQL reference into context would work wonders

Oh goodness I’m a fool to not have tried that. I’ll absolutely give that a go next time!

NRQL's abandonment of GROUP BY for FACET always throws me.

LLM context windows are quite large now. This is very likely a simple matter of including NRQL docs or specification in the prompt.