News for hackers. Stuff that matters.
Re:A guidance language for controlling LLMs by felideon on Tuesday May 16, 2023 @03:39PM (#35966705)
A number of years ago we were designing a way to specify insurance claim adjudication rules in natural language, so that "the business" could write their own rules. The "natural" language we ended up with was not so natural after all. We would have had to teach users this specific English dialect and grammar.

So, in the end, we abandoned that project and years later just rewrote the system so we could write claim rules in EDN format (from the Clojure world) to make our own lives easier.

In theory, the business users could also learn how to write in this EDN format, but it wasn't something the stakeholders outside of engineering even wanted. On the one hand, their expertise was in insurance claims---they didn't want to write code. More importantly, they felt they would be held accountable for any mistakes in the rules that could well result in thousands and thousands of dollars in overpayments.

Re:A guidance language for controlling LLMs by tomduncalf on Tuesday May 16, 2023 @03:57PM (#35966939)
> but it wasn't something the stakeholders outside of engineering even wanted

Ha this reminds me of the craze for BDD/Cucumber type testing. Don’t think I ever once saw a product owner take interest in a human readable test case haha

Re:A guidance language for controlling LLMs by jaggederest on Tuesday May 16, 2023 @05:17PM (#35967870)
I've used Cucumber on a few consulting projects I've done and had management / C-level interested and involved. It's a pretty narrow niche, but they were definitely enthusiastic for the idea that we had a defined list of features that we could print out (!!) as green or red for the current release.

They had some previous negative experiences with uncertainty about what "was working" in releases, and a pretty slapdash process before I came on board, so it was an important trust building tool.

Re:A guidance language for controlling LLMs by jamiek88 on Tuesday May 16, 2023 @06:16PM (#35968503)
> important trust building tool

This is so often completely missed in these conversations about these tools.

Great point.

Re:A guidance language for controlling LLMs by btown on Tuesday May 16, 2023 @06:52PM (#35968829)
“Incentivize developers to write externally understandable release notes” is an underrated feature of behavioral testing frameworks!
Re:A guidance language for controlling LLMs by hitchstory on Tuesday May 16, 2023 @07:40PM (#35969300)
Ive had product owners take an interest in docs autogenerated from tests. Stuff like this:

https://github.com/hitchdev/hitchstory/blob/master/examples/...

Gherkin isnt really a suitable language for writing test cases in though - it's verbose, lacks inheritance, has clunky syntax, is stringly typed and the parser demands you write regexes to parse it.

Re:A guidance language for controlling LLMs by TaylorAlexander on Tuesday May 16, 2023 @03:59PM (#35966960)
Just saw this on HN a couple days ago, sounds like just what was needed!

https://en.wikipedia.org/wiki/Attempto_Controlled_English?wp...

https://news.ycombinator.com/item?id=35936396

Re:A guidance language for controlling LLMs by Sharlin on Tuesday May 16, 2023 @04:30PM (#35967354)
SQL looks the way it does (rather than some much more succinct relational algebra notation) because it was intended to be used by non-technical management/executive personnel so they could create whatever reports they needed without somebody having to translate business-ese to relalg. That, uh, didn't quite happen.
Re:A guidance language for controlling LLMs by Swizec on Tuesday May 16, 2023 @05:14PM (#35967831)
On the other hand, many of the product manager's I've worked with are better at SQL than many of the senior fullstack software engineer candidates I've interviewed. It's a strange world out there.
Re:A guidance language for controlling LLMs by jimsparkman on Wednesday May 17, 2023 @01:49AM (#35971693)
I think this is the exception, not the norm. My experience is business users (incl. PMs) are lost outside of Excel.
Re:A guidance language for controlling LLMs by Swizec on Wednesday May 17, 2023 @10:20AM (#35975553)
In my biased sample of SFBA tech companies it’s pretty common for PMs to know at least enough SQL to be dangerous. In early stage startups there’s no analyst to lean on, so they need SQL. In late stage companies there are data lakes and analytics databases specifically designed to be easily queryable, so SQL offers the best flexibility.

But you’re right, many of them are wizards in <whatever query language our tool uses>. Like VizQL for Tableau.

Re:A guidance language for controlling LLMs by Too on Thursday May 18, 2023 @02:15AM (#35984607)
Having a personal need is often the best motivator for learning. Compared to closing ticket no 471961. A PM knows the shape of the data and what he is looking for, the sql is just a way to get there, rather than the other way around, when the goal often get lost in translation.

On the other hand, sql is also dangerous in that it gives you a result, even if the aggregation is completely wrong! Forgetting empty groups in a group by, loosing rows in a join or ignoring null values is common to see. Many cases of non techies “knowing” sql often turn out to be some basic query that give seemingly good data out but actually is complete nonsense when put under deeper review, a bit like AI hallucinations.

Re:A guidance language for controlling LLMs by dolmen on Wednesday May 17, 2023 @09:34AM (#35975027)