sandkoan
Re:Why Bell Labs Worked
on Sunday May 11, 2025 @07:57PM (#43958310)
DeepMind.
Re:Show HN: Sonauto – a more controllable AI music creator
on Wednesday April 10, 2024 @09:35PM (#39997571)
Re:world_sim: LLM prompted to act as a sentient CLI universe simulator
on Friday April 05, 2024 @09:48PM (#39949312)
sudo !!
Re:Extreme Video Compression
on Tuesday February 20, 2024 @12:53AM (#39438429)
Ilya says this here: https://www.youtube.com/watch?v=AKMuA_TVz3A
Re:Transformers struggle with generalizing tasks beyond pre-training data
on Tuesday November 07, 2023 @02:59PM (#38181983)
For anyone else interested, the paper he's referring to is "The Reversal Curse": https://arxiv.org/abs/2309.12288.
Re:MemGPT – LLMs with self-editing memory for unbounded context
on Monday October 16, 2023 @08:40PM (#37908727)
[delayed]
Re:Show HN: Fine-tune your own Llama 2 to replace GPT-3.5/4
on Tuesday September 12, 2023 @06:40PM (#37489588)
[delayed]
Re:Show HN: LLMs can generate valid JSON 100% of the time
on Monday August 14, 2023 @08:38PM (#37128672)
This is what we did at Trex (https://github.com/automorphic-ai/trex). The tricky part is doing it quickly and efficiently.
Re:Llama: Add Grammar-Based Sampling
on Friday July 21, 2023 @07:42PM (#36821529)
Also using a similar method: https://github.com/automorphic-ai/trex
Playground: https://automorphic.ai/playground
Re:Introducing TypeChat from Microsoft
on Thursday July 20, 2023 @06:19PM (#36807525)
Relevant: Built this which generalizes to arbitrary regex patterns / context free grammars with 100% adherence and is model-agnostic — https://news.ycombinator.com/item?id=36750083
Re:Structured Output from LLMs (Without Reprompting!)
on Tuesday July 18, 2023 @04:23AM (#36769426)
Yeah, then it seems we agree. I was just pointing out that it's not necessary to finetune OSS models to behave like OpenAI functions if you're able to do something similar to what we did (no tuning involved!).
Re:Structured Output from LLMs (Without Reprompting!)
on Monday July 17, 2023 @09:56PM (#36766808)
Ahh, no, the value of this isn't as much the model as it is the infrastructure enabling structure enforcement.
Can you beat our firewall?
Re:Structured Output from LLMs (Without Reprompting!)
on Monday July 17, 2023 @04:01PM (#36763259)
Re:Structured Output from LLMs (Without Reprompting!)
on Monday July 17, 2023 @03:16AM (#36755083)
You wouldn't actually want to, because you'd be losing generalizability, and it's a lot of unnecessary work.
I think approach #1 outlined above is the better (more cost- and time-efficient) technique—where a pretrained model already understands JSON (among myriad other formats), and you merely constrain it at text-gen time to valid JSON (or other format).
Re:Structured Output from LLMs (Without Reprompting!)
on Monday July 17, 2023 @12:57AM (#36754333)
Ahh, I've been meaning to try FLARE—was it a marked improvement over traditional RAG?
Re:Structured Output from LLMs (Without Reprompting!)
on Monday July 17, 2023 @12:45AM (#36754259)
Thanks for the reminder—done!
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @11:10PM (#36753634)
This is model agnostic, actually—any model on HuggingFace is compatible. So if someone wanted to run this with their own model, they could.
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @10:54PM (#36753547)
Custom LLM—hence the self-hostability.
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @10:07PM (#36753292)
Costs add up surprisingly quickly. A quote-colon-space-quote combo alone is four tokens wasted. Now scale that up....
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @10:04PM (#36753284)
https://news.ycombinator.com/item?id=36753254
Does this help clarify?
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @09:58PM (#36753254)
The prompt is given to our model as a guiding aid (a suggestion), and the cfg is used to constrain the model to generate only tokens that abide by the schema (an enforcement). That's how we ensure only valid outputs at text generation time.
We also prefill some tokens depending on the set of allowed tokens at a given state, so the model doesn't waste resources trying to predict them.
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @09:24PM (#36753073)
We have folks playing around with it mostly through the playground / raw HTTP endpoints as opposed to the Python API. And we've got some batch jobs running, which adds further traffic.
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @09:07PM (#36752991)
Problems with OpenAI:
1) You're wasting GPT tokens on outputting JSON instead of meaningful information.
2) GPT functions won't, with absolute, 100% certainty, return JSON in the schema you want. In 1% to 3% of cases it hallucinates fields, etc.
3) This also allows you to output data in arbitrary non-JSON formats.
4) You can't self-host OpenAI functions.
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @07:24PM (#36752420)
We enable conforming to arbitrary context free grammars in addition to regex patterns, and have a bunch of speed optimizations, as well.
Though it may not seem too fast right now on account of the hundreds of simultaneous requests we're getting :)
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @07:07PM (#36752333)
Ahh, that—due to compute limitations we're forced to run a very small model that isn't as capable of converting 5'8" to inches. The larger model is, though.
Anything Can Be a Weekend Hack
Structured Output from LLMs (Without Reprompting!)
Re:Structured Output from LLMs (Without Reprompting!)
on Sunday July 16, 2023 @02:30PM (#36750084)
Built a tool for transforming unstructured data into structured outputs using language models (with 100% adherence).
If you're facing problems getting GPT to adhere to a schema (JSON, XML, etc.) or regex, need to bulk process some unstructured data, or generate synthetic data, check it out.
We run our own tuned model (you can self-host if you want), so, we're able to have incredibly fine grained control over text generation.
Repository: https://github.com/automorphic-ai/trex
Playground: https://automorphic.ai/playground