Re:Building Boba AI: Lessons learnt in building an LLM-power...
by akiselev on Thursday June 29, 2023 @01:54PM
(#36524113)
> Along the way, we’ve learned some useful lessons on how to build these kinds of applications, which we’ve formulated in terms of patterns.
[ Parent
| on HN ]
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by frankgrecojr on Thursday June 29, 2023 @02:41PM
(#36524839)
> Stream the response to the UI so users can monitor progress
This is a game changer to the UX
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by senko on Thursday June 29, 2023 @03:40PM
(#36525683)
It's a crutch to minimize the user annoyance at having to wait up to a minute for the response. It sure beats the spinner but it's still a crutch.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by thelittleone on Friday June 30, 2023 @03:07AM
(#36531517)
Really? I feel like in most uses the stream is close to, if not slightly faster than my reading. I actually prefer that over an instant full-page response. It helps me keep my place in the text and feels like reduced cognitive load.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by golergka on Friday June 30, 2023 @04:43AM
(#36532128)
Also, the vibrations when LLM “types” in OpenAI iOS app are so satisfying for some reason. It may be a stupid subjective thing, but from my experience in game dev, that's exactly the kind of detail that creates the overall user experience feeling.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by behnamoh on Thursday June 29, 2023 @03:53PM
(#36525819)
Actually, it's annoying because as you start reading the first lines, the content keeps scrolling (often with jagged movements). I always have to scroll up immediately after the stream begins to disable this behavior.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by tobr on Thursday June 29, 2023 @04:14PM
(#36526074)
That’s totally fixable, though. ReadyRunner handles it simply by scrolling all the way from the start, leaving space for the message to grow.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by trafnar on Thursday June 29, 2023 @06:15PM
(#36527565)
Hey, that's my app! https://www.readyrunner.ai
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by darkteflon on Thursday June 29, 2023 @10:05PM
(#36529672)
That looks really nice. I do wish you would consider one-time pricing for those bringing their own API key on the “Dev” plan, though :) I’d pay ~$20-30 for a nice desktop app like this but won’t enter into another $48/year sub.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by huydotnet on Thursday June 29, 2023 @03:53PM
(#36525822)
Still not a reasonable way if you're expecting a structured data in the response, like JSON or something that you're required to parse before showing to the user.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by jelled on Friday June 30, 2023 @01:26AM
(#36530839)
On one of my apps I ask for a plain text response first and then make a second call to parse the original response to json.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by jamifsud on Thursday June 29, 2023 @06:45PM
(#36527893)
Anyone know of any good “tolerant” JSON parsers? I’d love to be able to stream a JSON response down to the client and have it be able to parse the JSON as it goes and handle the formatting errors that we sometimes see.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by icyfox on Thursday June 29, 2023 @07:22PM
(#36528193)
There's no bulletproof solution to this. JSON5 (https://www.npmjs.com/package/json5) gets you slightly more leniency, as does plugging the currently streamed content into another smaller LLM. I also wrote a deterministic parser more tailored towards these partially-complete LM outputs. Not perfect certainly but handles the 99% of cases well: https://github.com/piercefreeman/gpt-json. In particular the "streaming" functionality here might be of interest to you.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by darkteflon on Thursday June 29, 2023 @09:07PM
(#36529223)
This looks really cool, thanks for open-sourcing this. I’ve been similarly parsing and validating output from OpenAI’s new functions using a schema defined on a custom Pydantic class, but I can see that your code has a lot of niceties coming from proper battle-testing, including elegant error handling, transformations and good docs.
I’d like to incorporate this in a production workflow for generating schema-compliant test data for use in few-shot promoting - would you mind saying a few words about your medium term plans for the library? The LangChain API is changing all the time at the moment so we’re trying to figure out where it’s safe to stand. No expectations, of course, just curious.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by icyfox on Thursday June 29, 2023 @09:20PM
(#36529346)
Sure - I'm using it in a few different internal tools and know others are using it in production. The API should be relatively stable at this point since I intentionally kept the scope pretty limited. The main changes over time will be improved robustness and error correction as issues report different JSON schema breaks that we can fix automatically. Let me know if you see more cases that can be addressed here, would love to collaborate on it.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by darkteflon on Thursday June 29, 2023 @10:29PM
(#36529820)
Thanks! Absolutely, will do. I’ll have a play with it today and reach out with a PR any time it makes sense to do so.
I noticed some occasional funkiness from GPT-4 around sending back properly formatted dates yesterday but haven’t yet dug into it properly. Might be a good candidate for a transformation.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by n4te on Thursday June 29, 2023 @11:26PM
(#36530177)
My JsonReader in libgdx and JsonBeans can parse a more relaxed version of JSON. It uses Ragel.
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by anentropic on Friday June 30, 2023 @04:51AM
(#36532173)
> I’d love to be able to stream a JSON response down to the client and have it be able to parse the JSON as it goes
why though?
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by jamifsud on Friday June 30, 2023 @06:51PM
(#36543601)
In a non chat setting where the LLM is performing some reasoning or data extraction it allows you to get JSON directly from the model and stream it to the UI (updating the associated UI fields as new keys come in) while caching the response server side in the exact same JSON format. It’s really simplified our stream + cache setup!
Re:Building Boba AI: Lessons learnt in building an LLM-power...
by manojlds on Thursday June 29, 2023 @06:45PM
(#36527892)
The short courses from dl.ai are better at driving these points - https://www.deeplearning.ai/short-courses/