News for hackers. Stuff that matters.
Re:Building Boba AI: Lessons learnt in building an LLM-power... by anentropic on Friday June 30, 2023 @04:51AM (#36532173)
> I’d love to be able to stream a JSON response down to the client and have it be able to parse the JSON as it goes

why though?

Re:Building Boba AI: Lessons learnt in building an LLM-power... by jamifsud on Friday June 30, 2023 @06:51PM (#36543601)
In a non chat setting where the LLM is performing some reasoning or data extraction it allows you to get JSON directly from the model and stream it to the UI (updating the associated UI fields as new keys come in) while caching the response server side in the exact same JSON format. It’s really simplified our stream + cache setup!