News for hackers. Stuff that matters.

slundberg

Re:A guidance language for controlling LLMs on Tuesday May 16, 2023 @02:08PM (#35965532)
You can serialize and ship the DSL to a remote server for high speed execution. (without trusting raw Python code)
Re:A guidance language for controlling LLMs on Tuesday May 16, 2023 @01:47PM (#35965288)
As others mentioned, this was initially developed before LangChain became widely used. Since it is lower level, you can leverage other tools, like any vector store interface you like such as in LangChain. Writing complex chain of thought structure is much more concise in guidance I think since it tries to keep you as close to the real strings going into the model as possible.
Re:A guidance language for controlling LLMs on Tuesday May 16, 2023 @01:41PM (#35965206)
If you want guidance acceleration speedups (and token healing) then you have to use an open model locally right now, though we are working on setting up a remote server solution as well. I expect APIs will adopt some support for more control over time, but right now commercial endpoints like OpenAI are supported through multiple calls.

We manage the KV-cache in session based way that allows the LLM to just take one forward pass through the whole program (only generating the tokens it needs to)