Re:What We Learned from a Year of Building with LLMs
by 7d7n on Wednesday May 29, 2024 @02:23AM
(#40509067)
100% agree with Ted's take. One of the authors wrote about splitting up prompts here too: https://eugeneyan.com/writing/prompting/#split-catch-all-pro...
Re:What We Learned from a Year of Building with LLMs
by hubraumhugo on Wednesday May 29, 2024 @02:14AM
(#40509006)
Agree, that's a very good summary. Would love to see some benchmarks for the two approaches.
Re:What We Learned from a Year of Building with LLMs
by umangrathi on Wednesday May 29, 2024 @02:08AM
(#40508969)
smaller tasks also helps in choosing smaller models to work with, instead of waiting for a large model to respond (really not usable when doing customer facing work)
If each step of your task requires knowledge of the big picture, then yeah it ought to help to put all your context into a single API call.
But if you can decompose your task into relatively independent subtasks, then it helps to use a custom prompt/custom model for each of those steps. Extraneous context and complexity are just opportunities for the model to make mistakes, and the more you can strip those out, the better. 3 steps with 99% reliability are better than 1 step with 90% reliability.
Of course, it all depends on what you're trying to do.
I'd say single, big API calls are better when:
- Much of the information/substeps are interrelated
- You want immediate output for a user-facing app, without having to wait for intermediate steps
Multiple, sequenced API calls are better when:
- You can decompose the task into smaller steps, each of which do not require full context
- There's a tree or graph of steps, and you want to prune irrelevant branches as you proceed from the root
- You want to have some 100% reliabile logic live outside of the LLM in parsing/routing code
- You want to customize the prompts based on results from previous steps