We are talking about calling an API here people. Maybe what is behind the API is magical and powerful seeming but it's just an API that takes some context, the tokens to generate and a temperature setting.
Doing everything in that diagram is probably overkill for most uses. But using it as a starting point and trimming what you don't need will help a bit with triping over "oops I forgot to include that".
But caching, for instance, doesn't need to be it's own lib does it? I don't want 'semantic caching' I want to cache the exact same query and I can do that without being LLM specific:
from joblib import Memory
@memory.cache
def call_chat_completion_api_cached(max_tokens, messages,temperature):
...
But it also does feel like an land grab of supporting libs and infrastructure.
A browser database and React is all I have needed for my LLM apps.
But there's usually a lot of reasons why these architectures could be a useful reference. In my project, I host my own trained LLM and one of the cost efficiencies comes from being able to cache at every step along the way. Then there is a large private media hosting consideration.
There is room for all sorts of setups and I kind of liked how the article mapped out some of the common paths.
> Fine-tuned OpenAI models. The Fine-tunes API allows customers to create their own fine-tuned version of the OpenAI models based on the training data that they have uploaded to the service via the Files APIs. The trained fine-tuned models are stored in Azure Storage in the same region, encrypted at rest and logically isolated with their Azure subscription and API credentials. Fine-tuned models can be deleted by the user by calling the DELETE API operation.
My follow-up question is: how is this different from OpenAI's finetuning?
i built my own last weekend. https://github.com/smol-ai/logger
dumps things to json files, or to a log store. all you need for prompt engineering and monitoring really! no VC needed, no DataDog of AI yet
.
> quickly build a robust feature with only
Much like how building Twitter is a weekend-sized project.
More like a month, but yes. That's what we did - a month to launch, then another month to harden and add several features. Rolled out to all customers.
Building product features with LLMs is difficult, but not because of the architectural needs. It's an API you pass data to.
Just here to say that you can quickly build a robust feature with only OpenAI's APIs, redis, a text file for the prompt you parameterize (versioned), and a little bit of glue code (no LangChain). You can add instrumentation for observability around that like you would any other code.
I would wager that most enterprise use cases don't need most of the tools listed in this article, and using them is complete overkill.