roh26it

What do you use for the onboarding guide?

Well written, but does this all change in a claude-ified world?

Looks solid, going to try it out.

I'm going to be __that guy__, but just ask - is the functionality set similar to llamaparse or is this llamaparse + llm?

So, while time to first token is lower, throughput might also be lower in most cases?

What are the trade-offs you've made to achieve this?

The most interesting findings (at least from my perspective) - OpenAI measured "honesty" that could become an interesting model eval others should adopt. - Using instruction heirarchy to _avoid_ prompt injections - When pushed to pursue misaligned goals, it tried to disable oversight in 5% of tests!

I've been thinking about AI guardrails in production for almost 7 months now, and I'm convinced that using them to "control", "secure", or "block" LLM outputs is not the best use for them .

Using them to "guide", "accelerate", and "empower" your AI development process? Now we're talking.

Let me explain why I believe this shift in perspective could change how we build AI...

Most people see AI guardrails as training wheels. But what if we saw them as a nitrous boost for our AI hot rods instead?

When you're not worried about your AI going off the rails, you can push it to its limits. You can experiment with wilder ideas, test bolder hypotheses, and innovate at lightning speed.

Want to train on spicy data? Guardrails have your back. Thinking of an aggressive model architecture? Guardrails keep you in bounds. Controversial prompt? Guardrails ensure you don't cross the line.

Am I in the right direction, or am I just going off the rails?

Also wondering if this would become expensive very fast if it ends up using S3 with a large number of PUT calls

thanks for the support!

We use a bunch of caching mechanisms on the LLM requests themselves and extend the same to guardrails now.

So there's 2 levels of cache - the LLM request itself might be cached (simple and semantic) and the guardrail response can be cached as well.

We use a mix of a distributed kv store and a vector DB to actually store the data

Being one of the most downloaded datasets on Huggingface, I was a little bit surprised by how dirty this dataset was. Plus it had very limited information and some incorrect classifications as well.

For an internal experiment on building a "Truthful Evaluator", we picked up this dataset and tried fine-tuning a model on these 8000 odd examples.

Realised that it needed: 1. Cleaning up 2. Some reclassification

But, most importantly - it lacked context data. It only had a link pointing to the source which was also absent for a few rows.

We scraped the internet for the link in the dataset, matched it to the question and narrowed down on a small context to be added to the main dataset.

Releasing it publicly so that someone else may avoid the 2-3 days of pain of wrangling with this data.

Here's a mega guide on keeping costs low with LLMs - https://portkey.ai/blog/implementing-frugalgpt-smarter-llm-u...

tl;dr: - Keep prompts short, combine prompts or make more detailed prompts but go to a smaller model - Simple and semantic cache lookups - Classify tasks and route to the best LLM using an AI gateway

Portkey.ai could help with a lot of this

At Portkey, this is a problem we deal with quite a bit. Also the reason that Datadog and the traditional observability vendors did not work for LLM use cases since they're not built to handle large volumes of data.

We've done this through a careful combination of Clickhouse + MinIO for fast retrieval of log items + selected retrieval from the MinIO buckets.

Cost becomes a very big factor when managing, filtering and searching through TBs of data even for fairly small use cases.

One thing we lost in the process is full-text search over the request & response pairs and while we try to intelligently add metadata to requests to make searching easier, it isn't the complete experience yet. Still WIP as a problem statement to solve and maybe the last straw here. Any suggestions?

Openrouter packages APIs and most companies prefer having individual relationships with AI vendors. Choosing an AI gateway might be another way to go

Individual subreddits have usually been my go-to place to discuss and understand the nuances of academic papers. Love LocalLLama for discussions on generative AI papers.