News for hackers. Stuff that matters.
Re:We decreased our LLM costs with Opus by syntaxing on Tuesday April 28, 2026 @11:00PM (#47943726)
Is RAG dead? I would be very surprised a local small SOTA embedded model like llama-embed-nemotron-8b doesnt outperform the Haiku layer for this application. Should be pretty cheap and easy to prove out. With 32K context size, you can literally one shot the whole ticket.
Re:We decreased our LLM costs with Opus by preommr on Tuesday April 28, 2026 @11:16PM (#47943809)
Yea, but RAG takes effort. At the very least some kind of system to organize the documents and do the retrieval.

My theory is that the AI frenzy has reached new levels of insane, where it's literally just throw anything and everything at the model, and just burn tokens to let the AI figure everything out. Why bother paying the upfront cost for a RAG, when the models/agents are constantly evolving, so just slap in a markdown file telling it to check a folder, and call it a day.

Like in design world, people are doing minor tweaks like changing the spacing by typing in prompts instead of just changing a number in an input field. We are legitimately approaching just using llms instead of calculators, or memes like that endpoint that calls an llm to generate the code to do some business logic, rather than directly code the logic.

Re:We decreased our LLM costs with Opus by shad42 on Tuesday April 28, 2026 @11:32PM (#47943897)
IMO RAG is mostly dead. The game changer with newer models like Opus is the reasoning. So instead of pushing all the context up front (RAG style), it's better to give strong primitives (eg. bash, SQL) and let the agent figure it out.

It's what Claude Code is doing now and the principles we applied for Mendral as well.

That said, you're right that some smaller models can outperform Haiku and we're thinking supporting oss models at some point. But it does not change the core design principles IMO.

Re:We decreased our LLM costs with Opus by Salgat on Wednesday April 29, 2026 @12:25PM (#47950637)
It's more accurate to say that RAG is alive and well and is just incorporated into the agent's responsibility, it's just one more tool that it can call on instead of the user manually doing it.