everythingmeta

One of the authors here. I work at Parallel. We got Jev-pilled a few weeks ago and have been trialing decision models nonstop across all sorts of use cases, so we wanted to see how the Jev-like models compare. The tasks are a mix: some where we previously used fine-tuned models, some where we used general LLMs, and some we hadn't pursued at all before Jev. What I think makes our analysis a little different is that we tested on human-labeled data from our production pipeline rather than synthetic benchmarks, which can be made to show almost whatever you want (I suspect that's how so many OSS models were ranking above Jev in early evals).

What we found:

The ergonomics of serverless decision models were really hard to beat. Cost, latency and accuracy were all near the top for Jev. Latency especially was very consistent, which matters a lot to us; historically that's where we'd have reached for a fine-tuned classifier.

Perplexity's decision model was a close second. It sometimes outperformed Jev, but its latency was more variable. That said, I believe it's also half the price now.

Self-hosting open-weight decision models wasn't a big unlock. In principle, at peak utilization we'd get into the ballpark of Jev's API performance. That said there is a world of optimizations that we just didn't have the time to pursue.

Iterating on prompts got meaningful improvements out of almost every model. That's much nicer UX than having to retrain a classifier.

Caveats: there's already a new Jev out, plus a few other models, and we can't keep up. You should ultimately try these on your own tasks.

congrats Philip and the rest of the team!

nice to see a tutorial that recognizes the case where the underlying data can change and the embedding needs to be updated.

Any plans to write a tutorial for fine-tuning local models?

Retain.ai | Onsite | San Francisco | Founding Team Engineer - Generalist or Full Stack | $80k - $120k + substantial equity

At Retain we help companies understand customer interactions across teams, tools and applications. Our solution integrates with the day-to-day tools companies use (e.g. ticketing, calendar, collaboration tools, CRM apps) to obtain customer interaction data. Retain enables data driven decisions in everything from pricing (by increasing revenue on expensive customers) to product roadmap (by focusing on features that have the largest impact).

We’re venture-funded and this is our first engineering hire outside of the founding team. This is a true generalist role, so the expectations is someone who is willing to grow and adapt with the role. Substantial responsibility and equity upside. Web and data experience are likely to be biggest assets — we’re on AWS and Rails with the usual accoutrement but ultimately tech agnostic.

Requirements * You are nice, smart, and action oriented * At least 2 years experience building web applications with modern technologies with solid understanding of databases * Some experience with big data frameworks, analytics, ETL, or integrations is a strong asset

https://angel.co/retain-ai/jobs/272082-full-stack-engineer-f... or Ping me at [email protected] if interested.