Bocajmai

When Jev launched last month I assumed it was a novelty. Then we tried it on the job our AI receptionist finds hardest. A caller asks something oddly specific, like whether a free valuation ties them into selling with the agent, and the answer is buried somewhere on the business’s website.

We gave Jev a one-line description of every page on the site and asked it one choice question, which page answers this, with “none of these” as the last option. Across four of our customers’ sites (22 to 232 pages) and 168 caller questions, it put the right page first 88% of the time, against 70% for our production embeddings. The number I care about more is how often the LLM got handed the wrong page, 3% against 33%. When the site didn’t have the answer, Jev said so 95% of the time and our current search 23%. Each lookup took about a third of a second, and all 1,449 requests cost about 45 cents.

Fair warning, this piece is written for vibe coders and tech business owners rather than engineers, so it’s the plain English version. The method, charts and the bits that went wrong are in the companion write-up: https://sayora.ai/many-hands/jev-vs-embeddings/

Most of the failures were ours. Jev reads page descriptions literally. A plumber’s installation page mentioned water softeners in the body but not in its description, so Jev said “none of these” at 96%. Above 255 options or about 64k tokens you need embeddings to shortlist first. And we couldn’t find a published data processing agreement (DPA), which matters with UK callers’ data.

Happy to answer anything about the setup or where it didn’t work.

Fair, and no, it doesn't prove that. On ranking alone it didn't beat a regular LLM. GPT-6 Luna picking from the same list put the right page first 86% of the time against Jev's 88%, and the interval on that difference (-4 to +7 points) says there's not actually a clear winner.

Where they did differ however! Was speed. And saying "none". Luna took 0.66s at at the median and 1.27s at the p95. Versus 0.28s and 0.42s with Jev, which really matters for live agents, like our use case, which needs to happen quickly and naturally on the phone. It also caught 32 of the 40 unanswerable questions vs Jev's 38, so it handed over a wrong page 8% of the time against 3%.

On a wider data set, also fair. But it's interesting enough for us to pursue, so I'll report back with more data soon hopefully!

Thanks for taking the time :)

For me, the hardest part of Shape Up isn't understanding the principles, it's knowing when a pitch is actually done. What's the right level of detail? How do you know if you've shaped enough?

This is a reference I made for myself. Five required sections from the book, quality checks for each one, and one extension I added (Open Questions as a shapedness metric).

Would love to hear how others have adapted Shape Up for their context.

True, but is that a strategy problem rather than an operational one? Have they lost the ability to ship, or are they misaligned on what people want? You need both.

I'm not sure I know the answer, but it's a very good point! Thanks

I started researching how Michelin-starred restaurants coordinate their teams - not because I care about fancy food, but because they solve exactly this: small specialized teams creating something complex together under pressure.

The parallels to software are exact. Front of house/back of house coordination. Pre-service meetings. The "pass" as a coordination point. Station ownership. The quiet kitchen principle.

Then I discovered Basecamp's Shape Up is essentially their version of the brigade de cuisine system. 6-week cycles, betting table, appetite setting, cool-down periods. It maps perfectly.

I wrote this to work through what small bootstrapped software companies can learn from how these organizations actually operate. Would love feedback from others dealing with this coordination problem.

Hmm, your ChatGPT subscription would come up as "OpenAI ChatGPT Subscription" in your bank account, or it does for me. That sounds more like OpenAI API use? Which would not include "ChatGPT" in the transaction.

If it doesn't sound right, maybe check you've not leaked any keys in public github repos etc? There's crawlers hunting for exposed API keys. Stay safe out there!

That's a very valid concern. What kind of maintenance though? Most of the examples in this article are very simple. Maintenance might involve iterating on features, no real live services that would require any work.

Security patches though, yeah, that's tougher. My position is that if security is a concern, you need to hire someone. As much as many of these tools can integrate with databases and set up auth, I'm not sure how much I trust it personally. Especially if the actual code is hidden.

Agreed! The similarities are stark. let's take the comparison further... Nvidia is the dotcom bubble's Cisco.

But the internet stuck around. It was still useful.

Just because something is overhyped, or doesn't meet the standards of the world's most demanding engineering organisations, doesn't mean that it's not here to stay and improve.

Perspective.

That's very true. But programmers are extremely expensive. I don't think these tools or vibe coding replace expensive programmers, but they do make it possible to noodle where once it would have taken a potentially very risky commitment for a small business / solopreneur.

Yes but have you tried vibe plastering? It's like artex, but not intentional.

I would encourage you to try Replit!

Deployment is literally a button. Granted, to a replit subdomain. But following some documentation to add your own domain is not out of reach.

I'm not sure it's explicitly gatekeeping, more a lack of perspective.

The benefit and excitement is most felt by people with little to no experience writing code themselves. The fear seems to come from building.. what, code for critical infrastructure? That's just not what we're talking about here.

Is it propaganda if we can see it feel it and use it ourselves? I'm not so sure.

The hype is real, the bubble is real, but that doesn't mean it's meritless.

We're not talking about building an ecom store, though, are we?

We're talking about building useful tools. Small things, that work in a way that makes sense to the individual.

Something that could be churned out in a day or two by a competent dev, but is completely out of reach for Joe public.

That's where we're at.

It could go so wrong, but the vectors for disaster are already innumerable in this world we live in.

I prefer to think about the ways it could go right. To empower people to create is the very purpose of computers, no?