pu_pu

Is it open source?

The biggest problem with retrieval is actually semantic relevance. I think most embedding models don't really capture sentence-level semantic content and instead act more like bag-of-words models averaging local word-level information.

Consider this simple test I’ve been running:

Anchor: “A background service listens to a task queue and processes incoming data payloads using a custom rules engine before persisting output to a local SQLite database.”

Option A (Lexical Match): “A background service listens to a message queue and processes outgoing authentication tokens using a custom hash function before transmitting output to a local SQLite database.”

Option B (Semantic Match): “An asynchronous worker fetches jobs from a scheduling channel, transforms each record according to a user-defined logic system, and saves the results to an embedded relational data store on disk.”

Any decent LLM (e.g., Gemini 2.5 Pro, GPT-4/5) immediately knows that the Anchor and Option B describe the same concept just with different words. But when I test embedding models like gemini-embedding-001 (currently top of MTEB), they consistently rate Option A as more similar measured by cosine similarity. They’re getting tricked by surface-level word overlap.

I put together a small GitHub repo that uses ChatGPT to generate and test these “semantic triplets:

https://github.com/semvec/embedstresstest

gemini-embedding-001 (current #1 on MTEB leaderboard ) scored close to 0% on these adversarial examples.

The repo is unpolished at the moment but it gets the idea across and everything is reproducible.

Anyway, did anyone else notice this problem?

Not at all in my opinion. Its a zero sum game against anti bot technologies also employing AI to block scrapers.

This really feels like a missed opportunity to build something genuinely new, something that actually plays to the strengths of LLMs, instead of just embedding a fixed set of app screens inside chat.

Ideally, users should be able to describe a task, and the AI would figure out which tools to use, wire them together, and show the result as an editable workflow or inline canvas the user can tweak. Frameworks like LlamaIndex’s Workflow or LangGraph already let you define these directed graphs manually in Python where each node can do something specific, branch, or loop. But the AI should be able to generate those DAGs on the fly, since it’s just code underneath.

And given that LLMs are already quite good at generating UI code and following a design system (see v0.app), there’s not much reason to hardcode screens at all. The model can just create and adapt them as needed.

Really hope Google doesn’t follow OpenAI down this path.

So, I have not tried it on a very complex database myself so I can't comment how well it will work in production systems I have tried this approach with a single Big Query table and it worked pretty well for my toy example.

If by 3.5 you mean ChatGPT 3.5 you should absolutely try it with newer models, there is a huge difference in capabilities.

Can you please add support to add descriptions of each column and enumerated types?

For example, if a column contains 0 or 1 encoding the absence of presence of something, LLMs need to know what 0 and 1 stand for. Same goes of column names because they can be cryptic in production databases.

This is not a difficult problem to solve. We can add the schema, columns and column descriptions in the system prompt. It can significantly improve performance.

All it will take is a form where the user supplies details about each column and relation. For some reason, most LLM based apps don't add this simple feature.