News for hackers. Stuff that matters.

johnnymakes

Haha – sorry to disappoint!

SOTA on the hardest AI memory benchmark (BEAM, 10M tokens), with a smaller model

Hey HN. I'm Johnny, founder of Exabase. We just hit the highest reported score on BEAM, the hardest AI memory benchmark, at every scale up to 10 million tokens. We also ran our evaluation using Gemini 3 Flash, when all previous leaders depended on a much larger model (Gemini 3 Pro).At 10M tokens, the scale is vastly larger than any model's context window, so context stuffing isn't an option...

( Read More... | 3 comments | on HN )
Hey HN. I'm Johnny, founder of Exabase.

M-1 is our first-generation memory engine. We evaluated it against LongMemEval, the most comprehensive public benchmark for conversational memory retrieval: 500 questions, ~115k tokens of history, relevant information scattered across sessions and buried in noise.

M-1 scored 96.4% at top-50 retrieval, the highest reported score, with consistent performance across all top-k's. The most interesting part is that we did it with Gemini 3 Flash, while every other system on the leaderboard used Gemini 3 Pro.

A bigger model can compensate for weaker retrieval – absorbing a larger, noisy context at the cost of increased inference. We deliberately chose a smaller model to isolate retrieval quality from model capability and solve for real, production use. This result is Pareto optimal: cheaper and better performance, which is what we're solving for.

Our results are in the spirit of real, production use – so we used a single generic prompt for our answerer – stripping out the question-specific prompt language we observed in other benchmark attempts/runners. The methodology, prompt, and full results JSON are all linked in the research post.

The research post also has a discussion of the evaluation ceiling we hit at this accuracy level (there are errors in the benchmark itself which create a noise floor – we reported a few upstream to the benchmark creator).

Happy to discuss the architecture, methodology, or how we think about memory retrieval differently!

It's a reasonable question: in short, Notion is note and notebook centric – everything falls into one of these 2 categories.

Our structuring is that there are objects and wrappers, but we aren't so prescriptive on what these things have to be exactly... but an object could be a note.

So we're more of a filesystem that supports notes, rather than a notebook that can host rich media.

We break the 2 types out in the UI, listing separately. So the user gets to choose. This gives a kind of "best of both worlds" outcome, where a user never gets zero results (and can be make very low fidelity queries), but also doesn't have to sacrifice specificity when the query they're making is one of precision and intentional retrieval.

Show HN: A unified cloud drive with AI search for your files, photos, bookmarks

https://fabric.so/

Hi HN,Today we’re launching Fabric – an internet drive that combines varied data-types (bookmarks, files, notes, images & more) into a single home, with AI-powered semantic search, so a user can find anything again in their own words.Without Fabric, this data would scattered across different stores and the user is burdened with finding the right item in the right store (often via brute...

( Read More... | 5 comments | 4 points | on HN )