sachamorard

What does that have to do with the topic? Someone is sharing something serious on this thread, I don't see how your message helps.

It's far from being a foolish idea, and it seems to me that the project correctly documents its own limitations. It's rare to see a README list so precisely where the tool falls short. Then, yes, input tokens are cheaper than output. But when it comes to find ways to get reduce costs, you have to explore all the options.

More info in the GitHub repo, in the reports folder (sorry, I'm not sure I can add the link here without being flagged).

"Codex + Edgee consumes roughly half the fresh tokens of the normal Codex baseline. Output tokens are marginally higher (+3,312, +19.5%), suggesting the Edgee scenario produces slightly more verbose responses but dramatically reduces context ingestion."

The compaction problem described here is worse than it looks because of the asymmetry between the compactor and the reader. The model doing the compaction has full access to everything, it can see all six rules in the policy, the exact budget figure, every constraint. The model reading the summary has no reference point to notice what's missing. There's no checksum on memory.

The article mentions the void between volatile KV cache and permanent weights. One thing that lives in that void: compression results. At Edgee we cache prompt compression outputs in a globally distributed KV store specifically to avoid recomputing them on every request. It maps naturally to the architecture, the cache is already the right abstraction, you're just caching one layer higher.

The interesting property is that compression results for similar contexts are often reusable across sessions, which the KV cache itself never is. The Greg Egan framing is apt. The trajectory from MHA to GQA to MLA reads exactly like a series of decisions about what's worth remembering in full fidelity vs. what can be abstracted. The difference is Egan's citizens chose their own compression ratios.

Cool idea. I have had rather a bad experience with semantic caching. Do you have benchmarks that demonstrate the effectiveness?

Really cool concept! The spatial sorting into llms.txt is a clever touch. ;)

That's exactly the trade-off we're pointing at.

One nuance we've been seeing in practice is that the "utility" of a token isn't purely semantic: some tokens carry behavioral constraints (negations, numeric bounds, formatting rules, safety instructions) and their removal can cause discrete failures rather than smooth degradation.

And yes, since cost scales linearly with input tokens, reducing prompt size (I mean context size) can improve both spend and latency.

For years, JavaScript SDKs have been the go-to solution for integrating third-party services into web applications: analytics, A/B testing, tracking, personalization, and more. Let's hope Edgee Components will help developers to use Wasm with less complexity for concrete use cases. Good job Alex ;)

WebAssembly could change how we build and deploy applications, offering cross-platform portability, near-native performance, and enhanced security. Our latest article delves into five lesser-known aspects of Wasm, providing code examples and insights into its potential. Whether you're a seasoned developer or just curious about emerging technologies, this piece offers valuable perspectives on Wasm's capabilities.

Excellent article, Alex! It does a good job of outlining the differences between edge computing and cloud computing. A conversation that becomes increasingly relevant as architectures evolve.

On the other hand, what I find interesting is not at all the political substance of what Macron says, but rather the fact that he agreed to take part in a format that usually welcomes very different people.

The well-known podcast GDIY (Do It Yourself Generation in English) hosted French President Emmanuel Macron and has just published the episode. What's really crazy is that this podcast usually hosts European entrepreneurial figures. In the midst of a political crisis, and having just dissolved the French National Assembly, President Macron chose GDIY to talk...

Fastly just dropped a new feature called "AI Accelerator." I haven’t had the chance to play around with it yet, but I’m really keen to see how they're managing to speed up inference processes with clever caching. Sounds pretty incredible, right? While Cloudflare and Akamai are leaning hard into AI with edge GPUs, Fastly is sticking to its roots with a unique twist on speeding up LLMs... Interesting!