kmike84
My point is that no-AI assisted development is becoming very niche; it's not a right approach if you want to get things done, and AI-assisted doesn't mean lower quality. And if you against paying evil Anthropic and OpenAI corporations you have options like local LLMs.
But I feel it's a bit crazy not to use any AI assistance to develop real code these days, if you care about outcome. You can get to result much faster, and have a higher quality solution, if you use it right.
The problem for open source is low quality slop, not AI assistance. Not sure what's the solution though.
My $20 xeons 2696v2 and 2670 are tool old for real offloading, and sata ssd drives are likely a bit slow for ssd streaming :)
These old workstation platforms seem to be perfectly fine for hosting GPUs for local llms.
Estimated speed on your machine
Context tokens Tokens / sec
25 000 17
50 000 16
75 000 16
262 144 12
Is it a lack of optimizations, or incorrect numbers, or a benchmark artifact (e.g. something which is harder for spec decoding than usual agentic sessions)?
Quantization is another thing. There are so many engines launched with claims about speed, but in many cases it's optimizing specific lower-quality quants. When you have enough resources, you usually want something like W8A16 + full precision KV cache working as fast as possible, not yet another W8A8 or W4A16.
In general, it seems new models are released so fast now - engines don't always have time to really polish the implementation before the next model is released
I found it to be a good baseline, but at least on Mac there was always something way faster, and/or with better memory requirements - like you said, ds4, omlx, mtplx, etc. It seems if you use local LLMs for real, there is very little reason not to use one of the more optimized engines.
3 main failure modes I observed in the engines:
* Not using best available spec decoding
* Using too much VRAM for KV cache (e.g. KV cache used to take almost nothing in ds4, but huge amount of VRAM on unsloth/llama.cpp for deepseek models)
* Degraded performance at large context sizes - benchmarks at 4K or 32K are awesome, but at realistic 100-200K it's slower than some stupid baseline
As for KLD, have you tried it on something which is even closer to e2e task, like agentic traces from https://huggingface.co/datasets/nvidia/Nemotron-Cascade-2-SF... or similar datasets?
But I think there is a bit more to this:
> Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks.
A common issue is that it's rarely mentioned on which dataset KL-divergence is computed. It seems the most common dataset is wikitext - maybe because of tradition, to make numbers more comparable? It's not measuring how well the model follows bf16 on agentic tasks.
I've been trying to check KLD recently for some quants of Qwen 3.8 27B, and the numbers are dramatically different, depending on which dataset you use. KLD computed on agentic traces is much higher, and top-1 % is way lower than if you compute it on chats or wiki text.
You look at a published number, and see "oh, nice, top1 is 99% - quant is different just in 1 token out of 100", but chances are it's computed on wiki, and on agentic / coding it can be 10 tokens out of 100.
Common intuition is that on agentic tasks errors compound, and that's why it degrades more than metrics show - but maybe the metrics themselves are also wrong, too optimistic.
Still investigating it though :)
No person with real practical experience and real use cases will be using these ancient models as examples, when talking about local LLMs.
pi, with a plugin to do web search / web fetch.
> You have previously commented on using OC/GLM. R u going to stock with it?
For personal use - probably yes, z.ai + kimi + opencode go subscriptions, with some share of local models now. For work - claude code, codex.
Also, DeepSeek's KV cache requires less RAM than Qwen's. In concurrent situations (on servers) you load model weights once, but you have different context in each parallel session. So, it can also need less RAM than Qwen to serve, even if it's a larger model.
Qwen 3.8 27B doesn't look benchmaxxed. These "52 AA score" numbers feel real, which is surprising. I've been using it locally for a few days for other tasks as well. If not the speed, I'd be totally happy to use it as a daily driver instead of cloud models, it is that good.
--- (benchmark, to get an idea):
1. First, initial prompt which is not super precise - similar to how I'd write a task when talking e.g. to Opus. I'm describing an idea, and asking model to come up with some plan, and also to criticize the approach. Task is about implementing a particular pi extension. I'm checking if a model actually understands what I'm asking.
2. Then, as a follow-up, I ask to research alternative implementations, research UX of similar extensions, etc. It needs to do web searches, inspect open source codebases, read articles and papers, etc. I don't prompt to do this exactly, but I expect good models to figure out they need to do it.
3. Then, implementation.
Also, one finding: Q4 and Q8 seem to have very different behavior in this benchmark. Q4 produces 2-3x thinking in the end, and makes more turns - it seems it makes more mistakes, and needs effort to recover from them, while Q8 gets more things right in a first try. In the end, quality is roughly similar, but Q8 gets there much faster, especially the implementation (tried it several times). Could be a difference between concrete artifacts, or between runtimes, I don't know, but be careful - it seems the real-world experience with qwen 3.8 27B can be vastly different, depending on how it's set up.
Regarding DeepSeek 0731 vs Qwen 3.8 27B. On this benchmark, Qwen understand my intent better, it's better at research, and I also liked its implementation more. But: if you're more precise in what you ask, 0731 is also very good, and it's quite a lot faster on mac; raw speed is better, and it needs less thinking to get there. So, I'd say it's a tie in practice, both are awesome :)
Claude Code is better. But Opencode + kimi 2.6 is workable, which is big. For bare code writing, if you know what exactly you want, most popular models are fine (deepseek, kimi, etc), it feels more or less the same as anthropic models.
At the same time, Opus seems to understand my intent way better than e.g. deepseek. I need to be much more precise with my prompts when using deepseek - it often goes in a wrong direction if I'm lazy. This results in a workflow which feels quite a lot different from Claude Code.
Kimi is in between - for me it brings back "lazy prompting" workflow, and I can trust its plans more than deepseek. It enables a workflow similar to Claude Code, it's workable, but it is a bit worse everywhere. Smaller context, a bit more errors, decisions are a bit worse, recommendations are a bit worse, debugging capabilities are a bit worse, etc.
On the usage side, $100 Claude plan is a great value actually. On paper, per-token kimi is way cheaper, but Claude subscriptions are heavily subsidized - you get much more tokens than $100 can buy you. So, in the end, opencode + kimi vs claude code could be of a similar cost, for similar usage patterns. Deepseek can be cheaper, and it has insanely cheap cached tokens, but experience may vary - depending on your habits, you may need to adjust how you work, coming from claude code.
I'd say for side projects something like $10 Opencode Go plan + $10 of extra DeepSeek v4 credits (e.g. on OpenRouter) can be very workable.
* plugin for Logic Pro to simulate how a mix will sound on my macbook and phone (I captured real impulse responses for that, sounds very close)
* an app for spaced repetition for guitar video lessons / their parts (no idea why platforms like truefire don’t have this feature)
* workout planning/tracking app
* an app to create impulse responses for acoustic guitar, to make it sound good live
It feels insincere and manipulative, especially when I don't know upfront if the content (music, video, text) is from another human being or from AI.
AI will become good enough to write songs better than humans; it's a matter of time. But it feels like someone tries to hack my mind, exploit my human instincts, it doesn't feel like genuine art the way it was for the whole human history - people expressing themselves, creating and sharing something beautiful with each other.
The end result is an automated personalized "enjoy" button, and this is sad.
To compute accuracy, you compare the moves which are made during the game with the best moves suggested by the engine. So, the engine will evaluate itself 100%, given its settings are the same during game and during evaluation.
You get 99.9something% when you evaluate one strong engine by using another strong engine (they're mostly aligned, but may disagree in small details), or when the engine configuration during the evaluation is different from the configuration used in a game (e.g. engine is given more time to think).
> I have a good answer: my job is to double our value-add capacity over the next three years. Essentially, to double our output without increasing spending.
> You know what? With my XP plans and the XP coaches I’ve hired, it’s totally doable. I think I’m being kind of conservative, actually.
TBH, this part felt off to me as well.
rfc3986 may reject URLs which browsers accept, or it can handle them in a different way. WHATWG URL living standard tries to put on paper the real browser behavior, so it's a much better standard if you need to parse URLs extracted from real-world web pages.
We need a better URL parser in Scrapy, for similar reasons. Speed and WHATWG standard compliance (i.e. do the same as web browsers) are the main things.
It's possible to get closer to WHATWG behavior by using urllib and some hacks. This is what https://github.com/scrapy/w3lib does, which Scrapy currently uses. But it's still not quite compliant.
Also, surprisingly, on some crawls URL parsing can take CPU amounts similar to HTML parsing.
Ada / can_ada look very promising!
There is some debate if it is documentary or not (the story is almost too good), but it seems the evidence suggests it is real.
EDIT: sorry, I missed the "last 4 years" part in the question. This film is older than that.
But there is an issue if lru_cache is used on methods, like in the example given in the article:
1. When lru_cache is used on a method, `self` is used as a part of cache key. That's good, because there is a single cache for all instances, and using self as a part of the key allows not to share data between instances (it'd be incorrect in most cases).
2. But: because `self` is a part of a key, a reference to `self` is stored in the cache.
3. If there is a reference to Python object, it can't be deallocated. So, an instance can't be deallocated until the cache is deallocated (or the entry is expired) - if a lru_cache'd method is called at least once.
4. Cache itself is never deallocated (well, at least until the class is destroyed, probably at Python shutdown). So, instances are kept in memory, unless the cache is over the size limit, and all entries for this instance are purged.
I think there is a similar problem in the source code as well, e.g. https://github.com/Textualize/textual/blob/4d94df81e44b27fff... - a DirectoryTree instance won't be deallocated if its render_tree_label method is called, at least until new cache records push out all the references to this particular instance.
It may be important or not, depending on a situation, but it's good to be aware of this caveat. lru_cache is not a good fit for methods unfortunately.