hgoel

Do you not do any good deeds unless they directly benefit you?

Yes, I feel Qwen3.8-FN is the first model that seriously can replace some lower tier subscription usage without requiring much more than a high end gaming PC. Qwen3.6-35B was impressive but not quite good enough, and 27B takes a lot more time.

Lately I prefer GLM5.3-Flash since I feel it is noticeably smarter and I have the hardware for it, but FN's performance has me looking forward to Qwen4.

It also works for flash models that push what capability is reasonably available locally, eg with Qwen3.8-flash, Qwen3.8 27B and GLM5.3-flash.

Oh yeah, Space Bunny was awful, massive step back from even Qwen3.8-Flash-Next

People using API keys heavily, are either making enough money from whatever they're doing to afford Anthropic's high prices, or are using a mix of locally hosted and cloud-based models. E.g. using a cloud model as a planner/orchestrator, with a local model as a subagent that follows the orchestrator's designs/instructions.

A bit presumptuous to assume this is about PhDs leaving academia for cushy bigtech jobs.

My coworkers are all PhDs, people whose PhDs were in topics like QCD working on completely unrelated computational physics, various biology, chemistry PhDs doing experimental physics, astronomy etc (and vice versa). I am considered strange for still working on the same topic as my PhD.

As another post here says, they're probably referring to an internal monologue. I imagine that in reality it's more of a matter of how active that monologue is.

Though, also, I think that the internal monologue is maybe less an aspect of how we think, and more an aspect of how we remember. I haven't looked into it, but my feeling is that a more active internal monologue might be correlated with someone who has an easier time remembering sounds than, say, images.

It doesn't have to mean that every action is accompanied by a monologue, a task like tying laces can be done with no active thought about the specific movements.

This doesn't really fit with my experience. Most PhDs do not continue in the exact same topic they did their PhD in.

You hire a PhD if you need someone that can do research. It's a bonus if you find an exact match to your research area, but it is certainly not a strict expectation.

I don't consider myself qualified to make that call, though I believe there's plenty of room for improvement. But I don't mean inference compute, which is fairly accessible for well-off enthusiasts even now. Training is a lot more expensive though.

Except compute

Pretty funny to say "us peasants aren't concerned with ... cost/token", as if most people ("peasants") won't find it absolutely insane to be paying $200/mo for a subscription.

Opencode also has a setting to not route to China, though this would probably just disable the models that do not have providiers in alternative locations.

Biologists already addressed this idiocy a few weeks ago [1]. Making a biological weapon requires facilities that only some nation states - and comparable companies- have access to.

AI isn't going to magically assemble a DNA sequence for you and a computer design is not a guarantee of something that works.

AI restrictionists love to pretend that their toys will, with the power of intelligence, magically bypass fundamental limitations of reality to conjure up fantastical weapons of mass destruction if they throw enough computers at the task. Unfortunately for them, that is not how the world works.

[1] https://www.businessinsider.com/researchers-ai-apocalypse-sk...

You can program a computer to do things and they can interact with tools to do things too. Doesn't justify restricting what you can do with them.

If you program a computer to make cocaine for you, you're the one violating the law. If you ask an LLM to do the same for you, you're the one violating the law. The LLM's capabilities are irrelevant.

I'm not really seeing how an AI worm is meant to be ending the online world. Even a 9B model is a significant chunk of RAM on consumer devices.

They talk about distributing layers across devices, but then you'd have communication on the network and hours of driving the devices full tilt to get through even a single turn.

There's so much that can be monitored to catch these, even if we assume that the worm could hack the OS's usage monitors, it can't hide the heat and power draw.

Datacenters and PCs would adapt, requiring better monitoring than the AI companies' method of letting things run blindly for days/weeks.

IoT would be tougher, but it is also the weakest hardware, sometimes battery powered, thermally constrained, lacking in memory for processing even one layer or lacking the necessary connectivity.

Very surprising that a recognizable news outlet actually called out the blatant fearmongering in a headline instead of just uncritically regurgitating Anthropic's claim.

This is just yet another attempt from the cult to try to build a regulatory moat so they can stave off having to compete on price, like how OpenAI is having to do on the lower tier models (among other things).

To be fair, I think it's mainly America that bought its own propaganda about being inherently superior to everyone else.

Most of the populist nonsense ultimately comes down to their delusion that they're being cheated out of the position they deserve as inherently superior people.

Most of these custom engines exist because the more general purpose projects make contributing much harder than maintaining a custom engine (somewhat understandably). Consequently they take forever to adopt optimizations, and it is somewhat pointless if llama.cpp or vllm only reach performance parity with custom engines when the LLM architectures being optimized for are no longer the best ones.

The benefit of llama.cpp etc is just ease of use for people not already familiar with the local AI landscape.

One of the architectural changes expected in the next series of Qwen models appears to be well suited to SSD streaming.

But also, there probably still is a lot of room for tuning and better utilizing the streaming capability, e.g. better speculative streaming to have the activated experts loaded in RAM by the time they're needed.

I get the intention, but I feel it may be a particularly extreme comparison, since the US being what it was to the 20th century involved two world wars, decolonization, the cold war, and decades of reconstruction.

Currently, even if China pulls ahead, it doesn't have to mean things getting worse in the US. Contrary to what the current government seems to believe, growth is not necessarily zero sum. Despite Americans' perception of Europe, they too remain competitive in various domains of research. Similarly with, say, Japan.

Ah yes, I'm engaging in bad faith because I am opposed to the actions of western AI companies, while you're engaging in good faith because "Chyna bad" regardless of real differences between their approach to this technology and completely ignoring that several incidents were made public by third parties, flawless logic.

I had a feeling that transpilation was maybe somehow different from decomp/recomp but convinced myself they were close enough.

Thanks for the interesting "erm-akshully" on the distinction!

Why bother replying if you don't intend to engage in good faith?

What's low quality about a working transpile? Missing out on engine details/optimizations is a shame, but if someone cares about that they can do that themselves, it doesn't make the transpile itself low quality.

The Chinese are not as flush with compute as to afford to just forget a few thousand agents running for a few weeks.

They also aren't the ones screaming about how close they are to destroying the world. They're approaching the tech like building a tool rather than a god.

OpenAI and Anthropic have shown us that they have a strong incentive to leave holes in their systems so they can use them for marketing and regulatory capture.

"I cannot tolerate living in a world where we aren't the ones responsible for ending the world!"

It does not matter if competitors would not also agree to destroy their companies.

If I was competing with a bunch of people on building something that I came to believe would be an extinction level event, I wouldn't be saying "Even if I slowed/stopped, the others would not, so I have to keep going". I'd say that I want absolutely nothing to do with pushing it further, stop my work, then regardless of consequences, do everything possible to stop my competitors regardless of legality

I've been thinking that part of the point of handing out a bunch of science prizes to the techbros yesterday was to try to overshadow the Nobel prizes.

I think that's one form of FTL (versus what I described, which is not FTL at all). Another form of FTL would be the "jump"/wormhole kind where c stays the same, you bypass the distance instead.

My understanding is that most of these quants - to achieve better precision than just naively dropping bits - require some finetuning against the activations of the higher precision model, so the line already seems kind of blurry.

I understand that the interest in these extreme quants stems from wanting to maximize the capability the average user can get from a local LLM in this era of ludicrously expensive memory, but I wonder if this is maybe targeting the wrong axis?

We've been seeing various optimizations towards streaming, that have been much more impactful in the local AI space, e.g. MoE models where the less busy experts are offloaded to slow RAM or even pruned entirely, engram tables that can be read from NVMe instead of sitting around in RAM etc.

Maybe the trick with these extreme quants would be to increase the total parameter count while quanting individual weights, such that maybe the active parameter count comes down, or streaming weights from RAM or disk becomes more efficient, or cache behavior improves? Say, replacing a single 4bit/weight matmul with 3 1bit/weight operations that produce a much closer result than a single 1bit/weight matmul would.