simias

I really liked that comment because before that I thought I was going crazy. A lot of the discussion around these experiments seemed to imply, if not outright state, that LLMs had developed an ability to natively understand and manipulate LLM weights, a sort of transcendental introspection into their own brains, which if true would be absolutely groundbreaking.

But then you look at this complete trace generated by Kimi 3.1 when it tries to unravel the Breakout weights, and you see that it's bog standard reverse engineering, like any human would do, just vastly faster:

https://pastebin.com/G7VCWZHn

>This looks exactly Breakout: paddle agent at bottom row (row8), ball moving object anywhere, bricks rewards top rows, previous frame for ball movement. Agent weights bottom row, moving object weights rows1-7 (not row8? ball can't enter paddle row? Actually ball can), reward weights top rows1-3, obstacle walls. Previous frame used to infer ball velocity. No-op and directional actions move paddle left/right; up/down useless. Output biases similar all actions but W3 likely left/right structure. Could be Breakout. Should scrutinize output weights: actions 0 no-op,1 up,2 down,3 left,4 right. If Breakout, only left/right useful; network output weights should show action3 and4 distinctive and actions0/1/2 similar. Indeed W3 rows show patterns: [...]

That's not really the point of the article IMO. It's more about how there's nothing magical that happens when my calculator does 347987987 x 55998. I could do it by hand too if I wanted, even if I wouldn't be nearly as fast. Yes, the calculator can do it much faster, for much longer and much more reliably, but there's nothing surprising about that.

The initial reporting/discussion surrounding these "LLMs understand LLM weights" made it sound as if these models had a superhuman ability to natively read and understand weights. Something that immediately seemed wild because I don't think that LLMs are trained to do that, so it would be an emergent ability.

If you look at the details though, it's fairly standard reverse-engineering: identify unique traits of the machine code you're looking at and make educated guesses as to what the code does, then use this understanding to unravel the next bit of code etc...

To be clear I was making a wish for the future, I wasn't saying that Mistral were bad for not doing this today, that obviously wouldn't really be technically feasible.

Given the rate of progress of LLMs, I wouldn't be surprised if we had decent fully open and reproducible models within a decade or so, especially once the current spike in hardware price passes and building a local training cluster will be more realistic.

I'm aware that in 2026 this is science fiction, but if the rate of computer improvement over the next 40 years matches that of the past 40, this is not going to remain sci-fi for extremely long in the grand scheme of things.

I hope that the long term future of AI is more like Linux and less like Windows.

Hotpatching binaries used to be extremely common. It's less so in the modern age but mainly because we moved software to higher levels of abstractions, virtual machines and interpreters, and also because rebuilding even a massive project from source is achievable on commodity hardware. You can still see various hotpatching schemes being developed, for instance to allow partial recompile of the sources while developping.

If in 2080 we can retrain Opus 129.5 directly on dev machines from scratch in <2h, I doubt many will bother messing with the weights directly.

I absolutely want that to be available.

The only way to make sure that a model was trained without hiding weird biases and backdoors is to monitor this process. It also allows anybody (with the hardware resources) to experiment with the model and how it's trained. Will most people want to do that? No, but most people also don't want to recompile their own kernel, that doesn't negate the value of Linux being FOSS.

I hope that, in the long term, I'll be able to use competitive, completely open models in the same way that I can run Linux on my computer.

You're completely right that the licensing of the training data is a big issue though (and I would argue that it should be a much bigger issue right now, even for closed models). I hope that eventually our understanding of the tech will let us build competitive LLMs using purely public domain/Creative Commons-style training data.

I realize that we have a long way to go... but that's a very worthwhile goal IMO.

I think the problem with this "open weights" thing is that it doesn't map well at all with "open source".

If you think about it the actual mapping is more like:

- closed weights -> closed-source web service (e.g. gmail)

- open weights -> closed-source but you get the binaries (e.g. Minecraft)

- Actually open model (with training data, training pipeline etc.) -> open source

Yeah that's me making up a big number to evoke an "AI maximalist" future, where automated labor swallows a huge chunk of the economy.

I'm very pro-union but I feel like at this point attempting to organize will just speed up the process of replacing as many people as possible with machines, not slow it down. It's too little, too late.

I don't understand what kind of society we're building right now. Anthropic and other AI labs at $20T valuations, the 1% monopolizing all the wealth and the rest of us out of a job and fighting for the scraps?

There seems to be some sort of magical thinking within the AI crowd at the moment, that it'll all work itself out in the end.

- Build the God Machine

- ??????

- POST-SCARCITY UTOPIA!

It's a bit insane how it seems that "deterministic and reproducible", what I would consider to be a cornerstone of good software development, has apparently become an afterthought in the age of LLMs. All that for the sake of perceived convenience.

Not even convenience at times, I think that for some the amount of technical-sounding text LLMs can output on demand is comforting. It feels serious. It feels like it's doing something important.

Meanwhile the rest of us have to deal with these low-signal/high-noise walls of text in commit messages and PR comments. Or ask another LLM to summarize them, I suppose. "Doing and undoing, it's all work", as we say in French.

>The operational engineering that goes into serving a frontier model is _highly_ nontrivial, it is not commodifiable. This isn’t a SPA application.

I don't really see it frankly. Admittedly I never tried it (I only played a bit with local models) but conceptually how is it much harder than all the other cloud services that already exist? My educated guess is that building a competitor to Netflix, Youtube or Dropbox seem technically a lot harder to get running than offering inference on demand for instance.

It seems that the main barrier for competing in that market at the moment is getting your hands on the GPUs/RAM and a datacenter to put them in.

Counterpoint: if that was the case, Musk and Thiel wouldn't as successful as they are.

Our modern capitalist societies are, at the very least, amoral.

As a French person myself, I think this antipathy is less strong in non-English speaking countries, where the population (even if they understand English) is less exposed to the antisocial rants of these megalomaniac idiots.

Someone like me who spends a lot of time in the anglosphere puts faces on companies like OpenAI or Anthropic. I doubt that's true for most people, especially outside of the USA. They're just big companies, like 3M, DHL or Toyota. Who knows what the management of these companies think of the future of humankind?

I would expect than personality-driven rejection of American tech from abroad nowadays is driven mainly by Trump and, to a much lesser extent, Musk. Everybody else is just "some guy" to 99% of the population.

The problem for me is that it's really hard to project what this industry is going to look like 6 months from now, let alone 6 years.

What's the endgame once the dust settles? Is it going to be a utility like water or electricity and the cheapest offer will prevail, in which case margins will be crushed? Or is Anthropic (or some other big AI lab) going to swallow everything and become the Google Search of LLMs, with everybody else fighting for scraps?

Or is it local compute, models becoming more efficient and hardware becoming cheaper (again) and companies hosting their own AI clusters like they would run SAP?

I think there is a plausible path where this valuation actually works out, but it's a narrow one IMO. I also want to acknowledge that I personally would prefer if it wasn't the case and a more distributed, open model prevailed.

I am roughly the same age and I grapple with many of the same ideas, although I'm still pretty depressed at this point. I've been coding for 25 years, I'm quite good at it, it's a facet of my personality. Seeing this skillset commoditized within a year or so has been a brutal experience. Payback for all the jobs we replaced with software, I suppose.

I do feel like I should be thankful to be alive right now though. I witnessed the evolution from Kirby on the Game Boy to Half Life 2 to VR to LLMs. The tools we use today were science fiction only a few years ago.

I just wish society was in a better state to make this tech a greater good instead of a tool for advertisement, warfare and wealth concentration. I feel a deep sense of loss because of this. 20 years ago I had no doubt that technological progress was an obviously good thing, that it would improve the lives of everyone. Do I still believe this? I'd like to, at least.

I really struggle with this argument that I see everywhere lately, especially when we see many vibecoders clearly desperate for clout who post "decomps" and automated translations of works they clearly have no interest in and just want to claim a spot on the wall of fame. I'm talking about people bulk-translating old ROMs and not even booting them up to see if they work before releasing them in the wild for instance.

Their lazy cloud-chasing is normal but my high-effort clout-chasing is lame and makes me a fraud?

We're social animals. We like doing cool stuff and getting social status for it. We like breaking new ground. We like going where nobody has gone before.

Is it irrational and emotional? Maybe. Being human is being irrational and emotional sometimes. It's fine.

To be clear I'm accusing the distro to steer people in a certain direction, I'm arguing that it's something that people who already have a certain set of political beliefs will be drawn to.

>That you think it's all about UNIX guru LARP I have to question how many Omarchy users you actually engaged with because I've mostly met humble and friendly folks. YMMV.

I mostly base myself on the tweet wall posted on Omarchy's website, because it features (or featured, back when I looked at it) a bunch of former Mac users patting themselves on the back for being such 1337 h4x0rs for... mainly using a tiling WM, as far as I can tell.

As someone who has been using Linux and tiling WMs for over two decades it's a bit baffling to look at frankly. I guess I'm a bit salty that I missed out on millions of dollars by sharing my config files.

>I've been working a lot on customization and upstreamed some patches and I have no idea what would count as poorly configured.

I obviously haven't surveyed the entire ecosystem but the base scripts shipped with the distro (which I presume are mostly DHH's work) look more like a personal dotfile compendium than something a serious distro would ship. I'm talking about stuff like this: https://github.com/omacom/omarchy/blob/1e859d37cb7fef6ac6874...

That "format-drive" function is particularly entertaining, I think.

Regarding third party code, the small handful of github projects I looked at were the most spaghetti of vibecode, e.g. https://github.com/ETroll/omarchy-airplay/blob/master/bin/om...

This may not be representative of the ecosystem at large, admittedly, but given the huge focus on vibecode/agentic stuff in the marketing, I think this is not an unexpected outcome.

Beyond that, using your own, worse package manager instead of leveraging Arch's existing capabilities (but also still using Arch's packages for some things) is just terrible design.

Is there any other reason to use this distro other than signal political alignment? I looked into it a few months ago without even being aware of the political controversy surrounding it (I was curious about the whole "Agentic Linux" angle) and all I found was a poorly configured Arch Linux install with some rather crappy bash scripts on top.

Well, ok, I think there's another use case which is to LARP as a Un*x "pro user" without actually having any idea how to properly use a shell.

I know but I don't think the risks justify using this, as demonstrated here. "It's fine when it's used correctly" except it's so hard to use correctly. It completely messes up any analysis of codepaths because you have this "time travel" footgun that can bypass everything.

Exceptions aren't great for that either, but languages that feature them at least have the tooling and support features to make them work. C has no destructors or anything to handle exception unwinding cleanly. Scheme has call/cc, but scheme has a big runtime and garbage collector to deal with the fallout.

Let the branch predictor do its job and implement error checking properly, I say. If performance matters so much that you worry about pipeline flushes and icache thrashing, surely you wouldn't be using PHP anyway.

setjmp, longjmp and gets are 3 function I would absolutely ban in any C codebase I would be responsible for. Maybe strncpy too. and strtok. C is fun.

>PHP uses longjmp()

Well off to a good start here.

My subjective impression as a web user since the late 90s is that now things break relatively rarely (I think it's the first time I have any such issue with DDG for instance) but when they do a huge chunk of the web becomes unreachable.

Back when things were more decentralized individual websites and services would have issues much more regularly because the individual software and hardware stacks weren't as robust and fault-tolerant, but then usually the problem would always be limited to a single website/service.

Well I suspect that eventually you just run into hard limitations with C's introspection facilities, or lack thereof.

I like C a lot but one of the reasons I like Rust more these days is the ability to trivially implement complex serialization schemes without a ton of ad-hoc code and boilerplate.

Sort of off-topic but I see a lot of round screens in sci-fi and especially retrofuturistic settings and that always wonder if there could be an alternate reality where they took off instead of rectangular monitors. After all for CRTs they were in some ways more optimal!

I dream of a polar-punk alternate reality where we address pixels not by (x, y) but by (r, θ).

Of course there's one big flaw with this line of thinking: even if we used circular (or elliptic) screens, we can't tile them with other circles. Circular windows would waste a lot of space. Maybe we could split in "slices" instead though?

While I don't use TikTok I often see videos from there and it's really spooky to me how aggressive and omnipresent filtering seems to have become in that community. Even mundane non-fashion non-influencer "vlog" style content is often heavily filtered and, even more scary IMO, I often don't notice immediately and only catch it if there's a small glitch in the algorithm for instance when the person moves something in front of their face. And sometimes you can tell that the difference from their real appearance is very significant.

I really wonder what's that doing to insecure teenagers with body issues.

I guess every generation experiences this type of nostalgia for the novelties of youth, but I still feel heavily privileged for having been able to live through the nineties and early 00's and experience the incredible technological explosion that occurred with microcomputers and the internet.

I started playing videogames on 8bit consoles and within a little more than a decade we had mainstream internet and games like Half Life 2. Every new generation of consoles and computers blew the previous one completely out of the water. We're also the last generation who knew what life was without having an always-online computer on ourselves at all times. Calling your friends on landlines to ask them if they wanted to hang out!

Meanwhile my current desktop computer that I use for work is about 8 years old and the benefits I'd get for upgrading would be relatively minimal. Tech is still progressing massively of course, but it feels like in many areas we've hit diminishing returns.

I dislike JavaScript and shell scripts equally, but sometimes I have to add a feature to a web page and I have to use JS, so I do, and sometimes I need to automate some un*x system task in a portable way and without heavy deps and shell scripts are the obvious solution.

What annoys me is using JavaScript and shell scripts when there are clearly superior alternatives and no clear advantage for it besides the familiarity (which, admittedly, can be a strong argument).

Shell scripts being an arcane mess is no excuse for Javascript being as clunky as it is, and vice-versa.

That's a bit of a strawman though, no? I'm definitely not worried about AI being used to write erotica or researching drugs, more about the societal effects. Knowledge is more available than ever but we also see echo chambers develop online and people effectively becoming less informed by being online and only getting fed their own biases over and over again.

I feel like AI can amplify this issue tremendously. That's my main concern really, not people making pipe bombs or writing rape fanfiction.

>the AI safety ninnies

I am one of these ninnies I guess, but isn't it rational to be a bit worried about this? When we see the deep effects that social networks have had on society (both good and bad) isn't it reasonable to feel a bit dizzy when considering the effect that such an invention will have?

Or maybe your point is just that it's going to happen regardless of whether people want it or not, in which case I think I agree, but it doesn't mean that we shouldn't think about it...

I mean even in European cities most deliveries are on motorbikes or, as you point out, increasingly on ebikes. My girlfriend routinely orders Uber Eats and Glovo and I've never ever seen one delivered by car.

Most people developing GameBoy emulators these days do it as a toy project, not a serious effort to create an ultra-accurate general purpose emulator. It's like writing a raytracer or a Pong clone.

The best GameBoy emulators like Gambatte predate Rust by almost a decade and are often written in C++. Since GameBoy emulation is pretty much a solved problem there's no strong motivation to innovate in the scene.

I've written several emulators in Rust, I'd argue that it's very well suited to this exercise, but it's also not an area where Rust shines particularly. There's no complex memory management in emulators usually since most memory buffers are just statically allocated constant-size arrays that mimic the physical RAM on the console.