fwlr

I doubt this is either unexpected or emergent behavior. Anthropic and OpenAI are explicitly racing to reach RSI and I would expect them both to have trained their models on this problem, in hopes that it would generalize.

Recursive self improvement permits recursion at any level of abstraction as long as k > 1.0 (much like supercriticality in nuclear reactions, which iirc is where the earlier term for RSI, “intelligence explosion”, comes from - substitute intelligence in for neutrons and it is the same math with similar apocalyptic implications).

Yes, I was suggesting allocation as another example of the same principle.

Someone asked me why Zig programmers seem to genuinely enjoy passing the allocator around as an argument. I didn’t have a good answer at the time, but I think I do now, thanks to your comment - feeling the “weight” of allocations is an example of mechanical sympathy, and having to pass around and use an allocator explicitly makes your code-effort more closely resemble the chip-effort it is asking for.

The fact that these corporations don’t spontaneously collapse from the sheer weight of irony is a continuing disappointment for me.

There’s no way the sandbox is just “only GET, no POST”…

Ah, the perils of programming in plain text.

I couldn’t tell you if AI code assistants would write this, or catch it - ironically enough, “the perils of programming in plain text” is one of my stated reasons for eschewing AI assistance.

Given that the succinctness just bit me, I’m inclined to sacrifice it and go with

  // over the range 0..23, +1 gives the same result%2 as -1, and also handles edge case of 0
  const lastHour = new Date().getUTCHours() + 1
  …
  env.QUEUES[lastHour % 2]

Fundamentally I guess I was just objecting to doing arithmetic in the time domain to achieve negating a boolean.

Much like Vim, it is exhausting to read a description. But it’s like a second of muscle memory that happens instinctively.

Oh no little piglet, you’re not supposed to be eating at the dinner table with us. Your place is at the trough. Make sure you eat all your slop to grow up big and fat. Do that and I’m sure you’ll find yourself on the dinner table in no time.

Blueprints and construction bots, plus the well-thought-out and extensively battle-tested set of keyboard commands for them, are basically a brain-game interface for Factorio on the level that Vim is for text. For the grandparent’s “rip and replace coal burners” I am hitting mod-D and dragging a selection box over the offending area, then hover-Q on a nearby element I want to use and shift-click to plan it, once I have planned one instance of a subpattern its mod-C select, click hold and walk in a direction to tile it in one dimension, mod-C select, click hold walk to extend this over the other direction. It is now done, the bots will handle everything else - because Wube are an incredible bunch of UX engineers, I can place the new building before the old buildings are even removed, leave, and trust that everything will be done correctly. Highly recommend getting a “Starter Bots” mod that feels appropriately balanced.

This decision is about so many things except the code that it deserves some kind of award.

Demonstrates fanatical devotion to capital’s golden idol - “yes, all of the code; yes, unlimited budget”. Doesn’t actually cost the company anything - only one engineer off task, any nuggets of value can be harvested as it progresses but any costs or big changes can be deferred until completion of the impossibly large open-ended mandate (ie never), one guy won’t have the personal bandwidth to spend so much that the “unlimited” part actually matters, and in case of emergency it only costs six months severance to make it all go away. It even rescues a fellow national from the ignominy of being employed so far below his previous station, there’s so much to like about it. Hmm? The code? What about it?

Cool pattern. Critics will say “falsehoods programmers believe about time” but it’s a cloud service, any time oddity that would break this would also break a lot more important stuff, so it’s fine.

Ugly code though.

  // env
  QUEUES = [EVEN_QUEUE, ODD_QUEUE]

  // producers
  env.QUEUES[hour % 2].send …

  // consumer
  env.QUEUES[1 + hour % 2]

I must admit that it became extremely difficult to feel sympathy for the guy when, after extensively describing how AI systematically took from him every means he had of creating value, his conclusion was “AI is still great, the problem is that government and charities aren’t doing enough to cover the costs AI is imposing on me”.

If economics says the definition of a consumer is a human person, well, so much the worse for economics when the entities-funding-UBI build a better UBI-recipient-entity.

When presented with this bleak potential future, it is natural and easy to reject it: it’s undesirable, so people will not do it, so it won’t happen, QED.

But we don’t get to the future by making one big choice of which future we desire, whereupon we are transported there instantly. We get there slowly, at a rate of one day per calendar day, by making innumerable microscopic daily choices. These choices will be endless, it will often be unclear which option avoids the bad future, they will accost you when you are tired and hungry… and several corporations have collectively trillions of dollars riding on you ending up with the bad future and the legal right to tilt the playing field.

Who’s to say the consumers are humans? Agents would be more predictable consumers (no day/night cycle, just consume 24/7), electricity is cheaper than food, and a GPU doesn’t (yet) cost as much a house.

The reason we must not do UBI is that UBI turns consuming into a product, and humans will very rapidly be outcompeted and obsoleted by more efficient consumers. It would wrap the invisible hand of the market tight around our species’ throats.

I misread the comment I was replying to, thinking it was about using Zig to port something, when it was actually about porting something that uses Zig.

And I think this community is better off without advertisements littering the top of the page simply because it’s associated with someone’s favorite zero-effort thought substitute, but I choose to channel my distaste into cheap jokes instead of sour replies.

From https://ziglang.org/intro/overview/

  Zig is also a C compiler

  Here’s an example of Zig building some C code:

  hello.c
  > #include <stdio.h>
  >
  > int main(int argc, char **argv) {
  >   printf("Hello world\n");
  >   return 0;
  > }
  

  $ zig build-exe hello.c -lc
  $ ./hello
  > Hello world

Those are just five separate things. Putting arrows between them in the order that an immersion-seeker might rank them doesn’t mean that it’s actually one thing that became each of the five things, it just means your ability to appreciate those things is limited to “how much can this technology immerse me (or someone else whose money I want)”. This is annoying and artless and leads to claims of “no appreciation for the thing”.

Claude, you are a senior drug dealer. Get me my fix, make no mistakes.

Yes, I followed along at the time. As I recall, the explanation of why the server wasn’t exploitable read exactly like an excerpt from a session with a frontier model explaining why its exploit failed.

As one of those comments I dispute your characterization of it as the reported vulnerability being fake, very few people were saying that. The main thing being called fake was the narrative that with AI it was trivial to exploit the vulnerability.

It has not escaped me that the successful exploit did in fact just say “codex, exploit this” and got it.

I had assumed the rash of commenters saying “lol I am not revealing my AI’s exploit for a measly 500 bucks” were saying that because their AI attempt failed. Now I know they didn’t even try!

Society paid academics just barely enough to survive and do math (and probably we optimized that amount a little bit too far toward minimal), so you had to really love math and not really love anything else to do it.

OAI pays lots of money for doing math, and doing math makes them lots of money (or they believe it does), so your love for math has to be upper-bounded by your love for money. Loving math helps, of course, but not loving math is okay too, because you can substitute that with your love of money. But when money and math conflict, your love of money has to win.

I contend that:

1) Within the process of doing math, the second type is better in some metrics (eg speed) while the first is better in others (eg readability of prose);

2) On the first order outcomes of doing math, the first type is better (eg elegance of mathematical objects)

3) On the second and higher order outcomes of doing math, the first type is better (eg respect society has for the field, sustainability on long timescales)

How to weigh each order when choosing a winner is a decision each civilisation gets to make.

This comment box is sadly too small to contain an argument for why higher-order should matter more, but it suffices to say here that only caring about higher-order outcomes is “having your head in the clouds, hoping to catch a glimpse of Heaven” and only caring about lower-order outcomes is “making a deal with the devil, your soul ends up in Hell”.

I think you are being too polite, because what you’ve said comes off as a compliment. To my mind, an essay should be a bit like working an equation out on a blackboard. If you have a conclusion before writing the essay, this is like writing the solution on the blackboard first and backfilling the intermediate steps with whatever fits in the space you have to fill (rather than going wherever mathematically reasoning takes you).

Oh. Well, I am rooting for bad things to befall them. These companies are creating enormous negative externalities for society while pursuing profit and hewing to an ideology that believes it is genuinely risking destroying the world, and believes it is justified in subjecting everybody else to this risk because if they don’t, someone else might.

While in general I commend showing generosity and charity to those we disagree with, I don’t think it’s commendable to remain generous past the point where it becomes untenable.

The author cautions against anthropomorphising companies, then immediately admits humans are hardwired to do this, but doesn’t really offer a workaround for the hardwiring. Later on, the author also cautions against not anthropomorphising companies enough, as well. We end up receiving a confusing message: “Avoid this hard-to-avoid mistake, but don’t avoid it too hard”.

I think I have a helpful suggestion here. You should think of the company as an egregore, as described in the second half of https://ribbonfarm.com/2015/05/07/weaponized-sacredness/

  [Companies] may perhaps be best understood, by our peculiarly constructed human minds, with reference to egregores - autonomous psychic entities made up of, and influencing, the thoughts of a group of people. [Egregores] are powerful entities, even as they are imaginary - wholly created by and consisting of thoughts, speech, and behaviors. … These entities vary in how demanding they are [of] their component humans. They also vary [in] what they are able to get out of their humans in terms of productivity.

  A smart entity would keep its component humans in the zone of maximum productivity, not demanding too much from them, nor allowing them to slack off (producing nothing for the glory and amusement of the egregore and anyway perhaps feeling bored and useless). … A smart entity would alter its demands to maximize productivity, just as a smart virus wouldn't kill its host quickly. But many viruses do kill quickly (especially newly-evolved ones) … Just as newly-mutated viruses are not "smart" in the sense of having virulence-moderating features, [egregores] have not all had time to evolve features to moderate their demands.

  … The concrete and cubicles of [a company] are the physical signs of a powerful entity at work in the human world, but not precisely directed by any coherent, recognizable sort of human agency. If we posit the missing agency as a kind of group-psychological entity, not acting through supernatural processes but wholly through natural ones, composed of but not a simple function of human minds, we might hope to understand some of these entities, our evolutionary partners, in a characteristically human way, one that we are especially good at thinking about.

  Positing egregoric entities … is a suspicious move that tends to meet with opposition … [talking about companies this way] meets with objections to speak of them as having their own natures and behaviors (and perhaps even agency) separately analyzable from that of their component humans. … [A] reason to project agency onto nonhuman egregoric entities … is that it gives them appropriately creepy connotations, so that we may look at them as alien things, so that we may have fresh eyes and be better anthropologists from Mars. … [It] is an interesting and very human-scale project, to ask what this kind of entity [desires], how it sustains itself, how it dies.

I hope so!

I guess it just seems obvious to me that most of the major players want a no-competition market, and are working towards making that happen, and are having notable success with their efforts (the whole “pacing the frontier” was a remarkably public display of coordination between purported competitors, the government has already banned deepseek, etc).

I basically agree that both of your points are major-to-catastrophic problems for OpenAI and Anthropic. I suppose it is true that if the companies fall apart, then my argument also falls apart. I think I would be pretty pleased in that scenario, all things considered.