kragen

At least this slice

          h = X
          for i in range(nlayers):
              z = h @ P[2 * i] + P[2 * i + 1]
              # ...
              h = np.maximum(z, 0) if i < nlayers - 1 else z    # ReLU; linear output
              # ...

          logit = h[:, 0]

      # ...
      # accuracy of the last forward pass, i.e. before the final update
      return ((logit > 0) == (y > 0.5)).mean()

looks like a perfectly orthodox evaluation of a multilayer perceptron, with weights and biases in P, and that's what it's evaluating the accuracy on. I dumped out P for one of the [8, 64, 1] networks, and it does indeed look like about 650 parameters. I was going to post it here, but printed out as text it's like three pages long, so I won't.

I'm guessing that Nicholas Carlini could give you a much more accurate answer by reporting its Elo with a move limit of 50 milliseconds.

That's a good question, and I don't know the answer, but I can try to guess.

An 80386 is about 10–13 Dhrystone MIPS, while a 200MHz Pentium Pro is about 90–450, according to https://netlib.org/performance/html/dhrystone.data.col0.html. So we're talking about a further slowdown on the order of 20×. I don't know how to estimate how much that would lower its Elo.

The article says:

> By making a number of adjustments to the algorithm and tuning it to the Pentium, we can increase its Elo to 2846 ± 35 at 1 second per move, 3109 ± 79 at 10 seconds per move, and 3180 ± 80 at 30 seconds per move.

So making it 30× faster would raise its 1-second Elo by about 334 points, and ln(30) is 3.4, so that's about 100 points per factor of e. ln(10) is 2.3, so we'd interpolate a prediction of 230 points gained at 10× faster, reasonably close to the 263 points reported.

If we extrapolate that in the other direction, ln(20) is 3.0, so on the 80386 we'd expect a loss of about 300 Elo points, down from 2846 to 2550 or so. Still enough to wipe the floor with Crafty, but low enough that Kasparov would usually beat it.

But we don't really know if that extrapolation is valid.

I don't think there's a meaningful difference between the Pentium and the 80386 except speed. You might have to change the compiler options a bit.

And, with more work, that's true of computers in general.

I don't know, this might be year #81 of the 82 years in which humans create software. Or it might be year #81 of 82 billion years. Anybody who tells you they know what's going to happen is full of shit.

I think they're terrified of using the word "microwave" because it might scare the ignorant masses.

I'm probably a worse programmer than I was before I started using frontier AI models, in the sense that I probably have a harder time reading, writing, and debugging code. And I doubt that my knowledge of the AI models themselves will have lasting practical value. But I know substantially more about UEFI, PCIDs, seL4, WebAssembly, GLSL, Service Workers, CORS, server-sent events, CSS, content-security policies, memory-hard PBKDFs, Android, numerical integration, Verilog, V8 security, Linux process namespaces, hypervisors with VT- x, multitouch user interface design, radio communications theory, power electronics, physical-layer communications protocols, NTSC video signals, TLA+, bounded model checking, generative testing, theorem provers, FPGAs, Linux system call overhead, Unix mailbox formats, physical modeling synthesis, damped resonance, KaTeX, geology, dishwashing detergent, microbiology, Cyclone, thermodynamics, X-Windows, localStorage, AVX, and IndexedDB, for example.

Using microwave radar to watch your users' tongues is amazing. I had no idea that was possible.

Here is the code Opus 5.5 is claiming fits a 641-parameter MLP to parity of 8 inputs:

  # Fits small MLPs to 8-bit parity (the full 256-row truth table) with
  # full-batch AdamW, written out in numpy (no PyTorch, so it won't match a
  # PyTorch run bit for bit), and reports for each shape how many of 20 seeds
  # reach 100% accuracy and the median accuracy. Needs Python 3.9+ and numpy.
  import itertools

  import numpy as np

  X = np.array(list(itertools.product([0, 1], repeat=8)), float)  # all 256 inputs
  y = X.sum(1) % 2


  def nparams(widths: list[int]) -> int:
      """Parameter count of a dense MLP with these layer widths, biases included."""
      return sum(a * b + b for a, b in zip(widths, widths[1:]))


  def train(
      widths: list[int],
      seed: int,
      steps: int = 4000,
      lr: float = 1e-2,
      wd: float = 1e-4,
  ) -> float:
      """Full-batch AdamW on all 256 inputs; returns the final training accuracy."""
      rng = np.random.default_rng(seed)
      P = []    # parameters: W0, b0, W1, b1, ...
      for a, b in zip(widths, widths[1:]):
          bound = 1 / np.sqrt(a)    # same as PyTorch nn.Linear's default init
          P += [rng.uniform(-bound, bound, (a, b)), rng.uniform(-bound, bound, b)]

      m = [np.zeros_like(p) for p in P]    # Adam first moments
      v = [np.zeros_like(p) for p in P]    # Adam second moments
      nlayers = len(widths) - 1
      for step in range(1, steps + 1):
          hs = [X]    # hs[i] is the input to layer i
          zs = []     # pre-activations
          h = X
          for i in range(nlayers):
              z = h @ P[2 * i] + P[2 * i + 1]
              zs.append(z)
              h = np.maximum(z, 0) if i < nlayers - 1 else z    # ReLU; linear output
              hs.append(h)

          logit = h[:, 0]
          prob = 0.5 * (1 + np.tanh(logit / 2))    # sigmoid without overflow
          g = ((prob - y) / len(y))[:, None]    # d(mean cross-entropy)/d(logit)
          G = [None] * len(P)    # gradients, parallel to P
          for i in reversed(range(nlayers)):
              G[2 * i] = hs[i].T @ g
              G[2 * i + 1] = g.sum(0)
              if i > 0:
                  g = (g @ P[2 * i].T) * (zs[i - 1] > 0)

          # AdamW, using PyTorch's default values for betas (0.9, 0.999) and eps (1e-8)
          for k in range(len(P)):
              P[k] -= lr * wd * P[k]    # decoupled weight decay
              m[k] = 0.9 * m[k] + 0.1 * G[k]
              v[k] = 0.999 * v[k] + 0.001 * G[k] ** 2
              P[k] -= lr * (m[k] / (1 - 0.9 ** step)) / (
                  np.sqrt(v[k] / (1 - 0.999 ** step)) + 1e-8)

      # accuracy of the last forward pass, i.e. before the final update
      return ((logit > 0) == (y > 0.5)).mean()


  for w in [[8, 1, 2, 2, 2, 1], [8, 8, 4, 2, 1], [8, 8, 1], [8, 16, 1], [8, 64, 1]]:
      accs = [train(w, seed) for seed in range(20)]
      print(w, nparams(w), "params; seeds at 100%:", sum(a == 1 for a in accs),
            "/ 20; median acc", np.median(accs))

I might not know enough about ANNs to spot if it's cheating on some way, but it looks legit. It does indeed output that the [8, 64, 1] configuration gets 100% accuracy on all 20 seeds.

It looks like Astra picked the "random task", and the task it picked was 8-bit parity, which is easy to design a circuit for that computes the answer perfectly, but sort of maximally difficult to fit with any kind of gradient descent. It's a great choice for what you asked, which was to find a case where AdamW doesn't come close to the theoretical optimum fit. But I'm less convinced that it tells us much about Astra's capabilities at handcrafting weights in general.

8-bit parity is a^b^c^d^e^f^g^h, which you can compute with minimal depth as ((a^b)^(c^d))^((e^f)^(g^h)). You can compute x^y as x&~y | y&~x. With ReLU activation, i | j is ReLU(i + j): weights of 1, bias of 0. And i & ~j is ReLU(i - j): weights of 1 and -1, still bias of 0. So a two-input XOR is three neurons in two layers, and the tree of 7 binary XORs is 21 neurons in 6 hidden layers. I don't know why Astra needed 433 parameters. Only 42 nonzero parameters are needed. Maybe the other 391 parameters are zeros that the Transformer model demands to be present.

With an RNN you can compute the parity of a string of any length with only three neurons.

Actually, I handed that to Claude Opus 5.5 to check, and it sensibly pointed out that x&~y is mutually exclusive with y&~x, so XOR doesn't even need the third neuron, except in the last layer; you can just fold the addition into the weights of the next layer, which means you only need 4 layers and 15 neurons. But about half of those neurons need 4 nonzero input weights instead of 2, so you still need 42 nonzero weights. It's a good thing you don't need those OR neurons, too, because my definition is wrong: ReLU(1+1) = 2.

Also it explained how to design a 28-parameter net that I don't quite understand, and also a single-hidden-layer network with 8 ReLUs, which I also don't understand, but it needs more parameters: $\text{parity}=s+\sum_{k=1}^{7}2(-1)^k\,\mathrm{ReLU}(s-k)$ where I think $s=\sum_i x_i$.

Somewhat more surprisingly, it also said it didn't have any trouble fitting a 641-parameter network to this problem with AdamW.

Xerox made quite a bit of money selling laser printers for desktop publishing, too, a lot more than they ever spent on PARC.

I don't think it's accurate that "almost all the money in personal computing was made in mobile". Quite a bit of money was made in personal computing before the iPhone, and quite a bit of money has been made in non-mobile computing since the iPhone.

I also don't think it's accurate that mobile "didn't use any of these ideas". Android's UI is not quite as close to Smalltalk-78 as what I'm using right now, but you still have pointers, scrollbars, text selections, proportional fonts, vertical-line cursors, buttons, menus, object-oriented programming, sometimes a bytecode interpreter, and even windows. You just don't have as much freedom with where you position the windows. Android's UI is still a lot closer to Smalltalk-78 than it is to, for example, acme, GENESYS, the ITS HACTRN, FORTH, or PDP-11 Unix.

They're dogfooding the way Xerox PARC was dogfooding in the 01970s.

By giving hundreds of workers each a powerful desk-sized Alto minicomputer for their own needs, with a high-end graphical display, they gained insight into what desktop personal computing would look like 15 years later, enough insight to shape it profoundly. It's been 55 years, and although multitouch screens are proliferating, I'm still typing this on a keyboard hooked up to a framebuffer-driven display with a mouse hooked up to it, displaying hypertext, festooned with windows, icons, and a mouse that moves an onscreen pointer at the same 135° angle they settled on 55 years ago, positioning a vertical line to indicate the insertion point, with scrollbars. The display has twice as many pixels, the pixels are color, the mouse pointer is white rather than black, the screen is a flat LCD, the scrollbars are on the right, and the cursor blinks, but it's very recognizably descended from https://smalltalkzoo.computerhistory.org/HOPL-St78.html rather than, for example, ITS or GENESYS.

Only people with the luxury of unlimited zero-cost CPU cycles had a chance of seeing what the future looked like, and that allowed them to find out what worked and what didn't work in that environment. Nowadays, the regular workers who don't have the luxury of unlimited zero-cost tokens are manually reviewing voluminous AI code, because, even though that's a lot of slow work, they're still much faster programming that way than they were without it. Anthropic, instead, is doing insane things like eliciting a billion lines of Lean proofs to show that existing open-source libraries can't fail their OSS-FUZZ tests — and fixing all the bugs that effort turned up, in the process.

In the US, the IRS doesn't see it that way, at least when it comes to fringe benefits as a part of employee compensation. https://www.irs.gov/businesses/small-businesses-self-employe... says, "In general, the amount the employer must include is the amount by which the fair market value of the benefits is more than the sum of what the employee paid for it plus any amount that the law excludes."

As an example, as I understand it, if a hotel chain allows an employee to stay in one of their hotels for free when they go on vacation, the tax value of that benefit is what they would have charged someone for that hotel room, not how much it increased the hotel's costs to have one more hotel room full — which is generally almost nothing.

Of course, the IRS does agree with you when it comes to expenditures that are for the employer's benefit, rather than the employee's. If it didn't, almost all companies would have negative profits from the IRS's point of view.

I was commenting on why jonhohle might have been "finding more than a few issues with AFL at a FAANG in code written by engineers with the highest engineering title". I don't know the context he was finding them in, because he didn't think it was relevant enough to mention, and my whole claim here is that you have to take that context into account.

Historically, Debian didn't ship proprietary wireless firmware in the installer (because it was proprietary software, and Debian doesn't do proprietary), so you couldn't install it on most laptops without plugging them into wired Ethernet to download the wireless firmware packages from non-free.

They changed that a few years ago, but plausibly the underlying prioritization of end-user experience is still lower than Ubuntu's degree of user focus.

I still wish Mark would rename his company "Ubuntu". He's a great guy, the company does great work, but I keep getting undeserved credit for it because we wouldn't sell him the domain name.

Sometimes writing shitty code is a result of competence, not incompetence. I've seen much better programmers than I ever will be writing terrible code because it was the fastest way to get a prototype running and find out whether the problem was worth writing good code for.

Yes, but troff/groff is not the only descendant of RUNOFF, and apparently they're using a different one.

I think you can make the metadata files with roughly a five-line Perl script.

I have the impression that an HTML -> epub workflow mostly consists of making a zipfile?

Many people will prefer Markdown or Commonmark for markup.

The UNIX project was funded by management as a word processing system initially, yes.

troff is no longer fashionable, and it's not what you want for top-quality typesetting, but I don't think of it as "out there". My non-technical mother was using an HDOS program called TEXT.ABS, which is similar to a limited nroff, to format the articles she wrote in the 01980s. It wasn't as easy to use as something like MacWrite, but it was a lot easier than a typewriter. The text editor she used was called PIE.ABS, and from my point of view, the usability distance from a typewriter to PIE was bigger than from PIE to MacWrite.

She didn't have much choice of fonts on the pre-Graftrax Epson MX-80 she used. I think it was Pica or Elite.

Typesetting is certainly very satisfying, but it's not essential to writing.

No, "notepad" is a good name in a way that "text editor" is not, and Notepad has the advantage that it has been present in Microsoft Windows since two years before the first version of Windows, so nobody is going to call a different Microsoft Windows text editor "Notepad".

You might object that, when it was first released, before Windows, it was not the first text editor on the operating system, which was MS-DOS, and so "Notepad" might have already been too generic even then. Microsoft would have agreed with you, because that version was called "Multi-Tool Notepad".

I have told you before not to contact me, although you have flagged my previous comments to conceal this fact. This is, I think, the seventh time. I have lost count. Do not "forget" again.

There's no point in attacking the security of a company that doesn't try to provide any.

The things that Northflank and Railway were apparently breached to steal aren't just things that hyperscalers would have protected worse instead of better; they're AI API keys, things that only existed because people were using hyperscalers instead of running inference on their own hardware (or on Amazon AWS EC2 GPU instances).

Self has inheritance, using it far more pervasively than Smalltalk, and the original VB isn't OO.

It is important for beginner-friendliness that it be possible to ask for help.

"Hey, how do I insert one file into another in text editor?"

"Which text editor?"

"Text editor."

"Sounds like a personal problem."

Yeah, ae is awesome and inspirational, but it's missing some crucial features. It sounds like maybe he did eventually add a single level of undo, though?

Atto seems to be a little smaller than that version of ae, though.

Right. Vim is an Emacs in this sense, but mg, atto, joe, readline, etc., aren't.

In GNU Emacs, try running M-: (dot) RET.

I think this alias was an aid to porting Lisp from some other Emacs variant in the 80s that used "dot" instead of "point". Never figured out which. It isn't mg because mg doesn't run Lisp.