trott

> The AlphaFold3 analysis (the AI contribution) literally accounts for a few panels in a supplementary figure - it didn't even help guide their choice of small molecule inhibitors since those were already known.

(Disclaimer: I'm the author of a competing approach)

Searching for new small-molecule inhibitors requires going through millions of novel compounds. But AlphaFold3 was evaluated on a dataset that tends to be repetitive: https://olegtrott.substack.com/p/are-alphafolds-new-results-...

Regarding point number 11 (AlphaFold3 vs Vina, Gnina, etc.), see my rebuttal here (I'm the author of Vina): https://olegtrott.substack.com/p/are-alphafolds-new-results-...

Gnina is Vina with its results re-scored by a NN, so the exact same concerns apply.

I'm very optimistic about AI, for the record. It's just that in this particular case, the comparison was flawed. It's the old regurgitation vs generalization confusion: We need a method that generalizes to completely novel drug candidates, but the evaluation was done on a dataset that tends to be repetitive.

> According to Stack Overflow developer survey [0] Rust is at 12.5%, ... So definitely not niche.

The annual survey is very popular in the Rust community. Its results are often used for advocacy. Participation by Rust developers is very high. So what you have is a classic case of a selection bias.

They started with N >= 120x3 tasks, and gave each task to 4-9 humans. Then they kept only those 120x3 tasks that at least 2 humans had solved.

> Both "unchecked" Ada and "unsafe" Rust

But the `Conversion` function isn't using `Unchecked_*`. That's the point of the article. The type safety hole is in "safe" Ada.

> a mistake of the 1983 Ada standard ... which has been removed

The article was written in 2011, and the trick still seems to work in a 2024 version of GNAT.

> but Ada's real strengths lie elsewhere. Its strong typing,

Ada is not actually type-safe: https://www.enyo.de/fw/notes/ada-type-safety.html

> it will finally be fixed relatively soon

2015: "The work needed to close this has not yet landed. It's in the queue though, once we finish up rust-lang/rfcs#1214."

Yep, and even without dynamic memory management, Ada is not type-safe: https://www.enyo.de/fw/notes/ada-type-safety.html

Rust also has soundness holes, by the way. This one is almost 10 years old: https://github.com/rust-lang/rust/issues/25860

Another way to look at this is that there are 12,290 bits of information in choosing 817 samples from 10,000,000.

How do Maxima and SymPy compare in terms of capability, features and speed (native, not WASM)?

This is the paper: https://arxiv.org/abs/2411.02272

They won the 1st paper award: https://arcprize.org/2024-results

In their approach, the LLM generates inputs (images to be transformed) and solutions (Python programs that do the image transformations). The output images are created by applying the programs to the inputs.

So there's a constraint on the synthetic data here that keeps it honest -- the Python interpreter.

> This has me curious about ARC-AGI

In the o3 announcement video, the president of ARC Prize said they'd be partnering with OpenAI to develop the next benchmark.

> mechanical turking a training set, fine tuning their model

You don't need mechanical turking here. You can use an LLM to generate a lot more data that's similar to the official training data, and then you can train on that. It sounds like "pulling yourself up by your bootstraps", but isn't. An approach to do this has been published, and it seems to be scaling very well with the amount of such generated training data (They won the 1st paper award)

> Terra programs use the same LLVM backend that Apple uses for its C compilers.

Can it use anything else (as an option), e.g. Lua? That would be useful during development/debugging thanks to faster iteration and memory safety.

Not sure. But I think this follows logically from the definition of intelligence he is using. Also, see II.2.2 in the paper.

> That's fine if you want to measure sample efficiency, but ARC-AGI is supposed to measure progress towards AGI.

On the Measure of Intelligence defines intelligence as skill-acquisition efficiency, I believe, where efficiency is with respect to whatever is the limiting factor. For each ARC task, the primary limiting factor is the number of samples in it. And the skill here is your ability to convert inputs into the correct outputs. In other words, in this context, intelligence is sample-efficiency, as I see it.

> realise that you're wrong, accept it, and move on

What do you think about limiting the submission size? Kaggle does this sometimes.

With a limit like 0.1-1MB (compressed), you are basically saying: "Give me sample-efficient learning algorithms, not pretrained models."

(Spoiler alert)

In https://arcprize.org/play?task=79369cc6 , the yellow 3x3 square shows you the pink pattern to look for, while allowing rotations (and ignoring the fact that the pattern may be next to other patterns)

Mike and François,

Compute is limited during inference, and this naturally limits brute-force program search.

But this doesn't prevent one from creating a huge ARC-like dataset ahead of time, like BARC did (but bigger), and training a correspondingly huge NN on it.

Placing a limit on the submission size could foil this kind of brute-force approach though. I wonder if you are considering this for 2025?

Congrats, François, and good luck!

Q: The ARC Prize blog mentions that you plan to make ARC harder for machines and easier for humans. I'm curious if it will be adapted to resist scaling the training dataset (Like what BARC did -- see my other comment here)? As it stands today, I feel like the easiest approach to solving it would be BARC x10 or so, rather than algorithmic inventions.

> That's not general intelligence, that's gaming.

In fairness, their approach is non-trivial. Simply asking GPT-4o to fantasize more examples wouldn't have worked very well. Instead, they have it fantasize inputs and programs, and then run the programs on the inputs to compute the outputs.

I think it's a great contribution (although I'm surprised they didn't try making an even bigger dataset -- perhaps they ran out of time or funding)