parsimo2010

I also look at the ECI from Epoch.ai as well, but they haven’t evaluated Carbon yet. Probably because the model isn’t widely available yet.

We’re literally talking about the same thing, but ascribing different motivations.

I do think the test a new model and realize their index is broken- the new model isn’t on top, so they change the weights to make sure it’s on top.

The difference is that you seem to think they are doing so out of some innocent feeling that the new model is actually better, while I think they are doing so because they were paid to do so. I think my interpretation is more likely, but that’s just me.

Just so everyone is aware of the "coincidences" that happen with this index:

- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.

- Today the newest model announced from a big lab tops the charts.

I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.

It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.

Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.

If Borcherds accepts TAs then they work for him. The point was that he can do practically whatever he wants and doesn't have to worry about it harming him.

Thanks for explaining to me how the academic system I work in functions. Just like most managers don't literally pay their employees, the company does. The university pays TAs through funds from their various revenue streams, including endowment, tuition, grants to the university, and through indirect costs collected from research grants to researchers, probably including Borcherds own grants. So really a lot of different things pay the TAs, the university is just the clearing house.

Again, I’m not making a moral statement. But you yourself basically just said grading is a menial task that is below the station of a professor. (BTW, I teach at a small school and do all my own grading each semester for between 45-80 students depending on the courses I’m teaching that semester)

So if grading is part of the development process for a graduate student, then go ahead and continue. But that’s not the claim in the article. And if grading doesn’t serve a purpose anymore, then stop doing it. You can pay your assistants to do nothing for 20 hrs/week, or you can find literally anything else for them to do.

I don’t know Borcherds, so this isn’t intended to sound mean to him, but he could pay his TAs or RAs to do hula hoops naked in public for 40 hours a week and it wouldn’t affect him. He’s a high profile mathematician and the worst thing is that next year’s grant money is a little less and he can’t pay for as much naked hula hooping spectacle.

Entire math departments at multiple lower profile schools will close their doors before he has any actual concern about his future as an academic.

But your point is correct for other professors, much less secure in their positions, who have similar problems with AI and actually do have to concern themselves about using their TAs and RAs productively.

The irony here is perhaps missed by many. He didn’t grade homework in the first place. He paid assistants to grade. This is common practice among tenured professors with grant money to spend and graduate students beholden to them (I’m not making a moral statement, just noting this is common practice).

Aside from the irony that Borcherds probably hasn’t graded a homework set in a while, instead let’s examine the purpose of homework. It helps you learn. You can read a chapter and think you understand the material, but doing homework makes you engage more with the material. Recently we’ve used the external motivator of course points to encourage this good habit. But this turns the relationship between teacher and student into a transaction- you do a problem, you get a point. And most young students really don’t care beyond that transaction. They are going to college to increase their earning potential, and that’s about it.

So if we’re not going to rethink the purely transactional relationship schools have with most students, then you probably should stop giving points and grading homework. You have to figure out a new transactional structure that can’t have your tuition made irrelevant by a $20 subscription. That’s possible to figure out. A lot of people have pointed out various equity risks but it’s really just a matter of balancing the AI risk with the equity risk at the weighting acceptable to your institution.

You could also attempt the much more challenging prospect of turning the transactional nature of college into some other relationship which fosters a student’s intrinsic motivation of learning for the sake of learning, but I bet your institution goes bankrupt in the process. So really you should just stop grading homework and put 100% of the grade weight into proctored assessments with some sort of firewall against AI tools. Anyone that faces an equity disadvantage because of this was probably also disadvantaged by the fact that you let rich kids with premium AI subscriptions get better grades on their homework.

A sizable part of the country thinks it’s okay to regulate firearms. Under your logic, people should be responsible for who they shoot, don’t blame the gun when a human pulled the trigger. While gun control advocates have that pesky second amendment hampering their efforts, AI/LLM systems have no such protection.

Regulatory capture is certainly something to be wary of, and obviously Anthropic is only going to be in favor of regulation that helps them in some way. But some form of regulation is coming. There is nothing that can stop it, unless you somehow make more money than these guys do and fund a massive counter-lobby. Spoiler alert: the only guys that have more money than Anthropic and OpenAI have all invested in them.

While I understand your thinking, he rolled Twitter into SpaceX. He absolutely uses it as a financial vehicle, regardless of how legitimate the space business is.

Not sure if this has been noticed, but the pricing is “$299 (U.S.) or $49.83 (U.S.) per month for six months,” which means paying in installments is 2 cents cheaper than paying the whole price upfront. 49.83*6=298.98. So you’re not just getting a zero interest loan, you’re getting a discount.

I’m not advocating for the practice of chaining kids to a desk…

I’m trying to point out that a tool might be helpful for you to practice, but if it isn’t used then the tool can’t help. Maybe if you were forced to use the tool then you could see the benefit, but as you noticed, the act of “forcing” something may feel undesirable from a moral perspective.

I think this is what makes this so ridiculous. They put “room temperature” in the title and while it might not have been intentionally nefarious, it definitely has the effect that most readers assume it is about superconductors. There is nothing special about a room temperature semi conductor, your CPU, RAM, and storage drives have worked at room temperature for decades…

And if not for that word, the article would be completely unexciting. You found that you can make magnets out of multiple materials? We already knew that. If the materials aren’t abundant/cheap and easier to manufacture then this isn’t a story. And neither of those claims are tested or verified in this. So this isn’t a big deal.

A very honest take. They tried something, it showed a very small improvement, and they acknowledged it. Acknowledging that the cause appears to be a lack of engagement agrees with my own experience. You can give students great tools but that doesn’t make them interested in math. Khanmigo might be a good tutor but it won’t help unless you’re chaining kids to a desk and not letting them leave until they finish some quota of problems.

You could always curl the install script, and modify it to run the virus scan in between the build and install steps.

One reason is that the llama.cpp team (GGML) has strict requirements that a human must understand the code they are contributing. If a project is fully vibe coded they can’t contribute. So a lot of projects where an AI went and coded a bunch of custom kernels to increase speed are left to their own devices.

I think this is a fine behavior. We can have upstream purists that are strict gatekeepers but don’t get in the way of downstream forks. Debian has some this in the Linux landscape for a long time, and it has enabled Ubuntu, Mint, etc. to flourish without compromising themselves.

You can absolutely run an agent as a limited-privilege user that only has write privileges for specific files and only has execute privileges for certain files. If it is running as a limited-privilege user it can work on code in it's own copy of the repo and make commits and send pull requests, but it can't do the merge. The problem is that nobody wants to go through the effort to set up all these permissions and nobody wants to take the time to review everything and perform all the manual actions.

Agreed- this is the same problem we have with trusted admins or devs who have elevated privileges on their networks. We have to trust that the admins won't use their power to steal company secrets or misuse company resources. If you don't trust the admins, then they can't fix things on your network and there is no point in having them.

If you want an agent to act on its own, like pushing to a git repo, managing dependencies, building and testing, etc., then you have to trust it as much as any other privileged user.

If you don't want to trust it, then you're just forcing yourself into the reverse centaur role, where the agent edits some code, but then has to stop and ask you to push the changes or build the software again and run the unit tests.

I suppose there is a principled way of doing things like "I trust you do do basic commits but I will handle merge conflicts" and "you can build modules in this directory but you can't build outside of it" but this is just a lot of effort that most orgs won't bother with.

I mean that is a conclusion that I agree with, but I’d also like to see if I can do better if given more time on a bigger screen.

Not bad. I’d appreciate a much longer time limit. Got really rushed at the end. But this was crazy hard on my phone, I was straight guessing most of the time. I think I could be a little better with more time on a desktop screen.

Another idea would be games that were all pictures of a single category- like 10 AI and 10 real pictures of basketballs, or food on plates, etc.

It's basically a cinnamon flavoring added to regular oreos. They were aiming for the taste of horchata.

I 100% agree with you. It looks like multiple people (including me) were typing comments about how good the Selena Gomez Oreos were at the same time as you.

The post seems to have found a strange niche of HN, or maybe the Selena Gomez Oreos had wider appeal than I thought.

I really liked Selena Gomez's Oreos, but I would prefer that Oreo brought them back as "horchata flavored" or "cinnamon cream flavored" so I didn't have awkward conversations every time I accidentally said, "I like Selena Gomez flavor" and people would ask how I knew what Selena Gomez tasted like.

Honest question/suggestion for the HN audience- Since Qwen released the weights for Qwen3.8 2.4T-A95B and we already have the staring point of Qwen3.6 35B-A3B, couldn't someone distill the bigger model and make a "pseudo" Qwen3.8 35B-A3B? Sure, it wouldn't be an official Qwen release but couldn't someone improve on Qwen 3.6 and get the thing everyone is asking for?

I am calling this a suggestion for the audience because I don't have the will/resources to do this.

Actually, yes. I just didn't know about Grok's release because they aren't on the front page of HN.

It still matters as a point of comparison until other providers come online. If the consensus price from other providers is much different that can be compared then. But for now we have $0.435 / $0.87 for v4 Pro 0813 (with increase announced but we don't know the new pricing), and $2 / $6 for Qwen3.8-max. So until we get other data points that is what we have to look at.

The timing looks like they are trying to take the wind out of Qwen's sails by releasing this on the same day that Qwen released the weights of Qwen3.8-max. Or maybe it's coincidence...

For comparison I looked at Qwen's claimed benchmarks for Qwen3.8-max (https://qwen.ai/blog?id=qwen3.8). Assuming each published set of benchmarks is believable, it looks like v4 Pro 0813 is better on average but overall performance is comparable. Pro 0813 is much cheaper. If you don't need vision capabilities then you don't have much reason to use Qwen3.8-max.

- 43.6 on HLE (Presumably without tools). Pro 0813 is a little worse.

- 86.6 on Terminal Bench 2.1. Pro 0813 is better.

- 55.9 on NL2Repo. Pro 0813 is better.

- 27 on Agent's Last Exam. Pro 0813 is a little worse.

- 72.5 on Toolathon-Verified. Pro 0813 is better.

- 56.6 on DeepSWE 1.1. If the DeepSWE listed for Pro 0813 is the same version, then Pro is better.

- 27.3 on AutomationBench. If the AutomationBench (Public) listed for Pro 0813 is the same, then Pro is better.

I guess we do need to wait to see if the upcoming DS pricing increase is enough to change the value proposition. As it is now, they could double or triple prices and it still would be a better value to use DS. I bet they know that.

This is basically my thought- my school is considering all sorts of things to combat AI cheating, and most of them scale poorly. You can implement them and increase your reputation, but your tuition will have to increase to pay for a lower student/teacher ratio.

So we’re probably going to see a split- mass production schools that still act as job training for Industrial Revolution era jobs will keep doing what they are doing. Schools that want to keep their academic prestige will implement these inefficient safeguards and raise prices accordingly. Employers will distinguish between these two types of diplomas when there are enough options out there that are combatting cheating- much like in-person schools are generally valued higher than online educations, or how certain schools (MIT, Stanford) are valued higher then others. This might lead us to the transformation of higher education many have been anticipating, because the current system cannot remain as-is. We all thought technology was the thing causing the upheaval but AI has really accelerated things.

This has been obvious to everyone who knows that the key feature to the devs of these browsers is “-based.” None of the chromium-based browsers are anything other than UI tweaks and certain extensions being included by default. None of them rewrite significant parts of the browser engine, because that would be too much extra work.

The only projects which have a hope of maintaining MV2 compatibility are those who are a hard fork of an earlier version of chromium or a totally different engine.

To my point: The article mentions that Opera still says they will maintain MV2 for as long as it is “technically reasonable.” Mark my works, Google will find a way to break Opera’s MV2 patches that makes it too hard to maintain.

Out of all the companies that signed the open letter that Dario references, Google and OpenAI feel like they did it to poke at their competitor. I probably should have taken shots at Microsoft and Meta as well for not really supporting openness, but they don't feel like serious competition in the LLM space at the moment.

Such a cop-out. Dario, you got in the news because you were trying to say that Moonshot did something wrong by distilling Claude. You got in the news because you were trying to effectively make a "rules for thee but not for me" when you try to claim that you can train on whatever pirated works without any permission from the creators, but when someone uses your "work" to train without permission then all of a sudden it's a moral injustice. You can't have both.

Saying, "I'm not actually against open-weights, I'm against distillation" isn't addressing what made people mad. You're still trying to do some "rules for thee but not for me" nonsense and hiding behind some technicality. Trying to get the US government on your side to hold back your Chinese competition. If you had wanted the US government to support you, you should have let them make autonomous killer robots with Claude brains. They aren't going to help you, you didn't help them.

Just to be clear, I think that it is possible that literally everyone involved in this is full of crap and nobody is good. Dario and Anthropic are full of crap, for the reasons previously stated. The US government is full of lots of crap and should not be trying to make autonomous killer robots (not ever, but especially not when the bar for a "good" AI is knowing how many Rs are in strawberry or whether you should drive to a car wash). OpenAI is full of crap by signing some support for open weights models and they haven't touched open weights in a year (GPT-OSS released on Aug 5 so basically a year with no news). Google is less full of crap about the open weights stuff because of Gemma 4, but they are full of crap for a zillion other things I can't exactly feel good about them. So everyone sucks.

So cheers to Moonshot and Qwen and whoever else. Distill as much as you can and give us cheaper AI. I have the sneaking suspicion that a bunch of my tax money went to OpenAI and Anthropic in some shady way or another, and I want it back. I'll take it in the form of an open weights model being distilled from the fat cat models.