remus

> But yes, totally unsurprising that a megahertz machine with 100MB+ available ram has >2500 ELO.

I think this is underselling it a bit. If you'd told this to the IBM engineers at the time I think they would have taken great interest in how the performance was achieved.

These things definitely were considered valuable contributions pre-AI, but were perhaps second fiddle to proving theorems. For example, part of the reason Erdős is so acclaimed is that he posed many interesting open questions which have kept mathematicians busy for many years.

I think Tao's point is that there's a necessary rebalancing and more weight should now be given to the non-proof stuff.

It's not like no work is happening though. Your clients are getting a more secure website. Maybe the bugs that are being patched would have taken longer to be exploited in the pre-LLM world but the bugs were still there.

I think you are looking at wealth vs. income?

I think they're just writing cautiously because they're midway through the investigation and don't want to give false confidence. Doesn't seem worth reading too much into it until they provide a more thorough write up.

> I don't want it to do "gradual typing", if it is unsure about what type something is, I want that to be a failure instead of it modeling the unsureness and not showing potential errors. We live in the age of agents, having to add an extra annotation here and there is not so much of a problem any more.

A good goal, but a hard sell for a tool when the underlying typing spec is very flexible and designed to allow for gradual typing. And of course there are all sorts of python codebases with everything from no types to very tightly specified types.

It's not quite as egregious as someone purposely choosing the key. According to the write up on the Mozilla bugtracker there was a bug in the script used for there key generation ceremony which caused the bad certificates to be generated. Obviously completely unacceptable for a CA, but makes more sense than someone plucking 4 out of the air while generating a root cert.

I think transparency in government is really important, but I don't think making ~all communication between officials public is the way. For the well behaved people it has all sorts of funny second-order effects, like people being reluctant to ask questions which might make them look silly in case they're taken out of context. For the poorly behaved they'll just work around it.

While I think there is some truth to this I think the framing is very uncharitable. The obscurity in maths comes from layers upon layers of abstraction because this is a useful way for humans to work with these complex ideas. I think we see some of this with these proofs that have been translated to lean: some of the proofs are millions of lines of lean, imagine trying to wrap your head around a codebase that is millions of lines! And just for a single proof! These impenetrable abstractions that mathematicians have created are like a mental model for a massive code base.

I think the question is then "how do I gain more confidence this code is correct?" This doesn't have to be reading through everything line by line, it can be writing more tests, or working with the LLM to dig into how certain edge cases are handled etc. There are a lot of things you can do to improve your confidence in a piece of code.

> Tesla's stock price might be detached from reality, but is their actual business in trouble?

I am not sure you can detach these two concepts. Tesla is currently valued around 3x the next top 5 auto companies combined, so to an extent you need to judge the company on these expectations. I happen to think they do have a pretty good business selling cars, and maybe a very good business selling autonomous cars, but there's so much uncertainty I think it is hard to see the current market cap as anything other than hugely inflated.

> It's not a tradeoff, it's poor design.

How so? There seems a fairly inherent tension between having things just work by default and hiding surface area for fingerprinting behind permissions pop ups. For example say you go to a maps website, how do you decide whether to share the users location with the website? Or enable webGL functionality to render some fancy layout on the screen? As a user having to enable all that stuff per-website is annoying.

Maybe there's some other way Im missing?

It's a UX trade off. For non-adversarial websites it's very convenient for users because things just work without having to go through loads of permissions pop ups for every website, but of course this then makes this info a juicy target if you're trying to fingerprint someone.

I don't think this really works out. Putting a data center in orbit is basically a question of moving lots of mass which is always going to have a large energy cost for accelerating all that mass. On the other hand someone wanting to damage that data center only has to get the mass of one decently sized bomb to the same velocity.

Unless you're talking about an adversary who has ~no launch capability then the person trying to put the data center out of reach is always going to be massively disadvantaged in terms of the physics.

> In other words I want to spend 100% of my mental capacity in the problem domain for the things AI cannot do for me, like steering, grounding, verification and not for things AI could do.

It sounds like the parent is less talking about this, and more people burning tokens while not getting useful work done.

> Similarly now we're getting AI doing math. The proofs compile but are a mess. So just make the models better at writing clean proofs and explaining what they're doing to humans. That's the end of it.

The assumption here is that the true/false of the theorem is the important outcome. While it is certainly part of it, a big part of maths is the understanding you gain from a proof. Many of the best proofs elegantly explain some aspect of the maths which was previously unclear and expand our understanding of the world.

To use a programming related example, imagine that an LLM spits out a solution to the travelling salesman problem which works in O(n) time. On the one hand that's very convenient for whatever problem you happen to be trying to solve at the time...but there's also an answer to P=NP in there! The former means your delivery drivers app works a bit faster on their busy days, the latter fundamentally shifts how humanity thinks about certain problems.

Going back to the maths, there have been theorems that were proved (by people) where the proof is broadly seen as 'unsatisfactory' in that it doesn't really expand our understanding. I assume some of these LLM proofs are a bit like that: we now know that the thing is true, but we really want to know why it's true, and how that changes our understanding.

personally I do not take such a hard line, I think a more limited form of copyright does have a place in allowing authors to profit from their work.

I think what doesn't work is a massive, blanket term applied across all works. The situation we have currently is a handful of extremely valuable works which get milked for decades and decades, and huge bulk of material which is left to rot into obscurity because it is untouchable under current copyright law.

I have experience of this with a project I run collating historic material on climbing and mountaineering: pre-internet a lot of the discourse happened in magazines, and this discourse is generally pretty hard to get your hands on. You need to track down physical copies of obscures mags that have been mouldering away in someone's attic for the last 40 years. If you talk to the authors of the content in these magazines, none of them are bothered about trying to make money of 500 words of copy they wrote 40 years ago, but because of the current massive copyright terms this stuff will not become public domain for 80-90 years in many cases, by which time I am sure finding physical copies of this material will be very difficult.

I would like to see a much more limited term by default (20 years from publication perhaps?) which still gives an author plenty of time to capitalise on their work. I think it should then be possible to renew your copyright claim (every 10 years say?) but at an increasingly high cost. £250 the first time, £500 second time, £1000 third time etc. This would mean the bulk of material would fall out of copyright on a much shorter timescale, while if you happen to have made something popular and you want to keep benefiting then you can up to a point.

How would they know for sure that some details were not part of some other training data they use? The authors may have discussed some tangential details on a forum for example, in which case you might argue that the model picked up on these details the authors assumed were benign but novel and worked out how to apply them to the problem.

Maybe it is different for you, but for me the web app is flakey at best. A bug which has been a pet peeve of mine for the last 6 months: if you leave the page open for half an hour with nothing playing, sometimes all the play buttons just stop working and you have to reload the whole page before you can play anything again.

> So if I search for A, and I get results for B instead (where B has only the slightest connection to A), it is not a bug?

It depends. There could be a bug in the system like query = query.replace(A,B), or it could be that there are no good results for A. Returning nonsense results is a bad design, but not necessarily a bug.

I guess my underlying point is that bugs are about software not conforming to some specified behaviour, and trying to specify behaviour like "i should type in a search and get great results" is too loose of a spec to be meaningful so we should avoid saying things which don't meet that criteria are bugs. To use a concrete example, perhaps the set of search results returned for a query look useless to you but are actually useful for someone else.

Compare that with something like a calculator app where part of the spec is "Must handle integer addition with for inputs i,j where -10,000 <= i <= 10,000 and -10,000 <= j <= 10,000". Then if in your calculator you do 1+1 and get 3, then that is clearly a bug per the spec.

Bugs vs. bad design is a useful distinction to make imo.

For me the idea that poor quality search results are a bug muddies the idea of what a bug is. It's like saying the colour of the paint in my living room is buggy because I don't like it. It might be an ugly colour that I don't like, but it is what it is. A bug would be if the colour doesn't match what was shown on the tin.

> If the search bears little connection to what was searched for, then it is a bug, as far as the user is concerned.

A bad product perhaps, but not a bug.

Indeed. Pre-LLM this kind of thing was a little interesting because you'd need to put a little thought and effort in whereas now it's a one-shot thing straight from an LLM. Don't be a slop proxy, as the saying goes.

Nothing so scientific. Im just having a lot of fun using the software I write with LLMs because I can easily add polish and build things I didnt have time for before.

> And that's because the definition of "good enough" is drifting towards "crappy", has been for decades, now accelerated by the AI.

Interestingly I have found the opposite in the software I write. LLMs make it so easy to get the basics in place that I spend the time on the details and polish.

"We never use AI. For anything." becomes a much weaker statement when you exclude dependencies. Your whole app could just be a thin wrapper around some third party dependency which leans heavily on AI.

> So they _are_ going to train on them, no matter how many checkboxes you tick to stop them.

It is a risk, but is it a big risk? If one of the big labs were to do this and get caught it would be suicidal due to the loss of confidence in them and the inevitable lawsuits that would follow for breach of contract. Given the AI labs are all desperately trying to paint themselves as Serious Businesses so that other Serious Businesses will pay loads of money for tokens the last thing they want is a rep for siphoning off sensitive customer data.

> ...AI companies will still do scan'n'destroy because it's just cheap.

There is also a legal element. If they kept the physical copy around after scanning the argument is that they're making copies of the book which puts them on tricky legal ground. By destroying the physical copy they can argue that there is only one version of the book that now exists solely in digital form, so this usage is better protected under fair use.

> Sure, but how is OpenAI, and Claude handling the increased usage?

By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment.

> They are handling it a lot better than Github.

Are they? Anthropic has fairly frequent issues.

There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.

> I want the model I'm paying for to be "pure".

I don't think the models are pure in any meaningful sense. The labs have some idea of what kind of output they want from the models and then they put a huge amount of effort into training the models on the right sorts of data and massaging the models afterwards to push them towards the desired output. Then at a more practical level there's the layers of filters before your prompt even hits the model (e.g. anthropic's auto-mode classifier), system prompts, response level filtering etc.

Same as usual: add tests. Over time the test suite becomes the spec that describes how the software should function. As the test suite grows you squeeze out room for undefined behaviour and bugs.