hintymad

indeed!

> There are very good open-weight Chinese models

All the more reasons for us to figure out how to keep getting open models, or we are subject to the will of the the few frontier labs.

How so? I thought Anthropic was very capital efficient. They still have more than 100B cash, and their inference margin is 80%.

Not consumer demand. I was assuming that it's Nvidia's interest that intelligence is not only in a couple of frontier lab's hands, or they can choke the demand to suit their needs. If any company can access a frontier model without going through the frontier labs, the ecosystem will be larger and healthier for everyone, including Nvidia.

> That's why we had huge backlash w

Training a frontier model is prohibitively expensive, so it’s understandable that open weights aren’t truly open source. In fact, I worry whether open weights will even survive[1] once the current AI cycle cools down. An even bigger issue is that as long as model development remains this capital intensive, a genuinely open-source community around frontier AI is virtually impossible. That will have massive implications for our careers. The more capital concentrates at the top, the less leverage we’ll have in the job market.

[1] Arguably hardware companies like Nvidia have incentives to subsidize open models to drive GPU demand. That said, I'm not sure how sustainable and sufficient that incentive can be.

I find TurboTax is easy enough to use, except that data collection and entering is a huge hassle (to me, merely logging onto each website and getting the corresponding tax doc is painful chore). What would be a good way to collect the data easily? If I solve that problem, tax filing will become an easy task.

I assume that the demand will change. Say, a new generation of companies to replace the old. Besides, those who do have practical experience on building AI-native applications or building AI infra may be in higher demand as well. That is, maybe our experience will become not as valuable as before, therefore we'd need to refresh our experience with a different pace.

Senior ICs spend all their time either in meetings or on their way to meetings because there are so many teams to "align". It might be okay in the normal days: organizations grow, and we need people specialized in navigating company dynamics. But as we are facing the tsunami of AI, it is increasingly risky to stay a professional meeting goer and box drawer as an IC. The only way to derisk the situation is to go to a company that moves fast, and learn how to build next generation of software with AI, or building the AI infra.

I don't hate them. They're doing amazing work, and I'd like to believe they're genuinely trying to do good for humanity. I just think it's dangerous for any group of people to think they can impose their morality on everyone (only Dario knows what's right and wrong in military actions, only one company in the world, welfare to the current models, humanity has 10% of chance of getting destroyed by AI, and etc). For that, I'd rather see open-weighted and hopefully even open-sourced models win

What Dijkstra said still holds true, and the essence is consistent with Brooks' discussion on essential complexity vs accidental complexity. I do think one thing has changed, though: when someone asks AI to implement XXX with very vague English, the implicit ask is picking the best[1] ever solution that someone else has done to solve XXX. For instance, if I ask AI to extract the Japanese subtitle of Shirojima show and convert it to an epub with all the nantural and cultural explanations, I was assuming that AI will fill in the blanks and do a good job and AI did. This won't help expert, but does it matter? Being good enough can be good enough to take away many people's jobs.

[1] For many people, the result is actually the mean not the best, but it's another topic.

I think the hassle lies in uncertainty. Different hosts have different stands on house chores. One may expect me to simply put dishes in the dishwasher, while another may expect me to wash and dry all their bedsheets. Many people would rather just get out of this worry all together.

I’d rather use Booking because I didn’t have to worry about doing chores for the hosts. It was a much better experience.

> TiDB, the open-source distributed SQL database powered by PingCAP, today published the results of a rigorous source-code security assessment by NCC Group, one of the world’s leading cybersecurity consultancies.

It looks like you can assess the hell out of your source code, but your system can still leak if you don't secure your operations.

Yes. being good enough alone can destroy lots of job opportunities, unfortunately.

If AI can do what we deem creative work, say implementing a chunk of a Ruby gem, should we still consider such work creative in the first place? I was wondering if we should face an uncomfortable realization: much of what we call being creative was really the cost of overcoming cognitive friction due to the limit of our brains.

Cognitive scientists like Margaret Boden used to break creativity into three different categories: combinatory, exploratory, and transformational. It looks like a lot of our programming tasks are really combinatory and exploratory. Yes, they can be deemed novel and creative by individuals, but collectively they are standardized. It's just like finding integrals used to an art, but nowadays Mathematica can do it better to most of the integrals. Mastering special functions used to be the key to solving tricky PDEs, but nowadays few engineering students need to spend multiple semesters learning them. Quasi-RH was considered a Fields-level hard problem, yet AI could brute-force a proof with elaborate estimates. We humans have a brain that can process only a few variables at a time and it took ingenuity and years of effort to master a series of concepts before using them to solve a problem, but that could be just the cost of knowledge transfer, which AI can drive to zero in many cases.

However, the need for creativity does not go away. It just moves more to the transformational category. In addition, our craft will shift to arguably a less romantic tier: problem formulation, verification, and grounding. It will be a painful transition to many people, me included, but we have no choice but to adapt.

Note I'm not ignoring the economical impact of AI. It's very possible that lowering the cost of producing code increases the aggregate appetite for software, but the demand of software engineers can still go down. Even if total code volume explode tenfold, the labor market may not absorb displaced engineers. I wish there are solutions to this pain, but I fail to see what they are.

I honest don't understand why so many mathematicians think that math is doomed. I don't deny that many of their careers will be impacted. They spend their career studying certain problems yet now they may have to pursue a whole new direction. That can be very painful. But math, math itself has never been better. What will really change, is our definition of "hard problems". And I don't buy Prof. Tao's "math is contaminated" argument just because we knew that some problems are likely solved. If we suddenly got a few more Ramanujans and their manuscripts, would math be contaminated? Last I checked, Ramanujan's manuals are treasured as absolute gems.

Take the Quasi-RH proof as an example. At its core, OAI's proof relies on classical tools. It’s essentially a proof by contradiction, constructing conflicting upper and lower bounds. The high-level strategy is standard in my eyes. So why couldn't we humans crack Quasi-RH for so long? My understanding is that we human simply don't have the capacity to grind the solution out: the proof uses 277 estimates with a max depth of 12 layers, plus 64 lemmas. For context: Yau’s proof of the Calabi conjecture used a dozen or so estimates, with the key ones going only 3 layers deep, and that alone took peak Yau two to three years of intense regular work.

A human mathematician's absolute peak only lasts about a decade. It’s basically impossible for a human to pull off a chain of estimates so long, so deep, and so tightly cross-dependent as a machine can, especially when every single estimate has to be razor-sharp and mutually consistent without any conflicts.

But the real question is, why do we think that it's the apocalypse if a question like quasi-RH is solved (if it is solved)? Would it only mean that quasi-RH may not be as hard as we assessed? Or more generally, we thought that some problems were deeply complex and therefore required entire paradigm shift, but it turned out there was a computationally massive solution. So what? It's not like all the problems can be solved through sheer computational search, right? It can even be evaluable for someone to tackle a solved problem from a different angle too, like how Lomonosov solved the invariant surface problem with a new framework, or like how Newton generalized the binomial expansion.

I actually don't doubt Anthropic's sincerity. Per the NYT article, they seriously discussed how to give models welfare because the models may potentially possess moral status and awareness.

In fact, they are enforcing certain welfare policies, like banning accounts for repeatedly abuse their models.

Especially given the most recent report by NYT on how Anthropic tried to create a new belief, or religion if you will, I'd rather the world has more frontier models that are neutral to morality (basic standards, like upholding the law and user agreement are necessary, of course).

Ouch! I don't know how I missed that. In fact, I thought it was the opposite: a guest post by Tao for Álvaro Lozano-Robledo

> On the other hand, none of the proofs so far seem to contain “alien ideas,” a move-37, or completely novel arguments or new concepts that were not present in the literature in some form or another.

I have the same optimism as Prof Tao. That said, I can understand why so many mathematicians have been so upset or stressed out. It turned out much of the mathematical work is about clever combination of existing methods - this already requires enormous amount human ingenuity and years of dedicated learning. Unfortunately, or maybe fortunately, AI can be very good at knowledge transfer and finding combination of existing ideas to solve seemingly impossible problems. Even though mathematicians are extremely smart and capable, only a small number of them are capable of truly inventing "alien ideas", discovering new ground-breaking mathematical structures, or coming up with new problem-solving techniques. That is, AI can eat many mathematicians' cake. That said, I'm still hopeful. Mathematicians still understand mathematics deeply. If someone can prompt AI to solve an important problem, that person is more likely a good mathematician than an average joe like me. So, I think mathematicians do have a bright future: leverage AI, and make more and bigger math discoveries. It's still the same north star: we must know, and we shall know. It's just that with AI, we will know sooner and more.

I'm actually more optimistic. Math will still be fun and challenging and rewarding. It's just that we need to adapt how we choose which problems to solve.

Previously, due to the cognitive limitations of our brains, it was difficult to tell whether resolving a conjecture genuinely required years of dedicated effort, or if the answer was already hidden within existing human knowledge. Or whether a problem is just waiting to be searched out through ingenious or even brute-force means. With AI, we can now offload this search, especially across different areas of math, allowing us humans to focus our minds on discovering new mathematical structures and techniques.

Indeed, if one believes what Hilbert believed: we must know and we shall know, he should feel happy, as the belief has never been about who solves a problem, but about whether can advance our understanding of the universe. With the help of AI, we have a lot more possibilities.

You'd be surprised how VPs love to insert PMs. A famous game platform company, for instance, had two PMs for their storage team, one PM for their data team, one PM for the compute team, one PM for their dev tools team, if I remember correctly (the numbers could be larger, but won't be smaller)

It may appear so, but if you you talk to the Anthropic employees and management, especially those researchers, you'll see that they truly believe that the AGI is coming, if not has been here already.

> Google, a search engine which does not have human emotions, assumed that I had been spurned by a man in my life named Dario and decided what I wanted was an empathetic digital friend.

I'm not sure why this is a surprise. Google most likely uses an efficient yet limited language model to give us search answers. When a search query does not have strong or specific enough context, the model will sample and find what most of the training material will cover. That is, the most likely average behavior covered in the training data. In this case, "hes never coming over dario" is such a generic sentence, so unless Google happens to have indexed what the author mentioned and ranked it high, the answer would appear natural to the AI. Or using the AI jargon, the answer appears to be properly aligned.

> so they’re using LLMs to build native applications for every platform they support.

People say LLMs will let us build native apps instead of Electron, but is that missing the point? I thought people turned to Electron for a reason: building a fast, rich, collaborative Markdown editor on macOS is notoriously hard. Neither SwiftUI nor AppKit makes handling those rich interactions easy. AI can do wonders, but can it really overcome the inherent limitations of macOS's native UI frameworks?

I thought the difference is that clearly written rules or SLAs are acceptable and can even be negotiated before signing a contract, but having a person, in this case Amodei himself, manually approve DOJ's usage case by case with his own moral judgement after a contract is signed is not okay.

> Ability to use the plan outside of the official harness. Codex wins. Anthropic does shit like bills requests as extra usage if it sees a hermes.md in a commit.

I'm quite puzzled about why Anthropic is so hellbent on blocking other coding agents. It's not like Claude Code has any secret sauce, right? And doesn't Anthropic make monkey off API usage, and their magic is on the model side anyway?

Finally Apple has a screen time that allows different configurations for different time segments. What took them so long? /j

> Right now, overall compute is a moat.

Very true. Or further, access to capital is the moat. It is the very reason that we don't have a real open-source community that trains frontier models - individuals simply can't afford the training infrastructure, nor sufficient high-quality training data.

> I guess what they really want is to limit sale of AI models to compliant vendors and then raise the bar to compliance just high enough so they can pass it but smaller labs can’t.

If it's true, it makes them evil. Especially Dario, who speaks about moral high ground and fate of humanity all day, yet it's not that different from a cult leader does.