That's not an argument made in the linked tweet. His claim is "they couldn't replicate it because they didn't follow the steps", which seems like a very reasonable claim, regardless of the motivation behind making it.
Either the research is as much of a breakthrough as is claimed and Google is about to pull way ahead of all these other "idiots" who can't replicate their method even when it is described to them in detail, or the research is flawed and overblown and not as effective as claimed. This seems like exactly the sort of question the market will quickly decide over the next couple of years and not worth arguing over.
Why do a non-zero amount of people have seemingly religious beliefs about this topic on one side or the other?
The guys at EDA companies care because Google's result makes them look like idiots when you take the paper at face value, and does advance the state of the art a bit. They have been working hard for marginal improvements, and that some team of ML people can come in and make a big splash with something like this is offensive to them. Furthermore, the result is not that impressive and does not generalize enough to be useful to them (and competent teams at these companies absolutely have checked).
The fact that the result is so minor is the reason that this is so contentious.
https://sparktoro.com/blog/is-google-losing-search-market-sh...
If anything, now is the best time for TPU to grow and I'd say investing in TPU gave Google an edge. There is no other large scale LLM that was trained on anything but NVIDIA GPUs. Gemini is the only exception. Every big company is scrambling to make their own hardware in the AI era while Google already has it.
Everyone I know who worked with TPUs loves how well they scale. Sure Jax has a learning curve but it's not a problem, especially given the performance advantages it gives.
Google does indeed lock in their own ROI with deciding to not compete with AMD / Graphcore etc, but that also rooflines their total market. If they were to come up with a compelling Android-based Jetson-like edge product, and if demand for said product eclipses total GPU demand (robotics explosion?) then they might have a ramp to compete with NVidia. But the USB TPUs and phone accelerators today are just toys. And toys go to the Google graveyard, because Googlers don’t build gardens they treat everything like toys and throw them away when they get bored.
n.b. you're on a social news site
> pull way ahead of all these other "idiots"
Pulling way ahead sounds sufficient, not necessary. Can we prove it's not the case? Let's say someone says that's why Gemini inference is so cheap. Can we show that's wrong?
> "idiots"
?
Because lots of engineers are being told by managers "Why aren't we using that tool?" and a bunch of engineers are stuck saying "Because it doesn't actually work." aka "Google is lying through their teeth." to which the response is "Oh, so you know better than Google?" to which the reponse is "Yeah, actually, I fucking do. Now piss off and let me finish timing closure this goddamn block that is already 6 weeks late."
Now can you understand why this is a bit contentious?
Marketing "exaggerations" from authority can cause huge amounts of grief.
In my little corner of the world, I had to sit and defend against the lies that a startup with famous designers were putting out about power consumption while we were designing similar chips in the space. I had to go toe to toe with Senior VPs over it and I had to stand my ground and defend my team who analyzed things dead on. All this occurred in spite of the fact that they had no silicon. In addition, I knew the famous designers involved would happily lie straight to your face having worked with them before and having been lied straight to my face and having had to clean up the mess when they left the company.
To be fair, it is also the only time I have had a Senior VP remember the kerfuffle and apologize when said startup finally delivered silicon and not only were the real numbers not what they claimed they weren't even close to the ones we were getting.
If you have personal experience with Jeff Dean et al that you're willing to share, I'd be interested in hearing about it.
From where I'm sitting it looks like, "Google spent a fortune on deep learning, and got a small but real win. People who don't like Google failed to follow Google's recipe and got a large and easily replicated loss."
It's not even clear that Google's approach is feasible right now for companies not named Google. It is not clear that it works on other classes of chip. It is not clear that the technique will grow beyond what Google already got. It is really not clear that anyone should be jumping on this.
But there is a world of difference between that, and concluding that Google is lying.
From where I'm sitting it looks like Google cooked the books maximally, barely beat humans let alone state of the art algorithms, published a crappy article in Nature because it would never have passed editorial muster at something like DAC or an IEEE journal and now have to browbeat other people who are calling them out on it.
And that's the best interpretation we can cough up.
I'll go further, we don't even have any raw data that says that they actually did beat the humans. Some of the humans I know who run P&R are REALLY good at what they do. The data could be completely made up. Given how much scientific fraud has come out lately, I'm amazed at the number of people defending Google on this.
Where I'm from, we call what Google is doing both "lying" and "bullying".
Look, Google can easily defuse this in all manner of ways. Publish their raw data. Run things on testbenches and benchmarks that the EDA tools vendors have been running on for years. Run things on the open source VLSI designs that they sponsored.
What I suspect happened is that Google's AI group has gotten used to being able to make hyperbolic marketing claims which are difficult to verify. They then poked at place and route, failed, and published an article anyway because someone's promotion is tied to this. They expected that everybody would swallow their glop just like every other time, be mostly ignored and the people involved can get their promotions and move on.
Unfortunately, Google is shoveling bullshit around something that has objective answers; real money is at stake; and they're getting rightfully excoriated for it.
Whoops.
Likewise the importance of spending 20x as much money on the training portion seems easy to verify, and significant.
That they would fail to properly test against industry standard workbenches seems reasonable to me. This is a bunch of ML specialists who know nothing about chip design. Their background is beating everyone at Go and setting a new state of the art for protein folding, and not chip design. If you dismiss those particular past accomplishments as hyperbolic marketing, that's your decision. But you aren't going to find a lot of people in these parts who agree with you.
If you think that those were real, but that a bunch of more recent accomplishments are BS, I haven't been following closely enough to have an opinion. The stuff that crossed my radar since AlphaFold is mostly done at places like OpenAI, and not Google.
Regardless, the truth will out. And what Google is claiming for itself here really isn't all that impressive.
"If Cheng et al. had reached out to the corresponding authors of the Nature paper, we would have gladly helped them to correct these issues prior to publication" (https://arxiv.org/pdf/2411.10053)
That's how you actually do a reproduction study - you reach out to the corresponding authors and make sure you do everything exactly the same. But at this point, it's hard to imagine the AlphaChip folks having much patience with them.
I don't think it's easier to get into DAC / an IEEE journal than Nature.
Their human baseline was the TPU physical design team, with access to the best available tools: rdcu.be/cmedX
and this is still the baseline to beat in order to get used in production, which has happened for multiple generations of TPU.
TPU is export controlled and super confidential -- multi-billion dollar IP! -- so I don't see raw data coming out anytime soon.
If the Nature paper made it clear that RL is not seriously expected to work on non-TPU chips, it would have have probably been rejected. If RL works on many other chips, then evidence should be easy to publish.
As I said, the truth will out. I was just unhappy with the case previously being offered.
AlphaChip is used to make real chips in production. Google publicly announced its use in multiple generations of TPUs and Axion CPUs, and MediaTek said they've built on it as well.
Discussions like this are _how_ the market decides whether or not this achievement is real or not.
If they did not follow the steps to replicate (pre-training, using less compute, etc.) and then failed, so what's wrong with calling out the flaws in their attempted "replication"?
Actually, people should criticize flawed papers. That's how science works! When you publish scientific papers, you should expect criticism if there's something that doesn't look right.
The only way to avoid that is to get critical feedback before publishing the paper, and it's not always possible, so then the scientific debate happens in public.
Though, the broader question is how useful the results of the original paper are to other people who might do the same thing.
> It's not a value judgement, just doesn't help his case at all.
Calling it "bullying" looks like a value judgment to me. Am I missing something?
To me, Dean's response is quite sensible, particularly given his claims the other papers made serious mistakes and have potential conflicts of interest.
Second, even a healthy environment can be undermined by lack of skills or resources, intellectual dishonesty, or conflicts of interest.
Where does it say that? Dean outlines explicit steps that the authors missed in the tweet.
The tweet just says that the reproduction attempt didn't didn't actually follow the original methodology. There is no claim that the authors of the replication attempt were "idiots" or anything similar, you just made that up. The obviously fallacious logic in "they couldn't replicate it ..., therefore it's replicable" is also a total fabrication on your part.
Making a novel claim implies its *_claimed_ replicability.
"You did not follow the steps" is calling them idiots.
The only inference I made is that he's pressed to comment. He could have said nothing.. instead he's lashing out publicly, because other people were unable to replicate it. If there's no problem replicating the work, why hasn't that happend? Any other author would be worried if a publication about their work were saying "it's not replicable" and trying their best to help replicate it.. but somehow that doesn't apply to him.
But that was explicitly limited to 8 hours for all setups. Do they have another paper that shows that you can't increase the number of hours of a smaller GPU setup to compensate?
Jeff drank so much kool aid he forget what water is.