More here

https://hunch.net/?p=13764006

This should be the main link

Almost inclined to agree, but the headline of Langford's post needs some additional context.

Hugged to death like it’s 2008 and you’ve just lit your LAMP stack.

Thanks for the heads-up. I added an archive.is link to give it a breather.

Appreciate this, much needed context.

Ok let's put that link above!

Hey Dang thanks for merging the stories. There's a typo in the editorialised title though.

Discussion on source: https://news.ycombinator.com/item?id=50406589

New AI lab incoming?

Yeah, in India.

Satya Nadella is a total loser. It’s easy to lead when things are good…

[delayed]

Curse of money strikes again. $4tn market cap company thinks team of 50 researchers too expensive.

Think of the chi... investors!

[delayed]

If MS abandoned a promising idea, pursuing it outside makes perfect sense. I am sure there are investors willing to fund this. This could even be an opening to attack the moat of the current AI frontier model companies.

Real research at most big companies is dead. Unfortunately, what has replaced it is mostly venture-backed startups, where the timeline to a return is still expected to be quite short.

As a result, you end up with companies doing basic research under the guise of developing a product, announcing timelines which are never going to be met. Fusion energy and quantum computing startups are some of the most egregious examples of this.

I have been shorting most of the Quantum Computing startups for about a year. They have been printing. Missed out on Fusion! (?)

Real research has always been long term as well as heavy manufacturing. It requires long-term planning with long-term budget, which is hard to get when money is expensive.

There’s actually a bizarre irony.

Remember Bell labs, XEROX parc? Ask yourself why that existed and why research came out of them.

Monopoly.

So - many people don’t want to admit this but monopolies can be good from an R&D perspective, but without strong oversight by regulators - money will instead spent on R&D projects that specifically raise entry barriers.

[delayed]

> Anyway, you cannot have both monopolies and strong regulators...

> Monopolies occur because regulation doesn't exist or is captured.

IMO, the stronger the regulator, the stronger the incentive to capture it.

Unless by "strong" you mean "competent"?

I meant competent, but regulator strength in the way you're implying at its limit becomes monopolistic itself. So when I speak of regulation strength with respect to monopoly, I am speaking to the ability of a regulator to effectively both understand their duty and carry it out.

Agreed. I used to be an industrial researcher at one of the last companies that didn’t require their researchers to think of short-term productization goals. However, things changed at my former company, and I ended up leaving for a tenure-track position at a community college.

I don’t know what industrial research in CS is like in the era of agentic coding. On one hand I suppose building prototypes is faster than ever, but on the other hand I genuinely don’t know how well LLMs do with harder programming tasks. It’s one thing to have an agent work on a CRUD app, but I wonder how effective agents are at handling things like parallelism, distributed computing, load-balancing, and other topics that are harder to write than the average CRUD app. I am also aware of efforts such as Sakana AI’s automated scientific discovery research, and I wonder if many of the incremental, short-term research topics many companies pursue can be automated.

I’m a bit pessimistic about the state of research these days, from funding to publish-or-perish to the effects of generative AI (automation, AI slop publications, AI slop reviews). Then again, I’m a bit of a curmudgeon, and maybe things are better than I think.

John was the creator of Vowpal Wabbit, which used to be a popular ML library: https://en.wikipedia.org/wiki/Vowpal_Wabbit

Well I never thought I’d see Digg used as a primary source

This is certainly an interesting timeline in this section of the light cone

Almost as fascinating is that the article references Digg as the source - genuinely hadn't heard of Digg for maybe a decade? I thought the DVD encryption fiasco + reddit had ended it?

It got rebooted a couple years back. One of the founders of reddit was involved too lol.

https://en.wikipedia.org/wiki/Digg#2025%E2%80%93present:_The...

Digg was briefly rebooted but once again died mere months after launch. Now they're peddling AI scams and the like.

His complaints sound childish to me. If the research products were truly innovative and useful, he wouldn't need to force Microsoft to adopt them.

You think incumbent megacorps are all quick at adopting new, useful technologies?

Some are, some aren't. Some try, but botch it.

Autogen is open source. I don't know the availability of other projects. Generally engineers in megacorps aren't shy of picking up the latest AI tools. I suppose Microsoft may have barriers around this though, curious to discover more details.

Yeah, I agree. What he describes reflects the exact challenges I've experienced literally everywhere. He is basically saying work/life is hard so he looking for a better playground. I do not know his contributions to AI so maybe he deserves to be better listened to but I find that never reflects corpo life.

> If the research products were truly innovative and useful, he wouldn't need to force Microsoft to adopt them.

That's just patently not true. History is littered with products that were very innovative and useful (and often far superior to competing products), but were canned, bought by competetors, or shelved in other way - often because they threatened the profits being made by some entity (though that's not the only reason).

That's a name I haven't heard in a while

(In the context of Microsoft Research) Man, the history of how large corps fumble major wins should be studied.

I remember during my thesis how Microsoft Academic (a Google scholar/Scopus alternative) was ridiculously good at its job.

I'm talking, "input the key words and filter out the wrong fields and you've got 20 highly relevant papers that no other search term has found" level.

...Of course, they proceeded to close it down.

I can't even express in words how mad and disappointed I was and still am at Microsoft for this alone. It felt the closest to a genie or an oracle in being amazing. (The old Swype vs modern Gboard is probably a close example for those who'd used both.) It's just... crazy, frustrating, and crazy frustrating.

Yay China!

This seems to track with a recent blog post from MAI regarding their approach to pre-training their first Frontier level model which seems promising but does seem to have a focus on scaling.

https://microsoft.ai/news/frontier-pre-training-at-microsoft...

Companies with billions of $, stopping R&D on the most revolutionizing technology/ breakthrough is something I can’t understand. I do understand there’s an AI Race to win, and that is with transformer and current LLMs architectures, but if research could find a better, more efficient approach they would re-shape this race, lead it, or even dominate it.

They are losing tons of money on it. Simple as.

But is it a race to zero? Maybe AI will make very expensive things (eg software engineering) very cheap, but there will be enough competition that the margins are tiny and don't actually justify putting crazy dollar amounts into research

That's exactly what's happening. Software -> 0 marginal cost (obviously still need engineers to actually make the architecture). LLMs -> Price of Energy / Amortized Cost of GPUs. "AI Labs" -> Owned by Creditors

From an engineer, who’s devastated by this, i have to admit/raise my concern: why do we still need engineers to make architecture? Why is everyone saying that’s a must? For me the toughest problem was coding a huge repo/pkgs, architecture should be much of an easier task to do. I’ve seen it do sys design, scaled architecture better than 10X engineers. With good planning, context and prospective info -> it can produce extremely good architecture even for complex systems.

Im not sure I would say architecture specifically. Just anything on the input and output.

It cant lmao (skill issue)

The problem is the "if". There is no guarantee that there's a "better, more efficient approach" to be found.

So far, the attempts to replace the transformer LLM with something else have yielded either "the same transformer LLM but with this one tweak that makes it 10% better" or "a radically different architecture that's a bitch to work with, and performs, best case, about the same as the transformer LLM".

By now, I'm firmly in the camp of "the architecture is good enough". Rifle is fine. Transformer is fine. Get better at the transformer before you look at anything else.

And getting the transformer better may make finding the next thing easier

I think it’s more likely that some future LLM model finds a more efficient architecture than a human does, at this point. So yeah I’d go all in on LLMs too.

Big corporations, by nature, are not rational in the decisions they make.

That's why they'll always be room for startups.

Me neither. When revenues are growing every year, a fixed chunk of researchers' salaries (maybe growing a few % each year) doesn't really hurt, but R&D offers a nice upside.

Is this the same group that did ResNet? Bad news

He didn't say they were not doing transformers research. In fact, he specifically mentioned a type of transformer they were working on. And labs like Qwen and DeepSeek (at least) have shown how incredible results can be when you innovate in the area of LLMs.

I think the real point of the post is that office politics and poor management got in the way of innovation due to infighting and poor execution in transfer of technology from research to product.

[delayed]

It seems as if Microsoft is losing developers now.

The whole "let's go into AI" is not really compatible with oldschool software jobs. AI drains humans dry.

I mean we are already seeing the hype cycle coming to a close, with the stock market showing this, but key people leaving kind of means that Microsoft is losing its profile here. Add to this that many don't like Win11 and you see a huge paradigm shift here - and tons of associated problems.

"In practice, there are substantial issues because the philosophy of the ex-Inflection leadership of MAI is incompatible."

AI is killing people here - not necessarily literally (though we have to see how much economic damage AI has caused) but figuratively speaking. The counter-wave is coming here.

It is notable that with the advancements of 24/7 coding agents with basic research capabilities the premise:

> a clever idea pursued by one person it was likely to arrive late and unconvincing because one person could not move fast enough and the baselines which one person could assemble just were not strong enough to really show that an idea was worth taking flight

may no longer be true. The limiting factors are frontier model token and compute budgets, researcher skill, drive, foresight and imagination, not capability to work in teams.

Ironically now by collaborating with AI one person can absolutely move fast enough. I used to work in orgs like that and am now vastly more productive than I ever was back then. It could take me months to convince a peer of an idea that now only takes minutes to explain to AI.

> there’s a breakdown of the basic deal in MSR: do good work and good things happen

This really hit home for me. This is the contract that held for a long time at Microsoft and has been broken and rebroken with current leadership. It’s culture-destroying.

> there’s a breakdown of the basic deal in MSR: do good work and good things happen

This really hit home for me. This is the contract that held for a *long* time at Microsoft and has been broken and rebroken with current leadership. It’s culture-destroying.

It makes me sad because there are so many unexplored areas of AI research, even on text generation. Billions are invested in almost one idea, which very realistically isn't the most optimal way to do text generation, but a lot of companies are afraid of taking risks. I still wouldn't put $1b on JEPA but I prefer a world where we can try things like that.

LLMs have very favourable exploration<>consolidation dynamics. An LLM alternative would need to display fundamental improvements at relatively large scales which is extremely challenging for new research.

It's a lot easier to explore these other areas with the help of LLMs.

This is the canary in the coal mine. The canary is dead. The AI Winter is coming.

This scaling approach is unsustainable, it is generating diminishing returns and ballooning costs. It is already blowing up.

Did you read the article? it literally showcases research published 2 weeks ago on how to make *better models*

Relevant: https://www.quantamagazine.org/when-chatgpt-broke-an-entire-...

On the flip side, frontier models are so good and relatively cheap now that it's possible to pursue your pet research project on your own as long as resources are not a constraint.

Great link, thanks for sharing. I missed that when it came out. I'd love to read something similar about other fields, like chess AI up to 1996.

Yes, leave it to the Chinese to advance the state of the art. America throws money at it, the rest brains. Because they don't have the HW.

Same happened to computer chess in the 80ies. Clever small SW was leading the tournaments, until Ken Thompson could persuade IBM to throw hardware at it, Big Blue. This win was shortsighted. HW didn't scale and nobody profited from it. SW did, esp. reinforcement learning.

but Mircosoft AI model is growing greatly and recently beat the nanobanana in generate video domain. I can't image the situation of them are bad as you explained.

I think the dichotomy of open-research vs. commercially-driven innovation might be easier to coordinate, with three stages, not two:

1. Open research: Great creative freedom, with great resource constraints. Lots of ideas, lots of things tried, at insignificant cost relative to commercial work.

2. Results filtering and vetting: Opportunistic selection of apparently successful work from stage 1, for tentative testing at increasing scale. The best ideas end up with moderate budgets that verify benefits, and rework for compatibility, before any handoff to stage 3.

3. Commercial efforts aimed strictly at improving quality and economics of current revenue sources.

Mediating between just #1 and #2 involves too much dissonance over complementary but incompatible values. #3 can be a relatively small group, but keeps #1 and #2 from butting heads and facilitates them both.

By significantly lowing the uncertainty, risk and numbers of #1 efforts before consideration from #3.

I'll play devil's advocate. Of course John Langford thinks what he and his group were doing held a lot of promise, and the work was fun and challenging and super important. But in the end you also need to live in the present and deliver. In the words of Jerry Maguire, "Show me the money". He had years to show Microsoft the money, and in the same time other people at other firms showed their bosses the money, big time.

this is not true. even with ideas with provable/strong scalability, there was still a lack of resources (especially GPUs) to validate such ideas at scale. ultimately, "showing the money" still requires some initial faith from those higher up who control the resources.

John Langford was the creator of "VW" ("Vee Dub", or "Vowpal Wabbit"), an ML framework that was quite efficient back in the day ( ~ 20 years ago ).

https://vowpalwabbit.org/

Microsoft's now shown researchers their good work isn't safe.

Hmm... So strange. It seems obvious(?) that both research and scale are needed, and that one should never have one starve the other. In fact research can influence the scale needed, and vice versa.

I guess when money is concerned people will always seek out the the most reliable and predictable path - at the risk of being potentially left behind if it turned out to the wrong (or at least suboptimal) path.

> scaling is straightforward

I remember IllyaS saying scaling doesnt work anymore so we are just RLcrazy. Is that not true?

I'm about 90% certain that there are numerous huge inefficiencies in these big transformer models. Inefficiencies so big that eliminating them would make it possible to run the equivalent of the latest frontier models on your phone.

But nobody wants to look for them too hard because centralized AI as a service is where all the money is. So everybody keeps building bigger rather than smarter.

This was expected. Despite ill-advised views, AI was indeed NFTs in slow motion. We're finally returning to normalcy.