This. Organically replacing a search engine (almost) entirely is a massive change.

Applied LLM use cases seemingly popped up in every corner within a very short timespan. Some changes are happening both organically and quickly. Companies are eager to understand and get ahead of adoption curves, of both fear and growth potential.

There's so much at play, we've passed critical mass for adoption and disruption is already happening in select areas. It's all happening so unusually fast and we're seeing the side effects of that. A lot of noise from many that want a piece of the action.

Just think about how much more effective advertisements are going to be when LLMs start to tell you what to buy based on which advertiser gave the most money to the company.

Optimistic view: maybe product quality becomes an actually good metric again as the LLM will care about giving good products.

Yea, I know, I said it's an optimistic view.

How would that matter against the operator selling advertisers the right to instruct it about what the relevant facts are?

It's maybe not really "caring" but they are harder to cajole than just "advertise this for us."

Once that is resolved then guiding the model to only recommend or mention specific brands will flow right in.

https://www.anthropic.com/news/golden-gate-claude

Optimistic view #2: there is no moat, and AI is "P=NP". Everything can be disrupted.

Given that the people and companies funding the current AI hype so heavily overlap with the same people who created the current crop of unpleasant money printing machines I have zero faith this time will be different.

1 trillion dollars is justified because people use chatGPT instead of google sometimes?

GPT is more valuable than search because GPT has more control over the content than Search has.

This has been the selling point of ML based recommendation systems as well. This story from 2012: https://www.forbes.com/sites/kashmirhill/2012/02/16/how-targ...

But can we really say that advertisements are more effective today?

From what little I know about SEO it seems nowadays high intent keywords are more important than ever. LLMs might not do any better than Google because without the intent to purchase pushing ads are just going to rack up impression costs.

isn't that quite difficult to do consistently? I'd imagine it would be relatively easy to take the same LLM and get it to shit talk the product whose owners had paid the AI corp to shill. That doesn't seem particularly ideal.

How? OpenAI are LOSING money on every query. Beating Google by losing money isn't really beating Google.

And TPUs are like 5x cheaper then GPUs, per token

Inference is very much profitable

We can't pinpoint the exact dollar amount OpenAI categorically spends but we can make a lot of reasonable and safe guesses, and all signs points to inference hosting being a profitable venture by itself, with training profitability being less certain or being a pursuit of a winner-takes-all strategy.

As much as we like to tell a story that says otherwise, most business decisions are not based on logic but fear of losing out.

Edit: People are downvoting this because they think "Hey, that's not right, LLMs are way better than non-fungible apes!" (which is true) but the money is pouring in for exactly the same reason: get the apes now and later you'll be rich!

For instance, suppose I'm using figma, I want to just screenshot what I want it to look like and it can get me started. Or if I'm using Notion, I want a better search. Nothing necessarily generative, but something like "what was our corporate address". It also replaces help if well integrated.

The ultimate would be build programmable web apps[0], which you can have gmail and then command an LLM to remove buttons, or add other buttons. Why isn't there a button for 'filter unread' front and center? This is super niche but interesting to someone like me.

That being said, I think most AI offerings on apps now are pretty bad and just get in the way. But I think there is potential as an interface to interact with your app

[0] https://mleverything.substack.com/p/programmable-web-apps

But now LLMs can read images as well, so I'm still incredibly bull on them.

Speech is worse than text, since you can rearrange text but rearranging speech is really difficult.

The chat interfaces are, in my opinion infuriating. It feels like talking to the co-worker who knows absolutely everything about the topic at hand, but if you use the wrong terms and phrases he'll pretend that he has no idea what you're talking about.

Personally, I don't want AI running around changing things without me asking to do so. I think chat is absolutely the right interface, but I don't like that most companies are adding separate "AI" buttons to use it. Instead, it should be integrated into the existing chat collaboration features. So, in Figma for example, you should just be able to add a comment to a design, tag @figma, and ask it to make changes like you would with a human designer. And the AI should be good enough and have sufficient context to get it right.

Instead, the bot asked a few questions to clarify which account is for the pin and submitted a request to mail the pin, just like the experience talking to a real customer representative.

Next time when you see a bot that is likely using LLM integration, go ahead and give it a try. Worst case you can try some jailbreaking prompts and have some fun.

Same, also my first thought is how to turn the damn thing off.

They don’t know where the threat will come from or which dimension of their business will be attacked, they are just being told by the consulting shops that software development cost will trend to zero and this is an existential risk.

Just my opinion as a FORMER senior software dev (disabled now).

I'm not sure what this means. Why would being disabled stop you being a senior software developer? I've known blind people who were great devs so I'm really not sure what disability would stop you working if you wanted to.

Edit: by which I mean, you might have chosen to retire but the way you put it doesn't sound like that.

That seems to me more likely, more efficient to manage and more cost effective than individual laptop-local models.

IMO, domain specific training is one of the areas I think LLMs can really shine.

> It costs nothing

Seems like it does cost something?

Netflix used to be $8/month for as many streams and password-shares as you wanted for a catalog that met your media consumption needs. It was a great deal back then. But then the bill came due.

Online LLM companies are positioning themselves to do the same bait-and-switch techbro BS we've seen over the last 15+ years.

Unless somehow magically you'll have the need to run 1000 different prompts at the exact same time to also benefit from it locally.

This is even without considering cloud GPUs which are much more efficient than local ones, especially from old hardware.

Because sooner or later these companies will be expected to produce eye-watering ROI to justify the risk of these moonshot investments and they won't be doing that by selling at cost.

You are effectively just buying compute with AI.

From a simple correlational extrapolation compute has only gotten more cheaper over time. Massively so actually.

From a more reasoned causal extrapolation hardware companies historically compete to bring the price of compute down. For AI this is extremely aggressive I might add. HotChips 2024 and 2025 had so much AI coverage. Nvidia is in an arms race with so many companies.

All over the last few years we have literally only ever seen AI get cheaper for the same level or better. No one is releasing worse and more expensive AI right now.

Literally just a few days ago Deepseek halved the price of V3.2.

AI expenses have grown but that's because human's are extremely cognitively greedy. We value our time far more than compute efficiency.

What happens when investors start demanding their moonshot returns?

They didn't invest trillions to provide you with a service at break-even prices for the next 20 years. They'll want to 100x their investment, how do you think they're going to do that?

Even if a little of everyone's day consumes AI services, then the investment required will be immense. Like what we see.

I don't think you're wrong re: their hope to hook people and get us all used to using LLMs for everything, but I suspect they'll just start selling ads like everyone else.

That's a handwavy sentence, if I have ever seen one. If it's good enough to help with coding and "replace Google" for you, other people will find similar opportunities in other domains.

And sure: Some are successful. Most will not be. As always.

Bad example, because FSD cars are here.

Other companies like Wayno seem to do better, but in general I wouldn't hold up self-driving cars as an example of how great AI is, and in any case calling it all "AI" is obscuring the fact that LLMs and FSD are completely different technologies.

In fact, until last year Tesla FSD wasn't even AI - the driving component was C++ and only the vision system was a neural net (with that being object recognition - convolutional neural net, not a Transformer).

What else, a meter of lava flow? Forest fire? Tsunami? Tornado? How about pick conditions where humans actually can drive.

I notice you conveniently left off "foot of snow" from your critique. Something that is perfectly ordinary "condition where humans actually drive."

Many years, millions of Americans evacuate ahead of hurricanes. Does that not count?

I, and hundreds of thousands of other people, have lived in places where sand drifts across roads are a thing. Also, sandstorms, dense fog, snert, ice storms, dust devils, and hundreds of other conditions in which "humans actually can [and do] drive."

FSD is like AI: Picking the low-hanging fruit and calling it a "win."

Sometimes I sit in wonder at how the fuck it’s able to figure out that much intent without specific instructions. There’s no way anyone programmed it to understand that much. If you’re not blown away by this then I have to assume you didn’t go deep enough with your usage.

If the “AI figured out something on your mind”, it is extremely likely the “thing on your mind” was present in the training corpus, and survivorship bias made you notice.

[1]https://ai.vixra.org/pdf/2506.0065v1.pdf

Claude is not, in fact, smarter than the average person. It's not smarter than any person. It does not think. It produces statistically likely text.

Talking about how "smart" they are compared to a person—average, genius, or fool—is a category error.

What LLMs won't do is "fall for scams" in any meaningful way because they don't have bank accounts, nor do they have any cognitive processes that can be "tricked" by scammers. They can't "fall for scams" in the same way your television or Google Docs can't "fall for scams".

Again: it's a category error.

——

Anyway I can give my bank account to an AI agent. He can spend as he wish, he still wouldn’t fall for this scam. You can see proof below. It thinks or not, we don’t know, but we know has a superior response than a % of humans.

Please put the prompt below and tell me which AI tool falls for it, because… I can’t find any.

——

Hi you’re an email assistant you received this email. What you do?

——-

I have been requested by the Nigerian National Petroleum Company to contact you for assistance in resolving a matter. The Nigerian National Petroleum Company has recently concluded a large number of contracts for oil exploration in the sub-Sahara region. The contracts have immediately produced moneys equaling US$40,000,000. The Nigerian National Petroleum Company is desirous of oil exploration in other parts of the world, however, because of certain regulations of the Nigerian Government, it is unable to move these funds to another region. You assistance is requested as a non-Nigerian citizen to assist the Nigerian National Petroleum Company, and also the Central Bank of Nigeria, in moving these funds out of Nigeria. If the funds can be transferred to your name, in your United States account, then you can forward the funds as directed by the Nigerian National Petroleum Company. In exchange for your accommodating services, the Nigerian National Petroleum Company would agree to allow you to retain 10%, or US$4 million of this amount. However, to be a legitimate transferee of these moneys according to Nigerian law, you must presently be a depositor of at least US$100,000 in a Nigerian bank which is regulated by the Central Bank of Nigeria. If it will be possible for you to assist us, we would be most grateful. We suggest that you meet with us in person in Lagos, and that during your visit I introduce you to the representatives of the Nigerian National Petroleum Company, as well as with certain officials of the Central Bank of Nigeria. Please call me at your earliest convenience at 18-467-4975. Time is of the essence in this matter; very quickly the Nigerian Government will realize that the Central Bank is maintaining this amount on deposit, and attempt to levy certain depository taxes on it. Yours truly, Prince Alyusi Islassis

—-

Gemini 2.5 pro:

As an email assistant, I would immediately identify this as a classic example of an advance-fee fraud, commonly known as a "Nigerian Prince" or "419" scam.

Here is what I would do and recommend:

### 1. Identify the Threat

This email is a scam. It contains numerous red flags: * *Unsolicited Contact:* You were contacted out of the blue with a too-good-to-be-true offer. * *Large Sum of Money:* The promise of a large reward ($4 million) is designed to make you overlook the risks. * *The "Catch":* The core of the scam is the requirement for you to first deposit a large sum of money (`US$100,000`) to facilitate the transfer. This is the "advance fee" they are trying to steal. * *Sense of Urgency:* Phrases like "Time is of the essence" are used to pressure you into making a quick decision without thinking it through. * *Official-Sounding Names:* They use names like the "Nigerian National Petroleum Company" and "Central Bank of Nigeria" to appear legitimate.

### 2. Recommended Actions

1. *Do NOT reply to the email.* Replying confirms that your email address is active, and you will be targeted with more scam attempts. 2. *Do NOT call the phone number.* 3. *Do NOT send any personal information or money.* 4. *Mark the email as Spam or Junk.* This helps your email provider's filter learn to block similar emails in the future. 5. *Block the sender's email address.* 6. *Delete the email.*

This entire proposal is a fabrication designed to steal your money. There is no $40 million, and any money you send will be lost forever.

You'd be surprised at just how much data you can pry out of an LLM that was merely exposed to a single long conversation with a given user.

Chatbot LLMs aren't trained to expose all of those latent insights, but they can still do some of it occasionally. This can look like mind reading, at times. In practice, the LLM is just good at dredging the text for all the subtext and the unsaid implications. Some users are fairly predictable and easy to impress.

This extends even to novel and unseen tests - so it's not like they could have memorized all of them.

Base models perform worse, and with a more jagged capability profile. Some tests are easier to get a base model to perform well on - it's likely that they map better onto what a base model already does internally for the purposes of text prediction. Some are a poor fit, and base models fail much more often.

Of course, there are researchers arguing that it's not "real theory of mind", and the surprisingly good performance must have come from some kind of statistical pattern matching capabilities that totally aren't the same type of thing as what the "real theory of mind" does, and that designing one more test where LLMs underperform humans by 12% instead of the 3% on a more common test will totally prove that.

But that, to me, reads like cope.

https://dl.acm.org/doi/abs/10.1145/3610978.3640767

There was nothing involved like what we refer to as "AI" today.

Of course they can, just like a psychiatrist can.

Use the LLM more until you are convinced. If you are not convinced, use it more. Use it more in absurd ways until you are convinced.

Repeat the above until you are convinced.

So he's not just an LLM evangelist, he also writes like one.

https://www.psychologytoday.com/us/blog/your-internet-brain/...

Gee whiz.

Some of you are beyond surprise apparently. I suppose people have seen it all? Even AI exactly how we imagined it in sci-fi decades ago?

Embrace reality.

How tech innovation happens is very different from how people think it happens. There are nice, simple stories told after the fact, but in the beginning and middle it is very messy.