It's still not clear to me that LLMs are actually useful, but I'm perfectly willing to accept that I'm some sort of neo-luddite (how deliciously ironic a term by the way). I wonder, do you think there are examples of companies that have good strategies now? And if so, what utility are they getting out of it?

I'm in finance and we're using LLMs to summarize a ton of soft data on company performance. We're making a killing and beating the market substantially. Within the next five years I don't think there will be anyone left standing who isn't worth a trillion dollars or doesn't have all their quants augmented by LLMs.

Before this I was building a matching engine for online retail which used (then) state of the art multimodal models to decide which products would match your query to the lowest priced item. The dirty secret here is that we've solved search, it just costs too much to provide it for free. I think we'll soon see an online Costco show up which can provide this service to members and filter all the bullshit dropshippers.

I see that as a continuation of a long arc of using machines to read things in finance (not just on the buy side, btw.), don't think it will change the competitive dynamics between the more dominant players. (There might be some more concentration in some areas, but the dynamics for that pre-date LLMs or even AI/ML in some cases.)

The question is if their holdings will melt faster than they can develop the capability to use the new AI better than the small fries going after them.

So far this year the main thing slowing down development for us has been the SEC seizing our development boxes for insider trading once a month or so. Which given that each cost $50,000 has been something of a sore point.

Some of the "small fry" might get big, but similarly some of the big players will stay big - even pre-AI there have been sizable shifts in AUM.

> Within the next five years I don't think there will be anyone left standing who isn't worth a trillion dollars or doesn't have all their quants augmented by LLMs.

What an interesting comment. So, your thesis is that a few players that come out ahead will be worth substantially more than the biggest players now and they will all be using LLMs? What's driving the consolidation? High barrier to entry for making a foundation model for finance?

>I'm in finance and we're using LLMs to summarize a ton of soft data on company performance. We're making a killing and beating the market substantially.

so are you saying that you are okay with LLM hallucinations causing your company to make possible huge losses, even if rarely ?

I should mention that I am an LLM newbie, or rather a zerobie. all I have is a very high level sketchy overview of the field. but I am a dev with many years background in many areas, though not AI at all, except for reading a little bit now and then about it.

feel free to blast me, but preferably with data-backed statements.

I think customer support will be replaced or severely downsized within the next 5 years and that's a pretty big number of jobs.

I honestly don’t know. AI is so young; it’s like asking who had good internet strategies in 1994; in hindsight we know some did, but nobody really knew at the time (I would argue those who say otherwise are speaking from survivorship bias).

But take a look at GitHub’s Copilot Workspaces for ideas of how transformative AI will be.

I don’t have all the pieces (if I did, I’d be rich, right?), but I think AI will deliver mass customization of goods, software, and content by reducing up front costs to near zero. So if I were looking for good strategies, I’d start with companies that currently spend a lot of money on software or content creation and who are investing in automating that.

From own experience with ChatGPT, I asked it some months ago to go through an EU regulation, and create a checklist/audit programme for IT Security. Work that would have taken me a couple of weeks, it took 'it' a few seconds. I had to go through the text to make sure it's clean and doesn't contain hallucinations. Total work effort 1-2 hours max.

This doesn't mean that the human is no longer required, but for certain tasks a company can save a lot of time/money.

So I don't know if they dream to drop a dept of 50 people and replace it with a single machine, but there are some tasks that cannot be outsourced to machines (e.g. decisions impacting the lives cannot be made by automation)

"the existence of automated decision-making, including profiling, referred to in Article 22(1) and (4) and, at least in those cases, meaningful information about the logic involved, as well as the significance and the envisaged consequences of such processing for the data subject."

So good luck 'convincing' the regulator that an LLM can replace mortgage-decisions makers (it is considered a significant activity in a person's life).

If the regulations are so extensive that you would have needed a couple of weeks to create a checklist, how did you manage to see if the list was complete and correct within a couple of hours?

This was my first thought? The amount of effort I have to go through verifying what I get from gpt makes me wonder. On the other hand, there I’ve been thinking about how it might help me read datasheets. There are details that are easy to miss when skimming, that may be easy to verify through simple text searches.

That said, I know someone who almost took a job writing the actual text of laws, and they were excited about it because, while they didn’t control the laws in general, comma placement allowed them to subtly change the law’s meaning.

I currently would‘t trust an LLM to understand subtle comma placement.

LLMs are very good tools for automating tasks which shouldn't exist at all.

You mean automating bureaucracy? That would make room for more bureaucracy albeit in the form AI bureaucracy. I don’t think we’re ready for that though.

No, I really mean things that shouldn't exist at all. I see it used a lot by marketing people generating press releases nobody really cares about, online media generating images for short useless pieces meant to be filled a space between few really useful articles etc.

> generating press releases nobody really cares about, online media generating images for short useless pieces meant to be filled a space between few really useful articles etc.

I can't see any evidences that those aren't really important or useful given your comment, do you have any data to support your argument?

LLMs are tremendous force multipliers. They don't replace humans but increase the productivity of people quite a bit

I've seen that claim made but I've not seen it in reality.

There are examples of where LLMs are useful for information retrieval in that they can take vague queries and produce good summaries, but they're just as likely to produce reasonable-sounding nonsense that needs to be carefully verified.

I've also seen examples where LLMs can produce code that implements well-defined toy problems but when used on a real software system with millions of lines of context they do not produce code that can remain coherent with the system.

From what I've seen LLMs have yet to graduate from "impressive demo" and everyone characterizing them as being a revolution are still describing some potential future state with no real evidence that we'll get there with this technology or any time soon.

I agree that LLMs are less useful for smaller scale use cases. I’ve found success using them for creating mock data, reformatting unstructured text into structured data, and a few other menial tasks that would take me a while to do on my own.

I think so much of the hype is about potential larger scale applications but the models just don’t seem reliable enough yet for that.

We live in different realities then. I can't talk about details for my previous employer, but I can say there is an implementation involving LLMs saving Data Scientists a lot of time right now. I witnessed impact first hand and advised on the implementation and refinement for the group that built it.

That’s not a certainty but an AI salesman’s pitch.

Lived experience for me.

Motte: "AI will make humans more efficient and help scale small businesses and creators"

Bailey:" AI can replace your entire human labor (the most expensive cost of business) by scraping the content made by laborers on the internet without compensation"

Unfortunately, businesses are being sold the Bailey. And the sad part is that some companies aren't even pretending to hide behind the motte. I'm sure some, many businesses won't fall for the hype, but many others will and it will take a huge toll on the workers as a result.

But are you actually a luddite objecting to the impact that technology (AI in this case) is going to have or just a skeptic?

No, the internet was always obviously useful. It's the best communication tool since the written word and people were using it as such from the beginning. Of course not many people had access in the 1980s but those who did (college students) were using it constantly to communicate with people around the country (and soon the world).

AI (in the general sense) is also obviously useful, but the jury's still out on whether LLMs are the way to get there. Right now their best use-case seems to be cannibalizing the search market. It's not clear whether they can fully displace search engines though, given their stubborn propensity for confabulation.

> these companies would not be better served by staying away and ignoring the temptation.

There is a finite number of hours, dollars, and willpower within an org. Companies are pivoting into becoming AI companies without any understanding of what that means.

For every Netflix there were many Pets.coms and for every Blockbuster there were a lot of Petcos, whose website at the time of the collapse of Pets.com was still just a basic marketing page [0].

It's not that companies shouldn't be wondering if and how LLMs will change their industry, but there's a very real possibility that the answer for the next 20 years is "A bit, but not enough for us to need to drop everything yet. Let's keep watching and see how this pans out."

[0] http://web.archive.org/web/20001109045300/http://www.petco.c...

Kinda agree with you. But not all the techs that get hype live long, just take a look at NFTs.

NFTs are (dumb) products, not tech. AI is more foundational, like transistors or internet or GUI.

s/NFTs/blockchain/

People said the same things about blockchain—that it would be the foundational tech that would power a new iteration of the Internet, a web3.

Not that LLMs are in the same category, they certainly seem to have more utility in certain contexts than I ever saw in blockchain, but OP is correct that not every hyped technology turns into the next internet.

s/blockchain/distributed immutable public ledger

There are real use cases and value from blockchain, but it is far more niche than the crypto and NFT folks wanted to believe. IMO there was a lot of horse-cart thinking. “Blockchain is so amazing any product built on it will be successful!”

Which I guess gets back to AI. But AI is so much more general purpose than secure public ledgers. One is a screwdriver, the other is a million pound press.

even if you look at something like DLSS where “all” it’s doing is giving you a tractable approximation of a difficult optimization problem (weighting temporal samples inside a TAA algorithm), that’s such a broadly-applicable approach that it alone would probably change society. The value of increasing shipping or rail efficiency by 5-10% alone is worth billions, and there’s probably thousands of smaller optimization tasks it can be adapted to.

The LLM stuff is an even bigger pot of gold potentially, a workable semantic embedding for semantic queries or other things is another thing that a million improvements can be shaken out of.

it’s actually bizarre to me how much people stand on the “no commercial value!!!” thing, and I have to view some of it as just being the next round of histrionics from the same people who are mad their deviantart got scraped etc.

And everyone with any sense said blockchains were stupid tech.

So many people have been following the hype of foolish tech trends that they can no longer differentiate them from actual ground shifting technology.

It's sad what blockchain hype has done to a supposed tech audience. To this day all nobody can describe an actual large impactuf use for blockchains other than get rich quick. Even with the tech there it does nothing for day to day life.

Duplicating portions of human intelligence has tremendous practical use in every facet of society. The tech just needs to get there.

It's pet rocks vs electricity.

> Duplicating portions of human intelligence has tremendous practical use in every facet of society. The tech just needs to get there.

That last part is the only part that skeptics are skeptical of.

GPT-4 is amazing in a lot of ways and shouldn't be discounted, but the grandest predictions of AI's impact depend on the exponential growth we saw in 2022-2023 continuing. What we've seen this year (GPT-4o, increasing cries from the established players for governments to build them a moat) suggests that that's not happening. We seem to have hit a cap on capabilities and have started just making it smaller and faster.

This is great and will be useful, but GPT-4-level tech isn't going to revolutionize everything the way that the hype suggests. We need something more.

> And everyone with any sense said blockchains were stupid tech.

And yet practically every big company was falling over itself to have a ‘blockchain strategy’ (remember IBM’s thing?) Sound familiar?

> Duplicating portions of human intelligence has tremendous practical use in every facet of society. The tech just needs to get there.

Trouble is, that’s a _huge_ ‘just’.

Well it took Netflix years and years to reach the current point though.

Lots of companies that "chose" the internet first and foremost died, I mean, that's what the Dot-com bubble was all about. There was also a lot, a lot of money spent on networking/fiber around that same time, with the "winners" not being the people that actually had put in the money, but companies like Google, Facebook and the same Netflix about a decade later.

If blockbuster had waited and then just copied Netflix, they'd still be around.

They failed to do either one.

They had an opportunity to buy netflix for $50 million after the dot com bust.

It’s so different though, I don’t see why we even compare the internet and AI. The internet connected people, of course there would be money in that. AI cuts out people, how will that make money, except for the very few people at the very top controlling the marionette strings? And now all the people without a job have no money to buy products or services? That doesn’t bode well for any market.

You’re saying they’re different because good is not like bad. But companies don’t business plan from moral judgments.

AI will transform production , distribution, and consumption at least as much as the Internet did.

> And now all the people without a job have no money to buy products or services?

AI isn't at that level yet, and I don't know when it will be. When it does get that good, the meaning of wealth changes at least as much as it did in the Industrial Revolution when land took a second seat to capital, and I have no idea what our societies will do any more than Adam Smith could have imagined Communism.

>I think you could go back in time and s/AI/internet.

That's a great example. You could also include systemic events like the introduction of the microcomputer, smartphone, and even "Web 2.0." AI looks like it will be a systemic change, we just don't know what that change is yet. Companies that find the winning formula will have a competitive advantage. Companies that ignore it will start a downward spiral that will be difficult to survive.