What are people legitimately worried about LLMs doing by themselves? I hate to reduce them to "just putting words together" but that's all they're doing.

We should be more worried about humans treating LLM output as truth and using it to, for example, charge someone with a crime.

People are already just hooking LLMs up to terminals with web access and letting them go. Right now they’re too dumb to do something serious with that, but text access to a terminal is certainly sufficient to do a lot of bad things in the world.

It's gotta be tough to do anything too nefarious when your short-term memory is limited to a few thousand tokens. You get the memento guy, not an arch-villain.

Until the agent is able to get access to a database and persist its memory there...

In a similar way to the way humans keep important info in their email inbox, on their computer, in a notes app in their phone, etc.

Humans have a shortish and leaky context window too.

Leaky yes, but shortish no.

As a mental exercise, try to quantify the amount of context that was necessary for Bernie Madoff to pull off his scam. Every meeting with investors, regulators. All the non-language cues like facial expressions and tone of voice. Every document and email. I'll bet it took a huge amount of mental effort to be Bernie Madoff, and he had to keep it going for years.

All that for a few paltry billion dollars, and it still came crashing down eventually. Converting all of humanity to paperclips is going to require masterful planning and execution.

That’s called RAG, and it still doesn’t work as well as you might imagine.

The contexts are pretty large now

Your nefarious plan for enslaving humanity is still unlikely to fit into 128k tokens.

Operational success does not hinge on persisting the entire plan in working memory, that's what notebooks and word docs are for.

128k is table stakes now, regardless. Google's models support 1 million tokens and 10 million for approved clients. That is 13x War and Peace, or 1x the entire source code for 3D modeling application Blender.

Yeah, but most LLMs are barely functional after 16k tokens, even if it says 128k on the tin. Sure, they will have recall, but the in-context reasoning ability drops dramatically.

LLMs just aren't smart enough to take over the world. They suck at backtracking, they're pretty bad at world models, they struggle to learn new information, etc. o1, QwQ, and CoT models marginally improve this but if you play with them they still kinda suck

I kinda agree with him. But I have also seen if we do the right COT, they perform much better than any other technique. I found this good blog on Reddit which is about COT: https://hub.athina.ai/blogs/what-is-chain-of-thought-prompti...

Millions of people are hooked up to a terminal as well.

> their best researchers are being hired to other companies

I agree about the OpenAI moat. They did just get 5 Googlers to switch teams. Hard to know how key those employees were to Google or will be to OpenAI.