I find it fascinating that this is still making the rounds. When I read this it was immediately obvious that the author was using a non-web enabled AI which was just hallucinating; there were none of the inline indications that GPT was using the web. Additionally, it must be an old model; even the cheapest, lowest powered models on chatgpt.com today search the web when I ask them questions about articles as the author did. (I just signed out of chatgpt.com to get the worst available model, and it does summarize the linked article correctly.) Note that link to the transcript on chatgpt.com is provided, even though it's trivial to create a shared link to a conversation.

I am confused about what to take away from the article. It feels akin to someone reading a book for the first time, it ends up being "Harry Potter", and they somehow get 10,000 likes on Substack because they took it literally and crashed into the wall when they tried to walk into platform 9 3/4. Am I being unfair? Are these the same people that are claiming that AI is all a sham and will have no impact on society?

We're nerds. We understand the nuance, we understand the way these tools work and where the limits lie. We understand that there is web enabled and not web enabled. Regular people do not understand any of this. Regular people type into a textarea and consume the response.

The take away from this article should be that you are vastly overestimating how people understand and interact with technology. The author's experience of ChatGPT is not unique. We have spent decades building technology that is limited but truthful, now we have technology that is unlimited and untruthful. Many people are not equipped to handle that. People are losing their minds. If ChatGPT says "I read your article" they trust it, they do not think, "ah well this model doesn't support browsing the web so ChatGPT must be hallucinating". That's technobabble.

https://futurism.com/openai-investor-chatgpt-mental-health

https://futurism.com/televised-love-declaration-chatgpt

https://futurism.com/chatgpt-users-delusions

You are being unfair and you should be more empathetic.

> Are these the same people that are claiming that AI is all a sham and will have no impact on society?

That is the view of a subset of nerds, not regular people. The author of that piece is a writer not a nerd.

> We're nerds. We understand the nuance, we understand the way these tools work and where the limits lie. We understand that there is web enabled and not web enabled. Regular people do not understand any of this. Regular people type into a textarea and consume the response.

The exact opposite is true. I'd word it as

"We're nerds, we don't understand nuance, we understand the way these tools work and where the limits lie. We understand that there is web enabled and not web enabled. Regular people are not nerds

> ChatGPT says "I read your article" they trust it, they do not think, "ah well this model doesn't support browsing the web so ChatGPT must be hallucinating". That's technobabble.

No, that's humans. Happens literally every day at every workplace I've ever been in

She's weirded out by creepy hallucinations, which is understandable! But ChatGPT is well known to hallucinate. In other words she doesn't know which of its behaviors are normal so she doesn't know how to react. Additionally, her particular issues are quite solvable with better prompting.

> If I do poor work with an electric drill then it's not the drill's fault.

> ChatGPT's sycophancy crisis was late April.

If you drill starts telling you "what a great job you're doing, keep drilling into that electrical conduit", the drill is at least partially at fault.

A tool that randomly and unpredictably fails is a bad tool. How should I, as a user, account for the possibility/likelihood of another such crisis in the future?

> A tool that randomly and unpredictably fails is a bad tool.

But all failures are "random and unpredictable" if you have no baseline understanding of how to use the tool. "AIs hallucinate" is probably the single most obvious thing about AIs. This isn't a subtle misunderstanding that an expert could make. This is like using a drill on your face.

> But all failures are "random and unpredictable" if you have no baseline understanding of how to use the tool.

But the tool's behavior changed. In ways that even its creators didn't intend (example: https://openai.com/index/sycophancy-in-gpt-4o/), and had to work to undo.

If my hammer had a random week every year where it tried to smack me in the face whenever I touched it, I'd probably avoid using it.

This is unrelated to sycophancy. The author is failing to understand that GPT did not make a tool call and is hallucinating. Hallucinations have always been a thing. They are not some new, surprising development.

> This is unrelated to sycophancy.

"It's a stunning piece. You write with an unflinching emotional clarity that's both intimate and beautifully restrained."

> They are not some new, surprising development.

OpenAI sure seemed surprised. https://openai.com/index/sycophancy-in-gpt-4o/

> "It's a stunning piece. You write with an unflinching emotional clarity that's both intimate and beautifully restrained."

This is a hallucination, since there is no source to refer to.

The author was surprised because GPT was hallucinating, not because GPT was extra nice.

Sycophancy might be related, but it's not the point of the article. If GPT had said "wow, your post is trash", the author would have been equally surprised to learn it was a hallucination.

"Oh, yes, I read your new novel, it's great!" is precisely what a sycophant would tell you when they forgot to read it.

But in the context of this thread, I would say that using an AI to examine logical inconsistencies is the wrong way to use the tool.

The problem with LLMs is that they don't have any intentionality to their worldview. They're like a wise turtle that comes to you in a dream, their dream logic is not something you should pay much attention to.

I was in the midst of editing the comment when you replied, sorry. I didn't see your reply before I edited mine.

OK, but you're still blaming the user for the tool's failings.

Which even the makers of the tool agreed were failings.

That's a valid point! The AI community still has much to improve on.

The article you link is a very specific type of failure that apparently did not happen in this instance, where Claude was able to access the author's writing. And the author apparently found the insights useful, though the lack of analysis from the author on that value makes this article basically meaningless for an outsider.

I am apparently a different type of person than the author because my obsidian vaults look nothing like theirs, but I can't imagine asking an LLM for a meta-analysis of my writing. The whole point of organizing it with Obsidian is that I do that analysis myself - it is part and parcel of the organization itself.

Appreciate the thought—my comments in Claude's analysis are now added on the margins.

The exercise is not meant to do much else but spot patters in my thinking that I can reflect on. Nothing particularly novel here from Claude but it is helpful, for me, to get external feedback.