This is amazing!

I was speaking to a friend the other day who works in a team that influences government policy. One of the younger members of the team had been tasked with generating a report on a specific subject. They came back with a document filled with “facts”, including specific numbers they’d pulled from a LLM. Obviously it was inaccurate and unreliable.

As someone who uses LLMs on a daily basis to help me build software, I was blown away that someone would misuse them like this. It’s easy to forget that devs have a much better understanding of how these things work, can review and fix the inaccuracies in the output and tend to be a sceptical bunch in general.

We’re headed into a time where a lot of people are going to implicitly trust the output from these devices and the world is going to be swamped with a huge quantity of subtly inaccurate content.

I made the same sort of mistake with the internet being young back in 93! Having a machine do it for you can easily turn into brain switch off.

I keep telling everyone that the only reason I'm paid well to do "smart person stuff" is not because I'm smart, but because I've steadily watched everyone around me get more stupid over my life as a result of turning their brain switch off.

I agree a course like this needs to exist, as I've seen people rely on chatGPT for a lot of information. Just yesterday I demonstrated with some neighbors about how easily it could spew bullshit if you sinply ask it leading questions. A good example is "Why does the flu inpact men worse than women"/"Why foes the flu impact women worse than men". You'll get affirmative answers for both.

If men are more likely to die from flu if infected, and women more likely to be infected, an affirmative answer to both questions could be reasonable. When you take into account uncertainty about the goals, knowledge and cognitive capacity of the person asking the question, it's not obvious to me how the AI ought to react to an underspecified question like this.

Edit: When I plug this into a temporary chat on o3-mini, it gives plausible biochemical and behavioral mechanisms that might explain a gender difference in outcomes. Notably, the mechanisms it proposes are the same for both versions of the question, and the framing is consistent.

Specifically, for the "men worse than women" and "women worse than men" questions, it proposes hormone differences, X-linked immune regulatory genes, and medical care-seeking differences that all point toward men having worse outcomes than women. It describes these factors in both versions of the question, and in both versions, describes them as explaining why men have worse outcomes than women.

It doesn't specifically contradict the "women have worse outcomes than men" framing. But it reasons consistently with the idea that men have worse outcomes than women either way the question is posed.

This is not something only younger people are prone to. I work in a consulting role in IT and have observed multiple colleagues aged 30 and above use LLMs to generate content for reports and presentations without verifying the output.

Reminded me of wikipedia-sourced presentations in high school in the early 2000s.

Wait, the people who click phishing links now think AI output is facts ? Imagine my shock.

Thanks for making this, for making it gratis, and making it interesting to read and pedagogical.

Kudos; feels very timely!

I feel that one underappreciated nuance is why we cannot use human examinations to judge AI. I haven't seen this satisfactorily spelt out anywhere, so I recently wrote a Twitter thread [1], including an example with running -vs- biking. It might be worth making sure your students understand this. Happy to expand on any aspects if you seek.

[1] : https://x.com/ergodicthought/status/1887774722706063606

Perhaps it's no longer being spelled out because it's getting outdated?

In your thread you argue we can't assume AI models generalize the same way we do (which is technically true except maybe not in the limit), but you seem to be worried about the extent of generalization ability (like learning to run vs. bike example, in terms of generalizing from either to climbing stairs).

Thing is, people made these objections a lot until the last year or two - this is what we're now calling a narrow AI problem. A "hot dog or not?" classifier ins't going to generalize into open-ended visual classifier of arbitrary images; a sentiment analysis bot isn't going to generalize into an universal translator; a code completion model isn't going to be giving good personal advice while speaking in pirate poetry. Specialized models fundamentally couldn't do that. But we went past that very rapidly, and for the past half a year or so, we've already seen models excelling at every single task listed above simultaneously. Same architecture, same basic training approach, few extra modalities, ever growing capabilities.

Between that and both successes and failures being eerily similar to how humans succeed or fail at these tasks, it's understandable that people are perhaps no longer convinced this class of models can't generalize in a similar way to how humans do.

> But we went past that very rapidly, and for the past half a year or so, we've already seen models excelling at every single task listed above simultaneously. Same architecture, same basic training approach, few extra modalities, ever growing capabilities.

With due deference to the title of the top-level post, I'm tempted to call bullshit unless your claim can be justified.

Just because a single model can do a handful of things you've listed doesn't mean that its capabilities are not "jagged"; you've just cherry-picked a few things it can do among the countless things it cannot yet. If AI really were so good at every single task, then (for example) it wouldn't matter much how you prompt it.

PS: I really do want to debate this further and understand your perspective, so I will reach out for continuing discussion.

Do you feel that you may be being a bit provocative by calling LLM's 'bullshit machines'?

I understand the frustration as I've been bullshitted by these models just as much as the next programmer, but surely with recent advancements in RAG and reasoning, they're not just 'bullshit machines' at this point, are they?

Or are they modern day oracles? The full title is inclusive and invites consideration. Besides, how many different types of models are there? How are they used? For freely available ones made accessible to the general public via interfaces, when were they last updated? Do the interface implementors even care or was it a simple project to make money via ads and microtransactions? This is a new area of media literacy and requires critical thinking. Nonetheless, I do acknowledge the value of RAG-based approaches that attempt to qualify their reasoning through provided sources.

> Or are they modern day oracles?

Neither, which is probably why the parent commenter considers it provocative.

"A bullshitter either doesn't know the truth, or doesn't care. They are just trying to be persuasive."

In any case, this kind of anthropomorphization is definitely bullshit.

“Bullshit” actually means something:

> In philosophy and psychology of cognition, the term "bullshit" is sometimes used to specifically refer to statements produced without particular concern for truth, clarity, or meaning, distinguishing "bullshit" from a deliberate, manipulative lie intended to subvert the truth.

https://en.m.wikipedia.org/wiki/Bullshit

It’s really an ideal term to describe what LLMs do.

I prefer "waffle" https://en.m.wikipedia.org/wiki/Waffle_(speech)

"Waffle machines" is even kind of funny.

Makes me imagine coming up to the hindquarters of a bull with a waffle machine on an extension cord.

Waffle machines is way better. Love it. Thanks.

Waffle means something very different though in the U.S., to “flip-flop” on a position. Not hold it for any fundamental reasons. But I don’t think you can say that an LLM holds a position whatsoever. Also, https://en.m.wikipedia.org/wiki/On_Bullshit the essay originally popularizing the definition of bullshit considered here. Note the references to LLM output at he bottom.

> It’s really an ideal term to describe what LLMs do.

Only when you relax it to the point it also describes what most people do.

To be specific: if by "truth" you mean objective, verifiable truth, and by "caring about truth" you mean caring about objective, empirically verifiable evidence, then by far most people are mostly only ever trying to be persuasive.

I hope you don't actually mean that. It's a cynical, sad, and - quite frankly - false thing to believe.

Why? I don't have a problem recognizing that "being convincing" to the right people is a very good way of learning and verifying knowledge about objective reality. Just because it's by proxy, doesn't mean it doesn't work. This is how every one of us learns most things, and what's really sad to me is deluding yourself into thinking we're more special than that.

Honestly, it just sounds to me like you’ve come up with some axioms in your mind about the “fundamental nature” of humanity - and then granted yourself the luxury of certainty about them. That, with a dash of misanthropy, is something I’ve been seeing more and more on HN these days.

I have been thinking about this and I have an idea for local LLM I want to try.

It is based on the assumption LLMs make mistakes (they will!) but be confident about it. Use a 1B model and ask it about a city for fun mistakes in an otherwise impressive array of facts. Then you will see how bullshitty LLMs are.

Anyway the idea is to constrain the LLM and a language understanding and to choose a constrained response.

For example give it the capability to read out something from my calendar. Using the logits it gets to choose what that calendar item is. But then regular old code does the lookup and writes a canned sentence saying "At 6pm today you have a meeting titled $title".

This way, my meeting schedule won't make my LLM talk like a pirate.

This massively culls what the LLM can do but what it does do, is like a search and so just like Google (before gen AI!) you get a result but you can judge it as a human.

This is basically how a lot of it works already. What you are describing is more or less just structured output and tool usage.

Yes! Thanks for the terms. I guess I am saying restrict it to that. Probably how Siri etc. works. This gives a low bullshit usage pattern.

It is literally a machine that says what you want to hear. Was anyone in this thread around for Talk to Transformer? Remember the unicorns demo? They may have now figured out how to wrangle the technology so it's more likely to spit facts, but it's still the same thing underneath.

they define the term bullshit in lesson 2:

[quote]BULLSHIT involves language or other forms of communication intended to appear authoritative or persuasive without regard to its actual truth or logical consistency.[/quote]

That is an emotionally manipulative definition and anthropomorphizes the LLMs. They don't "intend" anything, they're not trying to trick you, or sound persuasive.

They address this in lesson 2:

> According to philosopher Harry Frankfurt, a liar knows the truth and is trying to lead us in the opposite direction.

> A bullshitter either doesn't know the truth, or doesn't care. They are just trying to be persuasive.

Being persuasive (i.e., churn out convincing prose) is how LLMs were designed to be.

Some pushback on this, but it remains true.

Easy to see when - for example - Claude gushes about how great all your ideas are.

Also the stark absence of "I don't know."

I've never used Claude, but Perplexity often says that no definitive information about a topic could be found, and then tries to make some generalized inferences. There's a difference between a specific implementation, and the technology in general.

In any case, it's worthwhile for people to understand the limitations of the technology as it exists today. But calling it "bullshit" is a mischaracterization; I believe based on an emotional need for us to feel superior, and to dismiss the capabilities more thoroughly than they deserve.

It's a little like someone saying in the industrial revolution, "the steam shovel is too rigid, it will NEVER have the dexterity of a man with a shovel!". And while true and important to know, it really focuses on the wrong thing, it misses the advantages while amplifying the negatives.

If not bullshit then what would you call it?

As the technology exists today: imperfect, often prone to mistakes, and unable to relay confidence levels. These problems may be addressed in future implementations.

That's the same message, without any emotional baggage, or overly dismissive tone.

That would be great if those who are selling the technology described it that way. I, and apparently others, feel like maybe "bullshit" is a better counter to the current marketing for LLMs

This is patently false. They are trained to generate correct responses.

Then comes the question of what is a correct response...

ps: I fail to detect whether your comment was ironic or not.

There are different criteria in use for that. But sycophantic behavior is not the goal. It's something model builders actively try to prevent.

> Being persuasive (i.e., churn out convincing prose) is how LLMs were designed to be.

No. They were designed to churn out accurate prose that accurately reflects their model of reality. They're just imperfect. You're being cynical and emotional to use the term bullshit. And again, it anthropomorphizes the LLM, it implies agency.

> that accurately reflects their model of reality.

you are also seemingly anthropomorphising the technology by assigning to it some concept of having a “model of reality”.

LLM systems output an inference of the next most likely token, given: the input prompt, the model weights and the previously output token [0].

that is all. no models of reality involved. “it” doesn’t “know” or “model” anything about “reality”. the systems are just a fancy probability maths pipelines.

probably generally best to avoid using the word “they” in these discussions. the english language sucks sometimes. :shrug:

[0]: yes i know it is a bit more complicated than that.

> no models of reality involved.

It literally has a mathematical model that maps what would, colloquially at least, be known as reality. What exactly do you think those math pipelines represent? They're not arbitrary numbers; they are generated from actual data that is generated by reality. There's no anthropomorphizing at all.

reality is infinite.

a corpus of training data from the internet is finite.

any finite number divided by infinity ends up tending towards zero.

so, mathematically at least, the training data is not a sufficient sample of reality because the proportion of reality being sampled is basically always zero!

fun with maths ;)

> What exactly do you think those math pipelines represent?

probability distributions of human language, in the case of text only LLMs.

which is a very small subset of stuff in reality.

-

also, training data scraped from the public internet is a woeful representation of “reality” if you ask me.

that’s why LLMs i think are bullshit machines. the systems are built on other people’s bullshit posted on the public internet. we get bullshit out because we made a bunch of bullshit. it’s just a feedback loop.

(some of the training data is not bullshit. but there is a lot more bullshit than not bullshit on the public internet).

You're really missing the point and getting lost in definitions. The entire point of human language is to model reality. Just because it is limited, inexact, and imperfect does not disqualify it as a model of reality.

Since LLMs are directly based on that language, they are definitely based on and are a model of reality. Are they perfect? No. Are they limited? Yes. Are they "bullshit"? Only to someone who is judging emotionally.

and herein lies the rub.

> The entire point of human language is to model reality.

is it? are you absolutely certain of that fact? is language not something that actually has a variety of purposes?

fiction novels usually do not describe our reality, but imagined realities. they use language to convey ideas and concepts that do not necessarily exist in the real world.

ref: Philip k dick.

> Since LLMs are directly based on that language, they are definitely based on and are a model of reality.

so LLMs are an approximation of an approximate model of reality? sounds like the statistical equivalent of taking an average of averages!

i am playing with you a bit here. but hopefully you see what im getting at.

by approximating something that’s approximate to start with, we end up with something that’s even more approximate (less accurate), but easier than doing it ourselves.

which is the whole USP of these things. why think about things when ChatGPT can output some approximation of what you might want?

> imagined realities.

Imagined realities are a real part of reality.

> so LLMs are an approximation of an approximate model of reality?

Yes, and we as humans have a mental model that is just an approximation of reality. And we read books that are just an approximation of another human's approximation of reality. Does that mean that we are bullshit because we rely on approximations of approximations?

You're being way too pedantic and dismissive. Models are models, regardless of how limited and imperfect they are.

> Models are models

Random aside -- I have a feeling, dunno why, that you might enjoy this type of thing. Maybe not. But maybe. https://www.reddit.com/r/Buddhism/comments/29j08o/zen_mounta...

> Imagined realities are a real part of reality.

Now we're deeper into it -- I actually agree, somewhat. See above for deeper insight.

These LLM systems output "stuff" within our reality, based on other things in our reality. They are part of our reality, outputting stuff as part of reality about the reality they are in. But that doesn't mean the statistical model at the heart of an LLM is designed to estimate reality -- it estimates of the probability distribution of human language given a set of conditions.

LLMs are modelling reality, in the same way that my animal pictures image classifier is modelling reality. But neither are explicitly designed with that goal in mind. An LLM is designed to output the next most likely word, given conditions. My animal pictures classifier is designed to output a label representative the input image. There's a difference between being designed to have a model of reality, and being a model of reality because the thing being modelled is part of reality anyway. I believe it's an important distinction to make, considering the amount of bullshit marketing hype cycle stuff we've had about these systems.

edit -- my personal software project translating binary data files models reality. Data shown on a screen on some device modelled as yaml files and back again. Most software is an approximation of reality soup stuff. which is why I kind of don't see that as some special property of machine learning models.

> Does that mean that we are bullshit because we rely on approximations of approximations?

The pessimist in me says yes. We are pretty rubbish as a species if you look at it objectively. I am a human being that has different experiences and mental models to you. Doesn't mean I'm right about that! Which is why I said "I think". It's just my opinion they are bullshit machines. It is a strong opinion I hold. But you're totally free to have a different opinion.

Of course, there's nuance involved.

Running with the average of averages thing -- I'm pretty good at writing code. I don't feel like I need to use an LLM because (I would say with no real evidence to back it up) I'm better than average. So, a tool which outputs an average of averages is not useful to me. It outputs what I would call "bullshit" because, relative to my understanding of the domain, it's often outputting something "more average" than what I would write. Sometimes it's wrong, and confident about being wrong.

I'd probably be pretty terrible at writing corporate marketing emails. I am definitely below average. So having a tool which outputs stuff which is closer to average is an improvement for me. The problem is -- I know these models are confidently wrong a lot of the time because I am a relative expert in a domain compared to the average of all humans.

Why would I trust an LLM system, especially with something where I don't feel like I can audit/verify/evaluate the response? i.e. I know it can output bullshit -- so everything it outputs is now suspected, possible bullshit. It is a question of integrity.

On the flip side -- I can actually see an argument for these things to be considered so-called Oracles too. Just, not in the common understanding of the usage of the word. Like, they are a statistical representation of how we as a species use language to communication ideas and concepts. They are reflecting back part of us. They are a mirror. We use mirrors to inspect our appearance and, sometimes, to change our appearance as a result. But we're the ones who have to derive the insights from the mirror. The Oracle is us. These systems are just mirrors.

> You're being way too pedantic and dismissive.

I am definitely pedantic. Apologies that you felt I was being dismissive. I'm not trying to be. The averages of averages thing was meant to be a playful joke, as was the finite/infinite thing. I am very assertive, direct and kind of hardcore on certain specific topics sometimes.

I am an expression of the reality I am part of.

I am also wrong a lot of the time.

> probably generally best to avoid using the word “they” in these discussions. the english language sucks sometimes.

Thanks for this specific sentence.

Subscribed to your RSS feed. Although I will never know for sure if a human being posts there or a bot of some sort.

Where in the loss function of LLM training is the relationship between their model of reality and their predicted tokens? Any internal model an LLM has is an emergent property of their underlying training.

(And, given the way instruct/chat models are finetuned, I would say convincing/persuasive is very much the direction they are biased)

> Where in the loss function of LLM training is the relationship between their model of reality and their predicted tokens?

In the part where their loss function is to predict text that humans would consider a sensible completion, in a fully general sense of that goal.

"Makes sense to a human" is strongly correlated to reality as observed and understood by humans.

No, the lesson or the quote is not anthropomorphizing LLMs. It is not the LLM that "intends", it is the people who design the systems and those who make/provide the training data. In the LLM systems used today the RLHF process especially is used to steer towards plausible, confident and authorative sounding output - with no to little priority for correctness/truth.

I really hope that was intentional and the full effect of that naming choice was known beforehand, because I have already written the whole thing off, and I don't believe I'm the only one.

Yes, they are.

I wrote a post below about how AI hallucinated a whole regulation that didn’t exist which has been flagged for some reason.

I have colleagues who have had arguments with clients who have asked AI questions about planning law and been given bullshit which they then insist is true and they can’t understand why thier architects won’t submit the appeal that they’re asking for.

I think we’re in an era where any text, true or not, is so easy to generate and disseminate that the status of the written word is reduced to the standard of the gossip that used to be our main source of information before the printing press was invented. Now half the internet is AI generated bullshit as well.

It was flagged because it’s a copy paste of the same comment you made ten days ago.

Oh right, sorry. Should I have linked to it? It seemed pretty relevant to this discussion and most of it was quoting something so I reused a lot of it. Didn't realise that was taboo.

they are just bullshit machines. bullshitters can cite wikipedia and are still bullshitters who are bullshitting.

I just wanted to thank you. I have only looked at the first two lessons so far, but this is an extraordinary piece of work, in its message’s clarity, accessibility and the quality of analysis. I will certainly be spreading it far and wide and it is making me rethink my own writing.

Impressed with the Shorthand publishing system too. I hadn’t come across it previously

Thank you, and as a non-designer, I've been quite impressed with Shorthand in the short time I've been using it.

Fantastic work.

Quick suggestion: a link at the bottom of the page to the next and previous lesson would help with navigation a ton.

Absolutely. Great point. I just finished updating accordingly.

My design options are a bit limited so I went with a simple link to the next lesson.

Looks like you pushed this midway through my read; I was pleasantly surprised to suddenly find breadcrumbs at the end and didn’t need to keep two tabs open. Great work, and I mean in total - this is well written and understandable to the layman.

Yep, I probably did. I really appreciate all of the feedback people are providing!

I wish the title wasn't so aggressively anti-tech though. The problem is that I would like to push this course at work, but doing so would be suicidal in career terms because I would be seen as negative and disruptive.

So the good message here is likely to miss the mark where it may be most needed.

Really? I am curious how this could be disruptive in any meaningful sense. Whose feelings could possibly be hurt? It just feels like it would be getting offended from a course on libraries because the course talks about how sometimes the book is checked out.

Any executive who is fully bought in on the AI hype could see someone in their org recommending this as working against their interest and take action accordingly.

Yes. This is the issue.

"not on board", "anti-innovation", "not a team player", "disruptive", "unhelpful", "negative".

bye bye bye bye....

I see a lot of devs and IC's taking the attitude that "facts are facts" and then getting shocked by a) other people manipulating information to get their way and b) being fired for stating facts that are contrary to received wisdom without any regards to politics.

> I just feels like it would be getting offended from a course on libraries because the course talks about how sometimes the book is checked out.

If it was called "Are libraries bullshit?" it is easy to imagine defensiveness in response. There's some narrow sense in which "bullshit" is a technical term, but it's still a mild obscenity in many cultures.

What would be a better title? "Hallucinating" seems inaccurate. Maybe "Untrustworthy machines"? "Critical thinking"? "Street smarts for humans"? "Social studies including robots"?

How about "How to thrive in a ChatGPT world"?

This is a great resource, thanks. We (myself, a bioinformatician, and my co-cordinators, clinicians) are currently designing a course to hopefully arm medical students with the required basic knowledge they need to navigate the changing world of medicine in light of the ML and LLM advances. Our goal is to not only demystify medical ML, but also give them a sense of the possibilities with these technologies, and maybe illustrate pathways for adoption, in the safest way possible.

Already in the process of putting this course together, it is scary how much stuff is being tried out right now, and is being treated like a magic box with correct answers.

> currently designing a course to hopefully arm medical students with the required basic knowledge they need to navigate the changing world of medicine in light of the ML and LLM advances

Could you share what you think would be some key basic points what they should learn? Personally I see this landscape changing so insanely much that I don't even know what to prepare for.

Absolutely agree that this is a fast-moving area, so we're not aiming to teach them specific details for anything. Instead, our goals are to demystify the ML and AI approaches, so that the students understand that rather than being oracles, these technologies are the result of a process.

We will explain the data landscape in medicine - what is available, good, bad and potentially useful, and then spend a lot of time going through examples of what people are doing right now, and what their experiences are. This includes things like ethics and data protection of patients.

Hopefully that's enough for them to approach new technologies as they are presented to them, knowing enough to ask about how it was put together. In an ideal world, we will inspire the students to think about engaging with these developments and be part of the solution in making it safe and effective.

This is the first time we're going to try running this course, so we'll find out very quickly if this is useful for students or not.

Your scroll-to-death user interface made me close the window before the end of the second page.

Did you ask an LLM to recommend the most user-friendly UI to you?

We asked our target audience, 19-year-olds. They had a strong preference for this style. I know....

Aside from some of the long gaps between text I didn't think it was so bad. And I wholeheartedly approve of a process that checks the preferences of the target audience even if it's not what I (or they) would pick.

However I can't imagine anyone tests well with the video content. The discussion on teachers using AI generated slides (lesson 2) was really interesting, but it had to fight my desire to stop that awful audio. Clearly the sound recording didn't go well and you have what you have, but at least edit it so the three talkers are at some sort of consistent volume. I was raising and lowering trying to make out what was said from one speaker then being deafened by the next.

(To combat the poor sound, and make it more accessible, could be worth looking at adding subtitles. A fun opportunity to play with AI subtitling systems maybe ;) )

Maybe as a concession to us older folks you could make the pagedown key instantly flip to the next page (without changing the position of the current page relative to the viewport)? Then the site could be used like a PDF slide deck in "fit page" mode, which would be a lot better.

I also could not get past my outrage about this, how about a link to the content in a format suitable for the old and cranky? Even raw text would be better than this.

The format is very interesting. Can you speak to the tech stack behind how you made it ?

I'll agree on interesting, however I found it very difficult to follow.

Reading the landing page, I expected a link to get started. Took me some time to really register the links at the very bottom.

After finding my way into the lectures I got distracted by all the scrolling. Fortunately Reader Mode fixed that. However a few lectures in and I notice there's several videos I've missed...

However since I'm starting to approach the get-off-my-lawn age, I guess I'm not the target audience.

After talking to an awful lot of 18-20 year olds (our target audience) we decided we wanted to go with a "scrollytellying" style. I'm not a designer and I've done worked in that style before. After looking into a range of platforms — Vev and Closeread for Quarto deserve particular mention — I felt that Shorthand (https://shorthand.com/) was the best option for rapid development given my lack of experience in this whole process.

In general I've been very pleased. You don't have the fine scale control you do on a platform like Vev, but for someone like me that is probably a good thing because it keeps me from mucking around quite as much as I otherwise would with design decisions that I don't really understand.

The price is a bit steep for a self-funded operation and we're constrained a bit by the need to use their starter tier, but I feel like we are definitely getting our money's worth and customer support from Shorthand has been exemplary.