Some great grand ancestor of mine was a civil servant, a great achievement given his peasant background. The single skill that enabled it was the knowledge of calligraphy. He went to school and wrote nicely and that was sufficient.

The flip side was, calligraphy was sufficient evidence for both his education to whoever hired him, and for a recipient of a document, of its official nature. Calligraphy itself or course didn't make him efficient or smart or fair.

That's long gone of course, but we had similar heuristics. I am reminded of the Reddit story about an AI-generated mushroom atlas that had factual errors and lead to someone getting poisoned. We can no longer assume that a book is legit simply because it looks legit. The story of course is from reddit, so probably untrue, but it doesn't matter - it totally could be true.

LLMs are fantastic at breaking our heuristics as to what is and isn't legit, but not as good at being right.

> We can no longer assume that a book is legit simply because it looks legit.

The problem is that this has been an issue for a long time. My first interactions with the internet in the 90s came along with the warning "don't automatically trust what you read on the internet".

I was speaking to a librarian the other day who teaches incoming freshman how to use LLMs. What was shocking to me is that the librarian said a majority of the kids trust what the computer says by default. Not just LLMs, but generally what they read. That's such a huge shift from my generation. Maybe LLM education will shift people back toward skepticism - unlikely, but I can hope.

One of the issues today is the volume of content produced, and that journalism and professional writing is dying. LLMs produce large amounts of "good enough" quality to make a profit.

In the 90s we could reasonably trust that that the major news sites and corporate websites was true, while random forums required a bit more critical reading. Today even formerly trusted sites may be using LLMs to generate content along with automatic translations.

I wouldn't necessarily put the blame on LLMs, this just make it easier. The trolls and spammers was always there, now they just have a more powerful tool. The commercial sites now have a tool they don't understand, which they apply liberally, because it reduces cost, or their staff use it, to get out of work, keep up with deadlines or to cover up incompetence. So, not the fault of the LLMs, but their use is worsening existing trends.

> Today even formerly trusted sites may be using LLMs to generate content along with automatic translations.

Yep - or they're commingling promotional content with their journalism, a la Forbes / CNN / CNET / About.com / etc. There's still quality content online but it's getting harder to find under the tidal wave of garbage.

> I was speaking to a librarian the other day who teaches incoming freshman how to use LLMs. What was shocking to me is that the librarian said a majority of the kids trust what the computer says by default. Not just LLMs, but generally what they read. That's such a huge shift from my generation.

I think that previous generations were not any different. For most people, trusting is the default mode and you need to learn to distrust a source. I know many people who still have not learned that about the internet in general. These are often older people. They believe insane things just because there exists a nicely looking website claiming that thing.

Not sure of the context here is for "previous generation" but I've been around since early in the transition from university/military network to public network, and the reality was the internet just wasn't that big, and it was primarily made up of people who looked, acted and valued the same things.

Now it's not even the website of undetermined providence that is believed; positions are established based on just the headline, shared 2nd or 3rd hand!

> The problem is that this has been an issue for a long time. My first interactions with the internet in the 90s came along with the warning "don't automatically trust what you read on the internet".

I received the same warnings, actually it was more like “don’t trust everything you read on the internet”, but it quickly became apparent that the last three words were redundant, and could have been rephrased more accurately as “don’t trust everything you read and hear and see”.

Our parents and teachers were living with their own fallacious assumptions and we just didn’t know it at the time, but most information is very pliable. If you can’t change what someone sees, then you can probably change how they see it.

I feel like there was also a brief window where "many amateur eyes in public" trumped "private experts"; wikipedia, open source software, etc. This doesn't seem the case in a hyper-partisan and bifurcated society where there is little trust.

Believe half of what you see and none of what you hear

True!

> Our parents and teachers were living with their own fallacious assumptions and we just didn’t know it at the time, but most information is very pliable.

Indeed. When I was 14-18 in the UK, the opinion pieces in the news were derogatorily describing "media studies" as "Mickey Mouse studies".

In retrospect such courses were teaching critical analysis of media sources, in much the same way that my history GCSE went into the content of historical media and explored how both primary and secondary sources each had the potential to be either accurate or biased.

Even now, even here, I see people treat the media itself as pure and only the people being reported upon as capable of wrongdoing — e.g. insisting that climate scientists in the 70s generally expected an imminent ice age, because that's what the newspapers were saying.

> Some great grand ancestor of mine was a civil servant, a great achievement given his peasant background. The single skill that enabled it was the knowledge of calligraphy. He went to school and wrote nicely and that was sufficient.

Similar story! Family lore has it that he was from a farming family of modest means, but he was hired to write insurance policies because of his beautiful handwriting, and this was a big step up in the world.

> The story of course is from reddit, so probably untrue, but it doesn't matter - it totally could be true.

What?! Someone just made up something and then got mad at it. This is specially weird when you even acknowledge its a made up story. If we start evaluating new things like this nothing will ever progress.

>That's long gone of course, but we had similar heuristics.

To quote someone about this:

>>All that is solid melts into air, all that is holy is profaned, and man is at last compelled to face with sober senses his real conditions of life.

A book looking legit, a paper being peer reviewed, an expert saying something, none of those things were _ever_ good heuristics. It's just that it was the done thing. Now we have to face the fact that our heuristics are obviously broken and we have to start thinking about every topic.

To quote someone else about this:

>>Most people would rather die than think.

Which explains neatly the politics of the last 10 years.

> To quote someone about this: >>All that is solid melts into air, all that is holy is profaned, and man is at last compelled to face with sober senses his real conditions of life.

So, same as it ever was?

Smoke, nothing but smoke. [That’s what the Quester says.] There’s nothing to anything—it’s all smoke. What’s there to show for a lifetime of work, a lifetime of working your fingers to the bone? One generation goes its way, the next one arrives, but nothing changes—it’s business as usual for old planet earth. The sun comes up and the sun goes down, then does it again, and again—the same old round. The wind blows south, the wind blows north. Around and around and around it blows, blowing this way, then that—the whirling, erratic wind. All the rivers flow into the sea, but the sea never fills up. The rivers keep flowing to the same old place, and then start all over and do it again. Everything’s boring, utterly boring— no one can find any meaning in it. Boring to the eye, boring to the ear. What was will be again, what happened will happen again. There’s nothing new on this earth. Year after year it’s the same old thing. Does someone call out, “Hey, this is new”? Don’t get excited—it’s the same old story. Nobody remembers what happened yesterday. And the things that will happen tomorrow? Nobody’ll remember them either. Don’t count on being remembered.

c. 450BC

Culd be my KJV upbringing talking, but personally I think there's an informative quality to calling it "vanity" over smoke.

And there's more reasons not to simply compare the modern challenges of image and media with the ancient grappling with impermanence. Tech may only truly change the human condition rarely, but it frequently magnifies some aspect of it, sometimes so much that the quantitative change becomes a qualitative one.

And in this case, what we're talking about isn't just impermanence and mortality and meaning as the preacher/quester is. We'd be lucky if it's business as usual for old planet earth, but we've managed to magnify our ability to impact our environment with tech to the point where winds, rivers, seas, and other things may well change drastically. And as for "smoke", it's one thing if we're dust in the wind, but when we're dust we can trust, that enables continuity and cooperation. There's always been reasons for distrust, but with media scale, the liabilities are magnified, and now we've automated some of them.

The realities of human nature that are the seeds of the human condition are old. But some of the technical and social machinery we have made to magnify things is new, and we can and will see new problems.

'הבל (hevel)' has the primary sense of vapor, or mist -- a transient thing, not a meaningless or purposeless one.

One is a complaint that everything is constantly changing, the other that nothing ever changes. I don't think you could misunderstand what either is trying to say harder if you tried.

"everything is constantly changing!" is the thing that never changes.

You sound like a poorly trained gpt2 model.

Heuristics don't have to be perfect to be useful so long as they improve the efficacy of our attentions. Once that breaks down society must follow because thinking about every topic is intractable.

The mushroom thing is almost certainly true. There's tons of trash AI generated foraging books being published to Amazon. Atomic Shrimp has a video on it.

You're attributing too much to Google.

Bots are now blocked because they've been abusive. When you host content on the internet, it's not fun to have bots bring your server down or inflate your bandwidth price. Google's bot is actually quite well-behaved. The other problem has been the recent trend in AI, and I can understand blockers being put in place, since AI is essentially plagiarizing content without attribution. But I'd blame OpenAI more at this point.

I also don't think you can blame Google for the centralization behind closed gardens. Or for why people no longer link to other websites. That's ridiculous.

And you should be attributing them the fact that the web is still alive.

> I remember the sea of excrement overtaking genuine human written content on the Internet around mid 2010s.

I mean the AI is trained and modeled on this excrement. It makes sense. As much as people think AI content is raw garbage… they don’t realize that they are staring into a mirror.

>I remember the sea of excrement overtaking genuine human written content on the Internet around mid 2010s.

Things have not changed much really. This was true since the dawn of man-kind (and woman-kind from the man-kind rib of course) even before there writings was invented, in the form of gossip.

The internet/AI now carries on the torch of our ancestral inner calling, lol.

Google didn't change it, it embodied it. The problem isn't AI, it's the pervasive culture of PR and advertising which appeared in the 50s and eventually consumed its host.

Western industrial culture was based on substance - getting real shit done. There was always a lot of scammery around it, but the bedrock goal was to make physical things happen - build things, invent things, deliver things, innovate.

PR and ad culture was there to support that. The goal was to change values and behaviours to get people to Buy More Stuff. OK.

Then around the time the Internet arrived, industry was off-shored, and the culture started to become one of appearance and performance, not of substance and action.

SEO, adtech, social media, web framework soup, management fads - they're all about impression management and popularity games, not about underlying fundamentals.

This is very obvious on social media in the arts. The qualification for a creative career used to be substantial talent and ability. Now there are thousands of people making careers out of performing the lifestyle of being a creative person. Their ability to do the basics - draw, write, compose - is very limited. Worse, they lack the ability to imagine anything fresh or original - which is where the real substance is in art.

Worse than that, they don't know what they don't know, because they've been trained to be superficial in a superficial culture.

It's just as bad in engineering, where it has become more important to create the illusion of work being done, than to do the work. (Looking at you, Boeing. And also Agile...)

You literally make more money doing this. A lot more.

So AI isn't really a tool for creating substance. It's a tool for automating impression management. You can create the impression of getting a lot of work done. Or the impression of a well-written cover letter. Or of a genre novel, techno track, whatever.

AI might one day be a tool for creating substance. But at the moment it's reflecting and enabling a Potemkin busy-culture of recycled facades and appearances that has almost nothing real behind it.

Unfortunately it's quite good at that.

But the problem is the culture, not the technology. And it's been a problem for a long time.

Thank you, you've stated this all very clearly. I've been thinking about this in terms of "doing work", where you care about the results, and "performing work", where you care about how you are evaluated. I know someone who works in a lab, and pointed out that some of the equipment being used was out of spec and under-serviced to the point that it was essentially a random number generator. Caring about this is "doing work". However, pointing it out made that person the enemy of the greater cohort that was "performing work". The results were not important to them, their metrics about units of work completed was. I see this pattern frequently. And it's hard to say those "performing work" are wrong. "Performing" is rewarded, "doing" is punished - Perhaps right to the top, as many companies are involved in a public performance designed to affect the short-term stock price.

Yeah. It's like our entire society has been turned into a Goodhart's Law based simulacrum of a productive society.

I mean, here it's late morning and I'm commenting on hacker news. And getting paid for it.

Workers are many times more efficient than they were in the 50s or 70s or 80s or 90s. Where are our extra vacation days? Why does the worker have to make up for the efficiency with more work while other people take the gains?

Do you seriously think that the purpose of life is to work all the time most efficiently? Enjoy your lazy job and bask in the ability for human society to be productive without everyone breaking their backs all the time.

Kropotkin wondered the same in the 1880s [0]

[0]: https://standardebooks.org/ebooks/peter-kropotkin/the-conque...

focusing on efficiency is very depressing. Machines seek efficiency. Process can be efficient. Assembly lines are efficient. It's all about optimization and quickly focuses on trimming "waste" and packing as much as possible into the smallest space. It removes all that's amazing about human life.

I much prefer a focus on effectiveness (or impact or outcomes, or alternatives). It plays to human strengths, is far less prescriptive and is way more fun!

Some of the most effective actions are incredibly inefficient; sometimes inefficiency is a feature. I received a letter mail thank-you card from our CEO a few years ago. The card has an approx. value of zero dollars, but I know it took the CEO 5-10 mins to write a personal note and sign it, and that she did this dozens of times. The signal here is incredibly valuable! If she used a signing machine, or AI to record a deep fake message I would know or quickly learn, and the value would go negative - all for the sake of efficiency.

I think this is a big part of it. Workers would feel a lot more motivated to do more than just perform if they were given what they know they’re owed for their contribution.

Europe stole them.

Faced with increasing efficiency, Europe by and large appears to have chosen to work less and let expensive things remain expensive. The US, by contrast, now has ridiculously cheap consumer goods, and works all the time.

crazy that this is true and GDP just keeps increasing anyway

Of course GDP increases when the money supply does. It’s like people being incensed at “record corporate profits” amidst inflation - profits will always be record (give or take) because remaining the same is losing money relative to the free money being minted each day, etc. For whatever reason, people naively buy into GDP as a valuable metric, even knowing well that there would be something extremely mysterious going on if that number somehow shrank while the real value of the medium exchange also shrank

this is not how GDP works

Since when did increasing money supply increase GDP? Why didn't that work for Zimbabwe?

This is a pretty clear summary of a real problem in most work environments. I have some thoughts about why, but I'm holding onto your articulation to ruminate on in the future.

"Doing work" vs. "performing work": the epitome of this is consulting. Companies pay huge sums of money to consultants that often spend most of their time "performing work", doing beautiful slides even if the content and reasoning is superficial or even dubious, creating reports that are just marketing bullshit, framing the current mission in a way that makes it possible to capture additional projects and bill the client even more. Almost everything is bullshit.

For better or for worse, you’ve described the exact output some of those client companies want, so they can show off the shiny slides and have the status symbol of expensive consultants.

The slippage between work and meaning is due to the arrival of post scarcity.

There isn't enough meaningful work to go around because work is still predicated on delivering some useful transformation. But when 80% of the useful transformations can be done by 20% of the people and we have enough to keep civilization going, you don't need full employment any more. But due to the moral hazard doctrine, the elites want to retain control and discipline as is.

For being afraid of a world where people don't care about maintaining the system anymore because it's not needed to house and feed everyone and people can just do art or learn or rest, the system keeps inventing meaningless work.

We're shipping fruit half way across the world to be packaged and then halfway back to be consumed. None of this is efficient or necessary. But it keeps the control system intact and that's the goal.

I like this take on modern tech motivations.

The thing that I struggle with is I agree with it, but I also get a lot of value in using AI to make me more productive - to me, it feels like it lets me focus on producing substance and actions, freeing me up from having to some tedious things in some tedious ways. Without getting into the debate about if it's productive overall, there are certain tasks which it feels irrefutably fast and effective at (e.g. writing tests).

I do agree with the missing substance with modern generative AI: everyone notices when it's producing things in that uncanny valley, and if no human is there to edit that, it makes people uncomfortable.

The only way I can reconcile the almost existential discomfort of AI against my actual day-to-day generally-positive experience with AI is to accept that AI in itself isn't the problem. Ultimately, it is an info tool, and human nature makes people spam garbage for clicks with it.

People will do the equivalent of spam garbage for clicks with any new modern thing, unfortunately.

Getting the most out of latest information of a society has probably always been a cat and mouse game of trying to find the areas where the spam-garbage-for-clicks people haven't outnumbered use-AI-to-facilitate-substance people, like here, hopefully.

Just one nitpick. The thing about test is that it’s repetitive enough to be automated (in a deterministic way) or abstracted into a framework. You don’t need an AI to generate it.

While I occasionally have the pleasure of creating or working with a test suite that's interesting and creative relative to the code under test, the vast majority of unit tests by volume are slop. Does it call the mock? Does it use the return value? Does "if err != nil { return err }" in fact stop and return the error?

This stuff is a perfect candidate for LLM generation.

I find it helpful for generating automated test suites in the style of the rest of the codebase. When working across multiple projects and clients, it reduces the mental load of having to remember or figure out how tests are expected to work in each codebase.

I agree with your theory about tests. The reality of it is most code is garbage - often including my own - and in a lot of environments, the task is to get the job done in a way that fits in with what's there.

AI seems really good at producing middling content, and if you make your living writing mediocre training courses, or marketing collateral, or code, or tests you're in big trouble. I question how valuable this work is though, so are we increasing productivity by utilizing AI, or just getting efficient at a suboptimal game? I for one just refuse to play.

Echoing other comments in gratitude for this very clear articulation of feelings I share, but have not manifested so well. Just wanted to add two connected opinions that round out this view.

1) This consuming of the host is only possible on the one hand because the host has grown so strong, that is the modern global industrial economy is so efficient. The doing stuff side of the equation is truly amazing and getting better (some real work gets done either by accident or those who have not-succumbed to PR and ad culture), and even this drop of "real work" produces enough material wealth to support (at least a lot of) humanity. We really do live in a post scarcity world from a production perspective, we just have profound distribution and allocation problems.

2) Radical wealth inequality profoundly exacerbates the problem of PR and ad culture. If everyone has some wealth doing things that help many people live more comfortably is a great way to become wealthy. But if very few people have wealth, then doing a venture capital FOMO hustle on the wealthy is anyone's best ROI. Radical wealth inequality eventually breaks all the good aspects of capitalist/market economies.

You can outfit an adult life with all of the useful manufactured objects that would reasonably improve it for a not-very-impressive sum. Beyond that it's just clutter (going for quantity) or moving into the lifestyle/taste/social-signaling domain anyway (going for quality). There is just not an unlimited amount of alpha in making physical things. The social/thought/experiential domain is a much bigger opportunity.

>Western industrial culture was based on substance - getting real shit done. There was always a lot of scammery around it, but the bedrock goal was to make physical things happen - build things, invent things, deliver things, innovate.

For a very short period between 1945 to 1980 while the generation who remembered the great depression and WWII was in charge. It's been longer since that's not been the case. And it wasn't the case for most of history before then.

I'm not sure it isn't a reflection of the rise of neoliberalism.

Who says these are mutually exclusive? Hard times make strong men, strong men make good times, good times make weak men, weak men make hard times.

So weak men make hard men? You're welcome.

How graciously the womenfolk leave us to our tragic autopoiesis.

Sure, by way of traumatizing an entire generation. Nobody thanked you.

I'm being sarcastic. This entire concept is bogus.

Hah! I'm certain a lot of people want to believe it's bogus, mostly because of what that says about their character. Observed reality suggests there's some truth here. An example in miniature would be the frequency with which family fortunes are lost within a generation or two.

Nobody, but the focus on individualism as the driver of everything might cause a breakdown in social objectives.

> draw, write, compose

The primacy of these artforms is subjective, and there's no accounting for taste.

> You can create the impression of getting a lot of work done. Or the impression of a well-written cover letter. Or of a genre novel, techno track, whatever.

Yeah, one of their most "effective" uses is to counterfeit signals that we have relied on--wisely or not--to estimate deeper practical truths. Stuff like "did this person invest some time into this" or "does this person have knowledge of a field" or "can they even think straight."

Oh, sure, qualitatively speaking it's not new, people could have used form-letters, hired a ghostwriter, or simply sank time and effort into a good lie... but the quantitative change of "Bot, write something that appears heartfelt and clever" is huge.

In some cases that's devastating--like trying to avert botting/sockpuppet operations online--and in others we might have to cope by saying stuff like: "Fuck it, personal essays and cover letters are meaningless now, just put down the raw bullet-points."

Very well written. I assume you haven't read "Simulacra and Simulation" by Jean Baudrillard, that's why your description is so authentic and is more convincing then just referring to the book. Saved this post for future reference.

this is a top notch comment !

> Western industrial culture was based on substance - getting real shit done.

And what did that get us? Radium poisoning and microplastics in every organ of virtually all animals living within thousands of miles of humans. Our reach has always exceeded our grasp.

> So AI isn't really a tool for creating substance. It's a tool for automating impression management.

I was speaking to a design lecturer the other evening. His fascinating insight was that:

1. The best designers get so much fulfilment out of practicing the craft of design.

2. With the advent of low cost “impression making”, the role has changed to one of “review a lot of mediocre outputs and pick the least crap one”

3. This is robbing people of the pleasure and reward associated with craftsmanship.

I have noted this is applicable so many other crafts, and it’s really sad!

Edited afterthought… Is craftsmanship being replaced by “clickandreviewmanship”?

Just yesterday I watched a video blaming China for "oversupplying" electric cars at low prices and how that was hurting car manufacturers elsewhere e.g Germany. These manufacturers were trying to secretly lobby for tariffs to be placed on Chinese cars, while publicly denying this because THEIR largest and growing market was China. They had some "expert" talk about how China is known to oversupply and is doing this to recover from their housing bubble. All of these experts were preaching the virtues of Western free-trade to the east when they were the ones exporting TO Asia 10 years ago, but now the balance flips and they all tell you how important tariffs are and how evil China is instead..

In summary, China makes useful things in mass and sells them to get out of a recession,the West prints money instead and shits on China for doing it better. They preach free trade while it helps them and put up tariffs when it doesn't.

I'm not Chinese or some mega fan but it really struck me how corrupt and full of propaganda western culture is becoming and people don't seem to recognize it.

Maybe we should stop subsidies to buy electric cars and let China subsidize on the production side instead, while lowering taxes on the production of our own cars.

I agree with you, but it’s really simpler than that. China makes things and the west increasingly just sends emails.

> All of these experts were preaching the virtues of Western free-trade to the east when they were the ones exporting TO Asia 10 years ago

10 years ago? Try 40-50 years ago.

Thanks for writing your comment, I think it’s a public service.

Cultural commentary that makes complex long term trends simple to understand isn’t often this clear or concise. What really makes it powerful though is that it manages to stay in a relatively detached observer-mode without becoming an angry rant. And so rather than provoking (understandable!) anger in others, hopefully it’s inviting more reflection.

People who haven’t thought about it this way might take a harder look at what they are doing and who they really want to be. People that are already thinking along these lines will probably benefit from a reminder that they aren’t crazy.

>Do you think AI has changed that in any way? I

I see this type of reasoning in all the AI threads. And yes, I think this time is different.

>AI gives even better tools to page rank. Detection of AI generated content is not that bad.

It is an arms race between the people generating crap (for various nefarious purposes) and those trying to separate find useful content amongst the ever growing pile of crap. And it seems to me it is so much easier to generate crap, that I can't see how the good guys can possibly win.