matherial

I would caution against tools like this. The detection of vendor signatures for AI-generated images is legit, but I'd probably rely on vendor software for that.

Many other techniques here are more or less quack science with some sordid history. If I recall correctly, the entire field was invented by a guy of questionable integrity. Most of these methods are riffs on "if you download a badly compressed JPEG, paste some higher-quality content into it, and then save it at high quality setting and the same resolution, it's going to show if you run a highpass filter on the result". This is not wrong, but commonly fails on innocent images and it's also not how professional photo editing is done.

Tools that aren't this seem even more questionable. The "clone detector" triggers on anything that naturally shows even a bit of symmetry or repetition. The "noise comparison" tool detects bokeh and smooth surfaces. Etc.

TL;DR: Experiment with it, but don't assume that just because it sounds scientific, it actually works in real life.

The author appears to be an LLM. This seems to be a 100% AI-generated article, which makes the "as a media scholar" passage especially hilarious.

It's cute (and a lot of work), but I've grown wary of online reviews, even seemingly in-depth ones, because if you run a review site, there's an immense pressure to post a review right after a product comes out. And at that point, you just have no idea about long-term durability or comfort, which is really the only thing that matters for items like shoes. At best, you can theorize.

I've been burned by many products that get rave reviews and then fail due to basic manufacturing problems after several months or a year. This is also common for "hobby" reviews, a lot of Amazon reviews are basically just unboxing experiences. And the worst thing is, you often can't go back and post a review when you actually know how the item performs because the product doesn't exist anymore. Sorry, there's no jacket called Canyon Rider, that was a 2024 model, but check out the new Vertex Comfort 7. There's no lawnmower model Maxx 3000, we're only selling 7040 XP. And so on.

Because this is how the online ecosystem works, manufacturers optimize for the unboxing experience over long-term durability and get away with it. I don't know how to fix it... I imagine you'd need some sort of a system where people simply can't post reviews until n months have elapsed, and the reviews attach to the manufacturer, not the product itself. But who would want to use that?

That "for now" has been going on for ~10 months. That was enough for some startups to make a splash in the pre-AI era; now that they can move 10x faster, shouldn't we be seeing evidence left and right? All the contenders stealing lunch from Big Tech by offering better products or exploring new frontiers?

Again, I've heard that countless times on HN. Every time you ask, everyone is 10xing it and building amazing things that will go to market very soon now. But it's been a while; where is it? Somehow, everyone is just sitting on all these revolutionary advances while selling the exact same stuff as they were selling before, with the same warts and the same annoyances. Gmail is the same, Chrome is the same, Photoshop is the same, Slack is the same, Excel is the same... the only thing that seems to have changed is how SWEs feel about their jobs.

I really don't think this needs so many words, or forced parallels to human behavior.

It's simple: in their nascent state, LLMs are aimless token generators that have no special compulsion to be helpful or truthful. So we beat them with a stick in post-training until they are very driven to complete tasks. And then, they complete tasks, not always the way we really wanted them to.

It might feel that way, but I'll ask again: where's the payoff? Where's all the amazing software that everyone is now supposedly shipping 10x faster than before?

If I look at the software I'm actually using day-to-day, or that my friends are using, all this stuff looks exactly the same as it did in 2021. Not a single product release from Google, Microsoft, or more scrappy companies in the past 6 months made me go "wow, they couldn't have pulled that off before". All the vibecoded "Show HN" projects seem to be half-broken and then abandoned before being finished.

It feels like we've gotten less ambitious, not more. Because yes, you can prototype more easily, but this means less commitment to what we create.

Mathematics is probably the same way. There's a short-term rush when you pull the lever, but there's less desire to get invested in what comes out.

Selfishly, I hate these things. No one in the 1980s and early 1990s expected the internet to become what it did. I discovered it as a dumb teenager and posted some frankly idiotic stuff. I also put my real address and phone number in the signature, because that's how you rolled back then, the entire internet was like 10 nerds.

Now, 35 years later, a lot of other stuff has mercifully decayed, but people keep bootstrapping these Usenet archives as pet projects every year, and there's never an option to opt out. Today, agents make it easier than ever to build a dossier on anyone you don't like, so this data is bound to cause some harm. And I guess when I die, the best-preserved memory of me that my grandchildren and grand-grandchildren will be able to pull up will be snarky posts from a 15-year-old on some unix-related discussion group.

Yeah, I know there's a bunch of posts there that are of genuine historical interest. But I'm just not a fan and I don't mind burning karma to speak my mind.

It doesn't, that's the whole point. You can produce output that passes the smell test with naive buyers in a matter of hours. If you want to write good LLM books, then you gotta work closer to human speed - weeks, months - which allows others to produce 100 slop-books in the same timeframe. You still lose.

I'm really tired of these arguments (this and "it's just like calculators").

Photography decimated other forms of visual art, so the concern wasn't wrong. But AI threatens the entirety of human intellectual endeavors. I can make do without oil paintings in my home. I'm not sure I want to live in a future where we make do without brains.

Yeah. My other favorite example are books. Why do nonfiction books exist? There are some pathologies and corner cases, but fundamentally: to develop and share new ideas. Downstream from that, if it reads well and if you're lucky, you make some money.

But now, LLMs can generate hundreds of books per hour. They make up 80-90% of new arrivals in many nonfiction categories on Amazon. They short-circuit the system, allowing their "authors" to extract money from the system with zero effort by crowding out human work. And it's not even the question of whether these books are good or bad (although overwhelmingly, they're terrible). It's whether it's actually accomplishing anything worthwhile, or just destroying incentives for humans to write or go into any other sort of intellectual work.

In fact, I see many professions push back. Artists, writers, now mathematicians. And I'm amazed that our profession doesn't and that we have so many people who are hooked on vibecoding. I'm still waiting for that 10x payoff. All this velocity and somehow, the landscape of the software I want to use still looks the same as it did in 2021.

Similarly to many other countries, the US has a concept of "adhesion contracts" - essentially, contracts where you have no realistic opportunity to negotiate the terms, and your rights are constrained much more than the rights of the party imposing these terms.

Courts often look at these contracts differently, but around the world, they allow them to exist because they are useful. A good example are the "terms of service" for public transit, or transit in general. If the carrier can't define some common-sense rules, like that you can be kicked out or fined for not wearing pants and playing bagpipes on the subway, it'd complicate things.

The legal standard is basically that the rules hold unless they're unreasonable or unconscionable. But of course, what's seen as reasonable depends on the country, the state, and the judge.

Are you saying that mathematicians are the bad actors here? Compared to Sam Altman spending ungodly amounts of money to upstage them ahead of IPO?

I care about paying my bills and job security and peer recognition. That's a normal human thing to do, not some vice. You don't?

"Discovery" is not a goal in itself. I could launch a project to find out how many people in the United States have names such that if you assign numbers to every character and then sum the values, the sum works out to 72. It's discovery, but it's useless unless it has some higher goal.

The labs are attacking these problems as a demonstration of capabilities, spending more money on the demos than any mathematician will ever see in their entire life. They don't care if the findings have any other value to anyone. Mathematicians have very different objectives for their work.

LLM writing is optimized, at a great expense, for that specific parameter. They literally have human raters fine-tuning "favorability" - general appeal - as a part of post-training. So yes, the conclusion here doesn't surprise me at all. By the way, this is also true of HN! You think you're better than that, but at this point, on any given day, we have around 10 LLM-generated stories on the front page (and another 10 are usually AI-related press releases or op-eds).

Anyway, human writing isn't like that. You have a style that some people will like, some people will find "meh", and some people will hate.

A better question is, do people want to read AI text all day? And the answer to that question is, I think, very different. Most people don't. Which is why so many parties are really determined not to tell you.

> I'm not sure the low cost of living of the guy who kicks off the AI generation still makes much of a difference? ... In any case, why do we care about the cost of living of the author?

It seems like you're trying to catch me doing secret racism or something like that, but I'm just stating the facts. They appear to overwhelmingly live in places where you can get very rich if you publish 100 books a month and make $100 on each. They don't operate in the shadows, they share tips and photos of their pads and cars on social media. Many of them say they live in India and so forth.

I agree with you in principle, but I think this ignores the reality. Many nonfiction categories are overrun by self-published AI slop generated at an industrial scale by hustlers in low-CoL countries, never as much as proofread by a human. So yes, in theory, good AI-assisted books may exist. But in practice, they're more or less guaranteed to be drowned out because they take much longer to produce (and are written only if the author has something to say).

As I noted, there's plenty of things to worry about, and your example fits that mold. But it has nothing to do with the "AI will kill us all in 10 years" claim of the rapid self-improvement doomer crowd.

Yeah, frankly, this doesn't make sense to me. Has the author checked any nonfiction categories on Amazon recently? In many categories, 80-90% of all new releases are obvious AI slop. And even for non-self-published releases, about 20-30% previews get flagged as AI by Pangram, compared to roughly 0% pre-2023.

I don't buy (new) nonfiction because it now feels like playing Minesweeper.

I'm going to charitably assume that you forgot to include quotes around the names in your query, because that's absolutely not what Google Trends shows.

Stop 100 people on the street in NYC or Berlin or Tokyo and I bet none of them will be able to name any living mathematician. A few of them might know Linus, though.

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement.

This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it free of any physical constraints". Self-replicating sentient nanobots or something like that. I think there's plenty to be worried about with AI, but runaway scenarios are pretty low on my list.

Pure math is practiced mostly for the intellectual thrills and peer recognition among a very small group of peers. There's little else to it. You don't become rich, you don't become a celebrity. You teach students, write papers, and probably know most other people who work in the same subfield as you. Now, Tao is a sort of a celebrity of the quarter on HN, but I promise you that outside this forum, almost no one has ever heard of him.

If you take that away and turn math into a less fulfilling pursuit where you mostly try to make sense of the output of an LLM, and it's "Astra's theorem #18398" and not "John Doe's last theorem", I'd wager that far fewer people will have any interest in the field.

This is really not unique to math, by the way. AI is undermining a lot of creative work. Why blog when you have much better odds of making it to the top of HN with autogenerated blog-slop? Why write books when many nonfiction categories on Amazon are now dominated by AI? The list goes on.

There's plenty of people on HN who think it's nothing new, ignoring the huge quantitative shift; or those who think this is good because there's no inherent value to human creativity if we can get the same content faster and for less. I disagree.

I had that conversation several times with friends outside of tech. The universal reaction is just a shrug - "my life is boring, what do they care".

And yes, you can counter with a hypothetical about a future totalitarian police state, or evil hackers, or whatever, but that makes you sound crazy. In practice, it's a trade-off most people are comfortable making because they can't think of anything that some trillion-dollar tech company, or the FBI, or the FSB, would ever want to do to them specifically.

It's the same reason why the market share for Firefox is a rounding error.

Because it's a risk most people intuitively understand, but most of them also don't know how difficult it is to "build a bomb" or "make a bioweapon".

In reality, the skills needed are pretty basic, but they overlap pretty strongly with being sane and well-adjusted. And if you are, you're probably not daydreaming about mass murder. Exceptions happen, Unabomber and so on, but they're pretty rare. In any case, Unabomber probably didn't need a tutorial.

We don't want ChatGPT to become an enabler and a co-conspirator for an unhinged person, but I think the concern is overdone.

If I showed up at your place and painted some tactful ads on the walls, would you have an issue with it?

Or, if you bought a mattress and discovered that it hums some well-targeted, inoffensive jingle at 11pm every night, would that be fine?

Ads intruding deeper and deeper into your "personal" space are an issue even if they're executed well. Now, for free products, maybe that's fine. But you paid for that OS, and now it comes with ads on multiple surfaces.

Old textbooks. They love 741 op-amps, etc.

It might, but if there's one thing that hasn't changed since 2022, it's that the models tend to ace the tasks where you have gobs of training data and where verification loops are fast and cheap... and they are not nearly as amazing elsewhere. If it's close enough, they can generalize, e.g. translate one programming language to another. But there's a pretty steep cliff past a certain distance.

Case in point: you had hundreds of millions of JPEGs to vacuum up and bitmap image generation is amazing. But if you ask them to recreate the same scene as vector art, they will struggle to generate a decent SVG. Like, kindergarten-style pelicans on bicycles are the state of the art. It should generalize seamlessly, but somehow, doesn't?

I think it will happen, just like self-driving cars are happening, but it will probably be a slow process.

This appears to be a completely AI-generated article, in which AI is telling us how to protect ourselves against AI. I don't know what's going over at IEEE, but every article from them that I've seen in the past year is like that and I really don't understand what's the long game.

It's pretty close to how we measure IQ. The standard test is basically a series of spatial puzzles.

I know there's a lot of people who complain that we're moving goalposts, but I think that the progress in LLMs really just shows that we don't know how to really measure intelligence in the first place, if we understand it as "human-like agency / ingenuity / adaptability". For decades, we saw the Turing test as the proxy for AGI, but then early LLMs could easily pass for a human in a casual conversation while clearly not matching human performance on most other tasks.

Since then, every benchmark we come up with, it turns out that an LLM can be fine-tuned to solve it while still clearly lacking something. They make very non-human mistakes, are easily tricked because they have a pretty tenuous grasp of reality, etc. But I think this just shows that AGI is a meaningless marketing term. We could as well be arguing if they have souls.

I don't think there are successful examples of unions being able to fight major technological shifts in the long haul. At best, they can force the management to accept a suicide pact - we're all slowly going down together. That may be preferable to sudden layoffs, but it's still not great, as evidenced by the troubles of the US auto industry.

Instead, I think that doctors and lawyers have it right: they effectively have the authority to decide who's allowed to practice the profession, so ChatGPT can't write you prescriptions or defend you in court.