They're not that bad! Comedian Don McMillan has done the user testing to see if AI will put him out of a job.

https://youtu.be/wBdkmI03V6g

I tested this. Out of 200 attempts I got 2 novel jokes that got good amount of upvotes on r/jokes.

Usually commentators say that they have heard so and so version of the joke, or that it's a repeat, but they did not

Both from opus. Sol/astra could not be funny. The versions that worked I had fed top posts from r/jokes as references and gave abstract directions.

However, the core premises were not new even if the construction was. And it's a lot of work to get something useful out.

I can believe this is true. It's great at going out there, connecting ransom dots in very clever ways, sometimes stumbling on something actually funny.

I'm curious, did you test with the 200 jokes if it could guess which were the beat ones according to a human audience?

In images, music, games, it usually can't. So the process of validation is offloaded to the audience. I'm very interested in seeing where this takes different platforms. I bet Spotify can pull off such a validation process. Music is passive, tolerating a bad song doesn't require much from you. And they have a massive audience. They can try out AI generated mess with a very low success percentage without saturating their playlists and alienating their users. But can Steam tolerate it? Trying out a game is a much bigger effort than listening to a song picked for you. Customers are already reacting to the number of demos in Steam Next Fest. The saturation point is much lower.

And what about jokes? What happens to r/jokes if even 10 standup comedians are validating 200 jokes per day?

And we see it in software too. QA is already being offloaded to the customer.

I want to see where this goes, even if I don't like it.

But if you find a media where AI can also validate the output instead of having to offload it, that's going to be very valuable!

It matches your experience. Its judgement on which joke will get laughs is only slightly better than random with lots of false positives.

I wonder how much different that is from humans, even professional comedians. You don't usually get to see all the jokes that comedians try that don't work, unless you're an avid attendee at comedy clubs.

I'd guess the difference is around a factor of 100 or so.

For comedians, perhaps. But there are plenty of humans who aren’t very good at comedy.

In these LLM comparisons, people tend to compare them to the best humans, not average or below average. It’s a kind of motivated reasoning, with the goal of deprecating the abilities of LLMs.

Well, you did ask for a comparison to professionals.

But I think humor is pretty easy to rate for a human. Did it make you laugh? Yes? Then it might be a good one. This simple human filter radically improves your efficiency as a joke validator, leaving any LLM far behind.

For pictures I'm not sure if humans are much better. For music, yes, after a few listens (recognizing a hit usually takes time). For games, definitely yes.

They can still unintentionally be funny sometimes though, especially certain quirky models. Mimo 2.6 Flash is my favorite in this regard because it will become absolutely unhinged in ecstasy when proposing an idea before even trying to verify if it's correct or see if there's a better one. Probably my favorite sequence from an LLM ever was watching Mimo try to diagnose a bug and coming up with a series of theories that it introduced as "NEW IDEA!!!", "ANOTHER IDEA!!!", "BETTER IDEA!!!", "EVEN BETTER IDEA!!!", "BEST IDEA!!!", and then "ANOTHER BEST IDEA!!!". It's been months and when someone is listing ideas I'll think of the phrase "another best idea" and need to stop myself from laughing.