News for hackers. Stuff that matters.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by thrdbndndn on Thursday June 08, 2023 @12:02PM (#36244124)
Someone in the Tweet reply [1] says it is due to a reddit user called "davidjl123", because of his posting history in /r/counting (?).

So what is /r/counting, and why its "top" user's name is a glitch token in GPT?

[1] https://twitter.com/SoC_trilogy/status/1666714127438434304

Re:Weird GPT-4 behavior for the specific string “ davidjl” by GaggiX on Thursday June 08, 2023 @12:04PM (#36244143)
It's a subreddit where people count. There are a lot of comments, and unfortunately the tokenizer was trained on this subreddit, which is why these weird tokens.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by georgeg23 on Thursday June 08, 2023 @12:08PM (#36244218)
My hunch is the token “ davidjl” is repetitively used in the data that determines the tokenization scheme but is nearly (or completely) absent from the actual pre-train data.

-Riley Goodside

Re:Weird GPT-4 behavior for the specific string “ davidjl” by GaggiX on Thursday June 08, 2023 @12:10PM (#36244265)
Yeah this is true with every glitch token, the distribution of the tokenizer and the GPT model is very different.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by mike_hearn on Thursday June 08, 2023 @01:13PM (#36245303)
That was investigated here:

https://www.lesswrong.com/posts/aPeJE8bSo6rAFoLqg/solidgoldm...

The idea that tokens found closest to the centroid are those that have moved the least from their initialisations during their training (because whatever it was that caused them to be tokens was curated out of their training corpus) was originally suggested to us by Stuart Armstrong. He suggested we might be seeing something analogous to "divide-by-zero" errors with these glitches. However, we've ruled that out.

The counting Reddit usernames are clearly a major source of glitch tokens, there's something about that sub that screws up the model, maybe the unusually predictable/repetitive nature of what goes on there.

The fact that the only non-counting Reddit usernames are both people at the center of massive Bitcoin psychodramas is suggestive however, especially given the associations with the petertodd token. My guess is that these names appear much more frequently in other people's posts than is normal for forum usernames (especially if it read bitcointalk.org and twitter too), and that maybe this has some effect.

To get an idea of just how frequently this token will crop up in the reddit corpus, just do a search for it:

https://www.reddit.com/search/?q=%22petertodd%22&sort=commen...

I'm not sure how many posts match but you can keep scrolling a long time.

Having been more exposed than normal to what happened back then I wasn't hugely surprised by what concepts clustered there, and sadly don't think it's actually random or unexplainable. Flick through discussions pre-2015 (when the Bitcoin forums started to be heavily censored) and you'll see a lot of very similar words as what GPT spits out being used in connection. It was all very nasty.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by samstave on Thursday June 08, 2023 @01:36PM (#36245619)
I wasnt 100% clear on a TOKEN was - so I asked GPT..

Now I understand it.

So I asked GPT to write me a limerick and draw out all the tokens with subword tokens.

Here is the limerick it wrote:

--

write a limerick and then draw a subword tokenized table of it

ChatGPT:

Sure! Here's a limerick for you:

There once was a model so smart,

Trained with data and a lot of heart.

It spoke with finesse,

Generating success,

With AI, it became an art.

And here's the subword tokenized table for the limerick using the WordPiece algorithm:

(table)

Re:Weird GPT-4 behavior for the specific string “ davidjl” by mike_hearn on Thursday June 08, 2023 @02:30PM (#36246471)
Here's my non expert explanation, take it for what it's worth.

Tokens solve the following problem: the first layers of the neural net(s) are a big array of numbers, where the size is a hyperparameter (a parameter that can be used to change the size or behavior of the network). A typical size for a big model like the GPTs is larger than 50,000. You have to somehow encode language as a sequence of assignments to this array of numbers. How do you do it?

The first and most obvious idea is characters. You could assign each unicode code point to one of the slots in the input array, and then use what they call a one-hot encoding where every number is zero except the character. You can do this, but it's not very efficient because virtually all the training text is written in Latin languages and so almost all the slots will be unused.

A better way is to start with a big pile of text that you're going to use, and then iteratively assign the slots to sequences of characters based on how common they are. This makes it easier for the network to learn and reason. These sequences are tokens. For example "and" is very common, and so would "ing" (from the suffix), so those should get their own tokens. SuperGoldMagiCarp isn't so that really shouldn't - it should be represented as a sequence of tokens instead. There are algorithms that figure out the most efficient assignment of tokens to character sequences, which you can think of as the model's vocabulary, and then to convert text into a sequence of these token numbers. OpenAI's software is called Tiktoken and is written in Rust for speed.

The output of the network is likewise also tokens, and so you run the process in reverse at the end to get text back out of the array of floats that the network produces. Or more accurately, the network produces a set of probabilities for each token in its vocabulary, and then you can pick the most probable (oversimplification - in reality the way you select the token is more complicated than that as otherwise you get bad results).

The problems here are to do with bugs in the training process, but are no less interesting for that. Some character sequences that are very rare and should really be represented by many different tokens in a row have somehow ended up being considered important enough to be given their own whole token. The most common cause of this seems to be cases where the token vocabulary was computed on text that contained garbage highly repetitive text like debug logs from video games, hence the prevalence of obscure game characters like Leilan, and Reddit threads are clearly over-represented. But then GPT struggles to work out what these tokens actually mean because they hardly appears in the training set, and so these tokens seem to float together in space and get easily conflated inside the model, also they end up representing very vague or abstract concepts.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by mgsouth on Thursday June 08, 2023 @02:48PM (#36246721)
Well, for one thing the "limerick" doesn't scan. I think the first line was a gimme--"limerick" and "There once was a ..." have got to be really, really correlated in the model, so only "model so smart" needed to newly-minted with the correct rythm. The "It spoke with finesse" and "generating success" lines are short, so fewer chances to mess it up (although the "ing" in "generating" is a little soft; I think "generated" would have a better emphasis pattern). The other two lines are just terrible.

Ah, a Google search for 'limerick "finesse" "success"' had a few matching ones on the first page.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by mcv on Thursday June 08, 2023 @04:11PM (#36248142)
Not particularly humble, is it?
Re:Weird GPT-4 behavior for the specific string “ davidjl” by neonate on Thursday June 08, 2023 @10:41PM (#36252435)
Re:Weird GPT-4 behavior for the specific string “ davidjl” by countmora on Thursday June 08, 2023 @12:28PM (#36244558)
> unfortunately the tokenizer was trained on this subreddit

Do you have a source for that or was it just an assumption?

Re:Weird GPT-4 behavior for the specific string “ davidjl” by mike_hearn on Thursday June 08, 2023 @02:38PM (#36246568)
It's pretty much guaranteed. Where else on the internet would this sequence of characters appear so frequently that it gets selected as one of the internet's top ~50,000 words?

Also, that Reddit is frequently used to train LLMs is widely known. It's an unusually clean source of conversational text because you can slice threads (i.e. pick a root comment, then pick a child, then a child of the child etc and then concatenate the results), and you'll get a coherent conversation. There are relatively few places on the internet where that is true. For example most phpBB forums conflate many different conversations into single threads, with ad-hoc quoting being used to disambiguate which post is replying to which. That makes it a lot harder to generate sample conversations from.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by dontupvoteme on Friday June 09, 2023 @06:12AM (#36255693)
>There are relatively few places on the internet where that is true

Imageboards.

DailyMail.

Slashdot.

Even a somethingawful dump would have been superior.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by mike_hearn on Friday June 09, 2023 @06:56AM (#36256002)
Slashdot doesn't have the volume. Don't know about image boards but are they threaded and do they cover as many topics?

The Daily Mail (the newspaper) has been used for training LLMs in the past, yes. I don't know if it still is.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by red-iron-pine on Friday June 09, 2023 @08:39AM (#36256970)
imageboards sure do -- poorly.

listen, some of the niche corners of that world aren't so bad, but it ain't the place to be training AI to do something, unless that something is a hate crime

Re:Weird GPT-4 behavior for the specific string “ davidjl” by dontupvoteme on Friday June 09, 2023 @10:42AM (#36258557)
So filter the content before you use it. Clearly openAI did the bare minimum on this front.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by gl-prod on Thursday June 08, 2023 @02:45PM (#36246659)
There was a video[0] on Computerphile about this topic

[0] https://www.youtube.com/watch?v=WO2X3oZEJOA

Re:Weird GPT-4 behavior for the specific string “ davidjl” by klooney on Saturday June 10, 2023 @09:25AM (#36270268)
See the old SolidGoldMagikarp drama- it's happened before.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by perfmode on Friday June 09, 2023 @02:22AM (#36254020)
is this why reddit is charging for its API? to try to capture value from the entities training LLMs on reddit data?
Re:Weird GPT-4 behavior for the specific string “ davidjl” by asddubs on Friday June 09, 2023 @03:04AM (#36254332)
that ship has probably sailed, since pushshift hosts a copy of all of reddit, including removed content. it's really weird to me that they even allowed that level of access for that long (FWIW I don't agree with the recent changes, but I see little legitimate reason to give someone a way to scan and copy all recently made posts across all subreddits).
Re:Weird GPT-4 behavior for the specific string “ davidjl” by pjc50 on Friday June 09, 2023 @04:11AM (#36254808)
Archiving. https://news.ycombinator.com/item?id=36254172

It's very much "he who has the power to destroy a thing controls that thing"; so long as there's only one canonical copy of a post and it's on Reddit's servers, that's vulnerable to actions by the subreddit mods or Reddit themselves.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by jallen_dot_dev on Thursday June 08, 2023 @12:29PM (#36244576)
> ETIQUETTE:

> 1. If you're counting, upvote your fellow counting comrades.

Looks like a voting ring for farming karma. Not sure why reddit allows it.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by hn_throwaway_99 on Thursday June 08, 2023 @01:25PM (#36245490)
That's not what it is. It's really just kind of a wacky subreddit for bored and/or OCD people.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by Freedom2 on Thursday June 08, 2023 @11:38PM (#36252810)
I disagree, and agree with jallen. If nothing else, this is fraud! A voting ring like this has to be dismantled immediately, or else there will be significant ramifications, which im sure GP and expand on.
Re:Weird GPT-4 behavior for the specific string “ davidjl” by jallen_dot_dev on Friday June 09, 2023 @01:09AM (#36253485)
I know they're just Internet points but they are supposed to serve as a social signal that somebody has contributed (meaningful) content. Also some subreddits require a certain amount of karma to post. And finally, this activity is just useless spam.

I never implied it was such a big deal as to warrant your snark.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by LegionMammal978 on Friday June 09, 2023 @01:29AM (#36253647)
> And finally, this activity is just useless spam.

It's far from spam: it's a social activity. When "running" with someone, you post a comment with a count, watch your inbox patiently for the other person to post the next count, reply with the count after that, and repeat. In the process, you collaboratively climb the infinite ladder together. Hence the words of congratulations and thanks every time a milestone (a "get") is reached.

Counters can and do hold side discussions along the way; pretty much no one comes to the subreddit to act like a perfect robot. Indeed, the other person's humanity is plainly observable in the variation in their response times.

Also, the main visible karma on Reddit is post karma, and r/counting is the exact opposite of where you'd go to farm post karma, given that a new post on a given chain is only made every ~1000 counts.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by jallen_dot_dev on Friday June 09, 2023 @01:36AM (#36253695)
Fair enough on the social activity aspect. So why is "upvote your fellow counting comrades" rule #1?
Re:Weird GPT-4 behavior for the specific string “ davidjl” by LegionMammal978 on Friday June 09, 2023 @01:48AM (#36253791)
It's not a rule in the sense that "no double-counting" is a rule (you won't find it anywhere in the rulebook), it's more of a soft suggestion. In practice, I've seen people follow it or disregard it to their heart's content. I personally follow it, because my philosophy is to upvote all constructive comments I reply to, and the other counter is literally helping me construct the chain.

But I mostly don't really worry about it either way, since comment karma doesn't count for much.

Re:Weird GPT-4 behavior for the specific string “ davidjl” by fennecfoxy on Monday June 12, 2023 @05:09AM (#36290484)
So...they're just counting in comments? That just seems ridiculous.

Also what sort of "get" is that? They're counting, there's no game to it. Of course they're going to reach Y*0.

Gets made sense when they were chan related, your random post id having so many 0s or identical numbers.

https://knowyourmeme.com/memes/dubs-guy-check-em

Re:Weird GPT-4 behavior for the specific string “ davidjl” by dontupvoteme on Friday June 09, 2023 @02:44AM (#36254193)
Maybe karma as an indicator of text goodness isn't all it's cracked up to be