News for hackers. Stuff that matters.
Re:tiktoken: OpenAI’s Tokenizer by charcircuit on Friday December 16, 2022 @03:18AM (#34012137)
This project has nothing to do with TikTok though
Re:tiktoken: OpenAI’s Tokenizer by ianai on Friday December 16, 2022 @04:02AM (#34012415)
And the name implies there’s a connection. Which is problematic.
Re:tiktoken: OpenAI’s Tokenizer by jraph on Friday December 16, 2022 @04:37AM (#34012631)
Of course, it's because you are not tokenizing it right. It's tik-token.

(Could we say it is kind of a garden-path word?)

Re:tiktoken: OpenAI’s Tokenizer by _Algernon_ on Friday December 16, 2022 @06:01AM (#34013128)
It really doesn't. Does every name that has "book" or "face" in the name, imply a connection to Facebook?
Re:tiktoken: OpenAI’s Tokenizer by fragmede on Friday December 16, 2022 @06:10AM (#34013204)
Maybe not but Facebooker, Facebooked, Facebocker, and Facebooken all sound like they do.
Re:tiktoken: OpenAI’s Tokenizer by _Algernon_ on Friday December 16, 2022 @06:33AM (#34013415)
In none of your examples the resulting words ("booked", "bocker", "booken") are common nouns which also happen to relate directly to the product. It is different for a tokeniser to include "token" in its name.
Re:tiktoken: OpenAI’s Tokenizer by ianai on Friday December 16, 2022 @06:42AM (#34013489)
Which is clearly why they pointed out the use of the whole name of the service in both cases.
Re:tiktoken: OpenAI’s Tokenizer by _Algernon_ on Friday December 16, 2022 @06:57AM (#34013589)
I'm aware they both include the full name. That was not the point I was making. Since the difference I was highlighting was unclear:

facebook-ed: Includes the whole name, but nothing else. Obviously bad and not OK.

tik-token: Includes the whole name. Also includes a generic common noun that relates to the product at hand, and is completely fine to include in the name. Appart from that "tik" can not be said to "imply a connection to tiktok". This is fine.

Re:tiktoken: OpenAI’s Tokenizer by TechBro8615 on Friday December 16, 2022 @02:55PM (#34019406)
I see "tiktok en" like en.wikipedia.org or microsoft.com/en