News for hackers. Stuff that matters.
Re:tiktoken: OpenAI’s Tokenizer by voz_ on Friday December 16, 2022 @02:02AM (#34011549)
I do not like the name.
Re:tiktoken: OpenAI’s Tokenizer by bartread on Friday December 16, 2022 @03:08AM (#34012074)
The comparison is with an alternative called "huggingface". At least "Tiktoken" gives some clue as to what the component is for.
Re:tiktoken: OpenAI’s Tokenizer by jeroenvlek on Friday December 16, 2022 @03:19AM (#34012148)
Huggingface is a lot more than just a tokenizer.
Re:tiktoken: OpenAI’s Tokenizer by charcircuit on Friday December 16, 2022 @03:18AM (#34012137)
This project has nothing to do with TikTok though
Re:tiktoken: OpenAI’s Tokenizer by ianai on Friday December 16, 2022 @04:02AM (#34012415)
And the name implies there’s a connection. Which is problematic.
Re:tiktoken: OpenAI’s Tokenizer by jraph on Friday December 16, 2022 @04:37AM (#34012631)
Of course, it's because you are not tokenizing it right. It's tik-token.

(Could we say it is kind of a garden-path word?)

Re:tiktoken: OpenAI’s Tokenizer by _Algernon_ on Friday December 16, 2022 @06:01AM (#34013128)
It really doesn't. Does every name that has "book" or "face" in the name, imply a connection to Facebook?
Re:tiktoken: OpenAI’s Tokenizer by fragmede on Friday December 16, 2022 @06:10AM (#34013204)
Maybe not but Facebooker, Facebooked, Facebocker, and Facebooken all sound like they do.
Re:tiktoken: OpenAI’s Tokenizer by _Algernon_ on Friday December 16, 2022 @06:33AM (#34013415)
In none of your examples the resulting words ("booked", "bocker", "booken") are common nouns which also happen to relate directly to the product. It is different for a tokeniser to include "token" in its name.
Re:tiktoken: OpenAI’s Tokenizer by ianai on Friday December 16, 2022 @06:42AM (#34013489)
Which is clearly why they pointed out the use of the whole name of the service in both cases.
Re:tiktoken: OpenAI’s Tokenizer by _Algernon_ on Friday December 16, 2022 @06:57AM (#34013589)
I'm aware they both include the full name. That was not the point I was making. Since the difference I was highlighting was unclear:

facebook-ed: Includes the whole name, but nothing else. Obviously bad and not OK.

tik-token: Includes the whole name. Also includes a generic common noun that relates to the product at hand, and is completely fine to include in the name. Appart from that "tik" can not be said to "imply a connection to tiktok". This is fine.

Re:tiktoken: OpenAI’s Tokenizer by TechBro8615 on Friday December 16, 2022 @02:55PM (#34019406)
I see "tiktok en" like en.wikipedia.org or microsoft.com/en
Re:tiktoken: OpenAI’s Tokenizer by rob74 on Wednesday December 21, 2022 @04:43AM (#34078561)
When I first encountered huggingface (without having seen its logo), I had to think of these guys: https://en.wikipedia.org/wiki/Alien_(creature_in_Alien_franc... - so, by trying to pick a domain that sounded welcoming (and was still available), they achieved the exact opposite (at least in my case)...
Re:tiktoken: OpenAI’s Tokenizer by thatguyagain on Friday December 16, 2022 @04:03AM (#34012421)
It's just a little humor
Re:tiktoken: OpenAI’s Tokenizer by DanSmooth on Friday December 16, 2022 @04:51AM (#34012717)
Same. A shame "Token McTokenface" seems to be already in use. What about "Sentence Splitter Upper"?

More serious "qiktoken" would at least hint at the use for the package.