tiktoken: OpenAI’s Tokenizer

Re:tiktoken: OpenAI’s Tokenizer by est on Friday December 16, 2022 @04:48AM (#34012694)

Requires az:// blob download.

I hope pypi libraries can provide complete standalone offline versions instead of requests+urllib3+some_object_storage shenanigans.

[ Parent | on HN ]

Re:tiktoken: OpenAI’s Tokenizer by capableweb on Friday December 16, 2022 @07:21AM (#34013773)

Lots of ML/AI stuff wants to your sign some sort of license before you can use it, being able to deploy stuff offline kind of defeats that. Although of course there are ways around it (download once, store yourself, put in right directory), but they are unlikely to help you work around that.

[ Parent | on HN ]

Re:tiktoken: OpenAI’s Tokenizer by yencabulator on Friday December 16, 2022 @01:31PM (#34018223)

> Lots of ML/AI stuff wants to your sign some sort of license

What's the point of this MIT license, then: https://github.com/openai/tiktoken/blob/main/LICENSE

[ Parent | on HN ]

Re:tiktoken: OpenAI’s Tokenizer by capableweb on Friday December 16, 2022 @02:58PM (#34019450)

Maybe the license is for the code, not the model itself?

[ Parent | on HN ]

Re:tiktoken: OpenAI’s Tokenizer by yencabulator on Friday December 16, 2022 @04:16PM (#34020507)

What's the point of an "open source project" that requires a proprietary data set with seemingly no license? It might as well just the fetch & execute a random executable.

[ Parent | on HN ]

Re:tiktoken: OpenAI’s Tokenizer by seemethere on Friday December 16, 2022 @11:58AM (#34016936)

This is most likely because pypi has size restrictions on what you can upload and most users won't go the extra mile to actually pip install from your bespoke download site.

[ Parent | on HN ]