News for hackers. Stuff that matters.

A comment on PDF to Text, a Challenging Problem. Parent comment

Re:PDF to Text, a Challenging Problem by aorth on Wednesday May 14, 2025 @01:05AM (#43981025)
I tried with one PDF and was surprised to see it connect to some cloud service:

  2025-05-14 07:58:49,373 - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): openaipublic.blob.core.windows.net:443
  2025-05-14 07:58:50,446 - urllib3.connectionpool - DEBUG - https://openaipublic.blob.core.windows.net:443 "GET /encodings/o200k_base.tiktoken HTTP/1.1" 200 361 3922
The project's README doesn't mention that anywhere...
[ Parent | on HN ]
Re:PDF to Text, a Challenging Problem by degamad on Wednesday May 14, 2025 @02:26AM (#43981478)
The project's README mentions that it uses tiktoken[0], which is a separate project created by OpenAI.

tiktoken downloads token models the first time you use them, but it does not mention that. It does cache the models, so you shouldn't see more of those connections, if I'm understanding the code correctly.

[0] <https://github.com/openai/tiktoken>

[ Parent | on HN ]