News for hackers. Stuff that matters.

jialu1

The logging actually happens asynchronously and honestly, we have a lot of edge locations where we run. Literally reducing the latency to the minimum less than 35 ms. It’s actually not a huge problem when it comes to the time to first token which is often around 1 to 2 seconds in most of the LLMs. Adding 35 ms is almost close to nothing.