News for hackers. Stuff that matters.
Re:March 20 ChatGPT outage: Here’s what happened by xp84 on Friday March 24, 2023 @03:52PM (#35294536)
I'm confused on why the need to complicate something as seemingly-straightforward as a KV store into a series of queues that can get all mixed up. I asked ChatGPT to explain it though, and it sounds like the justification for its existence is that it doesn't "block the event loop" while a request is "waiting for a response from Redis."

Last time I checked, Redis doesn't take that long to provide a response. And if your Redis servers actually are that overloaded that you're seeing latency in your requests, it seems like simple key-based sharding would allow horizontally scaling your Redis cluster.

Disclaimer: I am probably less smart than most people who work at OpenAI so I'm sure I'm missing some details. Also this is apparently a Python thing and I don't know it beyond surface familiarity.

Re:March 20 ChatGPT outage: Here’s what happened by zmj on Friday March 24, 2023 @05:08PM (#35295536)
I'm not familiar with the Python client specifically, but Redis clients generally multiplex concurrent requests onto a single connection per Redis server. That necessitates some queueing.
Re:March 20 ChatGPT outage: Here’s what happened by oxymoron on Friday March 24, 2023 @07:40PM (#35297167)
Redis latency is around 1ms including network round trip for most operations. In a single threaded context, waiting on that would limit you to around 1000 operations per second. Redis clients improve throughput by doing pipelining, so a bunch of calls are batched up to minimize network roundtrips. This becomes more complicated in the context of redis-cluster, because calls targeting different keys are dispatched to different cache nodes and will complete in an unpredictable order, and additional client side logic is needed to accumulate the responses and dispatch them back to the appropiate caller.