For caching the query results you get from your database. Also it's easier to spin up Redis and replicate it closer to your user than doing that with your main database. From my experience anyway.

I think the idea is that if your db can hold the working set in RAM and you're using a good db + prepared queries, you can just let it absorb the full workload because the act of fetching the data from the db is nearly as cheap as fetching it from redis.

> For caching the query results you get from your database.

This only makes sense if queries are computationally intensive. If you're fetching a single row by index you aren't winning much (or anything).

Of course? I'm not really sure what the original question actually is if you know that users benefit from caching the results of computationally intensive queries.

OpenAI uses redis to store pieces of text. Fetching pieces of text is not computationally intensive.

Most likely they have them in an rdbms, so it's more like joining a forum thread together. Not expensive, but why not prebuild and store it instead?

> This only makes sense if queries are computationally intensive.

Or if the link to your DB is higher latency than you're comfortable with.

People know it, that's all.

For caching somewhat larger objects based on ETag?

Yes! I have been spending the last couple months pulling out completely unnecessary redis caching from some of our internal web servers.

The only loss here is network latency which negligible when you're colocated in AWS.

Postgres's caches end up pulling a lot more weight too when you're not only hitting the db on a cache miss from the web server.

1. Redis can handle a lot more connections, more quickly, than a database can. 2. It's still faster than a database, especially a database that's busy.

#2 is an interesting point. When you benchmark, the normal process is to just set up a database then run a shitload of queries against it. I don't think a lot of people put actual production load on the database then run the same set of queries against it...usually because you don't have a production load in the prototyping phase.

However, load does make a difference. It made more of a difference in the HDD era, but it still makes a difference today.

I mean, redis is a cache, and you do need to ensure that stuff works if your purge redis (ie: be sure the rebuild process works), etc, etc.

But just because it's old doesn't mean it's bad. OS/390 and AS/400 boxes are still out there doing their jobs.

A pretty small Redis server can handle 10k clients and saturate a 1Gbps NIC. You'd need a pretty heavy duty Postgres database and definitely need a connection pooler to come anywhere close.

I agree that redis can handle some query volumes and client counts that postgres can't.

But FWIW I can easily saturate a 10GBit ethernet link with primary key-lookup read-only queries, without the results being ridiculously wide or anything.

Because it didn't need any setup, I just used:

  SELECT * FROM pg_class WHERE oid = 'pg_class'::regclass;

I don't immediately have access to a faster network, connecting via tcp to localhost, and using some moderate pipelining (common in the redis world afaik), I get up to 19GB/s on my workstation.

> SELECT * FROM pg_class WHERE oid = 'pg_class'::regclass;

This selects every column (*) from every table (ObjectID is of type regclass)?

Sorry, I should have used something more standard - but it was what I had ready...

It just selects every column from a single table, pg_class. Which is where postgres stores information about relations that exist in the current database.

and those have reliable backup/restore infrastructure. Using redis as a cache is fine, just don't use it as your primary DB.

I'm confused on why the need to complicate something as seemingly-straightforward as a KV store into a series of queues that can get all mixed up. I asked ChatGPT to explain it though, and it sounds like the justification for its existence is that it doesn't "block the event loop" while a request is "waiting for a response from Redis."

Last time I checked, Redis doesn't take that long to provide a response. And if your Redis servers actually are that overloaded that you're seeing latency in your requests, it seems like simple key-based sharding would allow horizontally scaling your Redis cluster.

Disclaimer: I am probably less smart than most people who work at OpenAI so I'm sure I'm missing some details. Also this is apparently a Python thing and I don't know it beyond surface familiarity.

I'm not familiar with the Python client specifically, but Redis clients generally multiplex concurrent requests onto a single connection per Redis server. That necessitates some queueing.

Redis latency is around 1ms including network round trip for most operations. In a single threaded context, waiting on that would limit you to around 1000 operations per second. Redis clients improve throughput by doing pipelining, so a bunch of calls are batched up to minimize network roundtrips. This becomes more complicated in the context of redis-cluster, because calls targeting different keys are dispatched to different cache nodes and will complete in an unpredictable order, and additional client side logic is needed to accumulate the responses and dispatch them back to the appropiate caller.

My redis clusters are 10x more cost effective than my postgresdb in handling load.

Better concurrency (10k vs ~200 max connections compared to postgres). ~20x faster than Postgres at Key-value read/write operations. (mostly) single threaded, so atomicity is achieved without the synchronicity overhead found in RDBMS.

Thus, it's much cheaper to run at massive scale like OpenAI's for certain workloads, including KV caching

also:

- robust, flexible data structures and atomic APIs to manipulate them are available out-of-the box

- large and supportive community + tooling

Do you want to ask why we use caching instead of main db in RAM? Or why we use redis instead of postgres for caching?