News for hackers. Stuff that matters.
Re:Postgres, Kafka, and a mysterious 100 GB by whartung on Tuesday December 07, 2021 @07:13PM (#29479898)
So this was 21 days of idle chatter? 1 WAL per 5 minute event, 16MB per WAL, 100GB/16MB = 6250 * 5m = 31,250m/1440m/day = 21ish days.

Is that what happened?

Re:Postgres, Kafka, and a mysterious 100 GB by ritmatter on Wednesday December 08, 2021 @09:25AM (#29485035)
In the post there is actually a graph showing that Transaction Logs filled the database in just 6 days (11/24 through 11/30). This doesn't add up to ~100 GB if we assume that each event takes up 16MB of space, but maybe each event took up even more storage than that.
Re:Postgres, Kafka, and a mysterious 100 GB by merb on Tuesday December 07, 2021 @07:30PM (#29480009)
probably. well in his timeline it was only two weeks less, but without more information (which is probably lost) it would be hard if the problem was made worse and other stuff accumulated. but yeah wal file are probably the culprint here. btw. most of the time it's not just aws rds, as somebody else stated a few things in postgres do happen to write data into wal files.

there is a lot more to it and depending on the config and replication it can behave a little bit differntly i.e. under normal use these files can get reused depending on min_wal_size/max_wal_size,etc... it's quite a complex thing and normally do not need to be fine tuned.

I'm also pretty sure the behavior changed in more recent versions (which probably made it worse in this scenario, but more default safety for everbody.)