Re:Neon – Serverless Postgres
by edmundsauto on Saturday May 28, 2022 @07:55PM
(#31544944)
How do you measure the 1 TB db estimate? Is that in index, as a CSV, or back of the napkin math based on estimates?
I've always struggled estimating my DB size, curious what you mean by your estimate.
Re:Neon – Serverless Postgres
by ed25519FUUU on Saturday May 28, 2022 @09:48AM
(#31540294)
Sounds neat. What’s the weak link here though?
No read replicas that can be promoted to master in an outage?
Re:Neon – Serverless Postgres
by davidzweig on Saturday May 28, 2022 @05:16PM
(#31543937)
Ah, nop. If it goes down, we'll figure it out.. 24hrs outage wouldn't be the end of the world. Would have been nice to have a second (read replica) machine in East Asia, to reduce latency for users there, but didn't find a provider like Hetzner.. maybe could take a server in a suitcase and install it in a collocation center there. Bit of a hassle though.
I have a similar setup as OP, and I use ZFS snapshots to be able to quickly rebuild the whole machine on another host. Of course, it still requires manual intervention in an outage (it could be automated, theoretically), and I may lose up to 5 minutes of data due to my snapshot schedule.
Maybe that zfs snaphot under a running database? Is that ok?
Yes, ZFS snapshots are atomic, so as long as the database use sane semantics to do disk I/O, you're fine (AFAIK, PostgreSQL and MySQL/InnoDB are ok, MySQL/MyISAM is not though).
A snapshot is effectively identical to a killed database server (due to OOM, or power outage), and database servers should be crash-resistant.
With PostgreSQL, checkpointing the database before taking the snapshot may make the crash recovery quicker when loading the snapshotted database.
We used firestore before.. got a bit tired of some of the limitations (latency, indexing). Cost-wise I don't think it's that different actually, but we aren't using much bandwidth, then self-hosted can be dramatically cheaper. Have to manage some details of course (zfs filesystem parameters, set up backups, config postgres etc.), but I found that stuff quite interesting and it's knowledge that will always be useful.