News for hackers. Stuff that matters.
Re:Ask HN: Do you self-host your database? by cutthegrass2 on Tuesday June 29, 2021 @05:19AM (#27673595)
This sounds super interesting, are you able to share any further detail?

I'm currently wrestling with a 35TB Aurora DB and considering options for future growth. The system is multi-tenant and one of the options on the table is breaking out the tenants and hosting them on either their own system or alongside smaller subset of tenants.

Re:Ask HN: Do you self-host your database? by aynyc on Tuesday June 29, 2021 @09:50AM (#27675747)
Done that before, where my largest clients are exponentially bigger than my smallest client. The key is to realize one-size-fits-all doesn't work, not in the long term. I'm talking about feature development, storage, etc.. In the end, we break out large clients onto their own instances of RDS, where smaller clients stay multi-tenants. They share the same schemas, etc.. We wrote a lot of utility scripts to migrate clients from small to big, and vice versa. Smallest clients get automatic minor and major upgrades to latest features almost immediate. Larger clients generally only get major upgrades.
Re:Ask HN: Do you self-host your database? by gregwebs on Tuesday June 29, 2021 @07:14AM (#27674257)
The issue with this approach (other than isolation) is that the largest tenant may be orders of magnitude bigger than others and still need more sharding to perform well while their smallest still needs to share a db. Have you looked at TiDB (MySQL) or CRDB (PG)?
Re:Ask HN: Do you self-host your database? by cutthegrass2 on Tuesday June 29, 2021 @08:43AM (#27675009)
Absolutely.

Thanks for the suggestion of TiDB, haven't previously considered this but it looks interesting, will take a closer look.

Re:Ask HN: Do you self-host your database? by exikyut on Wednesday June 30, 2021 @06:42AM (#27686996)
I must admit I'm very curious as to what real-world thing(s) translate to 35TB of stored data (that presumably isn't large blobs, like video or ML models).

(And oh wow I just realized the billing must be amazing...)

Re:Ask HN: Do you self-host your database? by cutthegrass2 on Friday July 02, 2021 @10:58AM (#27712354)
Yep, the AWS cost footprint of this system is eye-watering. The compute spend is 2x the RDS spend!

Can't say too much, but it's Financial Services related.

Re:Ask HN: Do you self-host your database? by exikyut on Friday July 02, 2021 @11:17PM (#27719088)
Ah, a gigantic time series event firehose. (That was what I was curious about.)

I guess this cues the stereotypical "have you looked at kdb+" question, then, which you've probably already fielded :)

(https://kx.com)

IIUC, kdb+ does in fact have a real DB hiding in it, and it's not all just "weird programming language" - but it does seem to require/prefer a reasonable amount of buy-in to the way it does things to be valuable. I guess the lateral question is whether its performance obsession extends to smaller on-disk footprint, and whether the savings would be worth throwing out all the the existing Aurora integration work. (Answer: very probably not.)