Re:Ask HN: Is it possible to replicate 30TB from 20 DBs to P...
by dba_leveling_up on Sunday April 30, 2017 @09:30AM
(#14231413)
The size of the rows vary, but very few tables have large blobs, and there should be much fewer than 3M rows changing per second. Does the change capture use triggers or a different method?
This should be possible for much less than $100,000. Assuming each database gets its own process for reading the change feed, you're looking at 20 (add more based on individual table/db needs) instances for the change data capture, 3-5 instances for Kafka/ZK brokers, and a handful of machines for reading the changes and writing them into the destination database.
The important thing here is not the number of databases or the size of the tables - its the speed at which inserts happen. 100 inserts/updates per second * 30k tables = 3M per second. That can be handled by the architecture I described above.
If this is a real problem you are trying to solve, I'd love to talk to you. I'm a co-founder of stealth company that is building better tools for engineers solving problems like this.
Our product can solve this exact problem very easily. I'd love to hear what you are working on and show you what we've built. Let me know how to contact you.
Edit: corrected my math.