Re:Database Lab – Full-size staging databases as a service
by geoffbp on Saturday February 08, 2020 @02:53AM
(#22274587)
What are people using these days for data sanitization? I.e. to remove PII data from dev snapshots
[ Parent
| on HN ]
Re:Database Lab – Full-size staging databases as a service
by purrpit on Saturday February 08, 2020 @09:07AM
(#22275753)
Short answer: Different databases.
We have PII, analytics, catalog, transaction etc databases. We only let people clone non-sensitive databases. Others are cloned only at schema level and data is left out.
This also allows us to move and scale independently across different applications. But that's just a symptom of this design.
There are some complications with this approach but it's a trade off and we try to come on top using foreign-data-wrapper approach and using a database application router for read and writes. We have open sourced a Django db router which routes queries on the basis of tags/app name/table name etc