Re:Former Twitter Employees Charged with Spying for Saudi Ar...
by big_chungus on Wednesday November 06, 2019 @08:00PM
(#21469399)
Any chance you'd be willing to share more about your setup, at least high-level?
[ Parent
| on HN ]
Re:Former Twitter Employees Charged with Spying for Saudi Ar...
by AmericanChopper on Wednesday November 06, 2019 @08:29PM
(#21469568)
We had a data warehouse where we sent pretty much all the data we had in the organisation, and redash in front of it to allow query access, reporting, etc...
Everything in the data warehouse was anonymized, and people only had access to the schemas they needed (though this was defined quite broadly). Anonymization was handled by our ETL pipeline. When we first set it up, the requirements were pretty simple and we just wrote a little java app to do it. This scaled pretty poorly, and the team ended up putting a proper ETL product in there. I can’t remember which one they used, but there’s a lot of perfectly decent products in that space (even some open source).