Former Twitter Employees Charged with Spying for Saudi Arabia

Re:Former Twitter Employees Charged with Spying for Saudi Ar... by AmericanChopper on Wednesday November 06, 2019 @07:46PM (#21469310)

I’ve achieved this without so much hassle at a smallish/medium sized company (~60 engineers). A typical engineer had access to anonymized data, only a few had access to query raw data. All their queries were logged with justifications. They also wouldn’t query raw data as a matter of BaU (outside of some very rare situations), it was pretty much only ever done when making changes to the ETL pipelines.

The solution we had wasn’t particularly difficult to set up, and actually made life much easier for everybody, because we’d provided everybody with a very easy to use interface for the data they wanted (much better than the old school shelling into a DB to run your arbitrary SQL statements).

[ Parent | on HN ]

Re:Former Twitter Employees Charged with Spying for Saudi Ar... by big_chungus on Wednesday November 06, 2019 @08:00PM (#21469399)

Any chance you'd be willing to share more about your setup, at least high-level?

[ Parent | on HN ]

Re:Former Twitter Employees Charged with Spying for Saudi Ar... by AmericanChopper on Wednesday November 06, 2019 @08:29PM (#21469568)

We had a data warehouse where we sent pretty much all the data we had in the organisation, and redash in front of it to allow query access, reporting, etc...

Everything in the data warehouse was anonymized, and people only had access to the schemas they needed (though this was defined quite broadly). Anonymization was handled by our ETL pipeline. When we first set it up, the requirements were pretty simple and we just wrote a little java app to do it. This scaled pretty poorly, and the team ended up putting a proper ETL product in there. I can’t remember which one they used, but there’s a lot of perfectly decent products in that space (even some open source).

[ Parent | on HN ]