News for hackers. Stuff that matters.

marshallbrekka

Re:Former employees say Lyft staffers spied on passengers on Friday January 26, 2018 @09:46AM (#16239522)
For an API call that requires decrypting data it can add about 5-10ms, most of that comes from reading the values from the token database. Encrypting will incur a similar performance penalty (again from the database).

The core pieces (API Proxy, Encryption Service, secure/general infrastructure divide) we're done by me in the first 2-4 months.

We now have a team of three (myself included) who maintain those systems (among many other responsibilities).

Re:gRPC-Go Engineering Practices on Friday January 26, 2018 @01:55AM (#16237475)
The mono-repo of protos has been our approach, and has worked very well for us so far.

Only difference is we publish the resulting code for all languages into a single artifact repo, which other projects use as a dependency.

Re:Former employees say Lyft staffers spied on passengers on Friday January 26, 2018 @01:13AM (#16237355)
In short, our REST based APIs are done via a YAML syntax, but our newer APIs are through Protocol Buffers.

See my other comment for more details https://news.ycombinator.com/item?id=16237349

Re:Former employees say Lyft staffers spied on passengers on Friday January 26, 2018 @01:11AM (#16237349)
It's so specific to our architecture that there would be little value in open sourcing it.

That being said, there are some lessons we learned along the way that I do think are worth sharing.

As I mentioned above the API proxy supports REST and gRPC API calls.

For REST based APIs, we developed a YAML syntax for declaratively specifying (per route) the input/output keys that needed to be encrypted/decrypted. This is accomplished using a JsonPath like syntax, along with attributes for each key that specified the type of data, as well as letting us perform limited validation on the data.

However this approach provided little visibility into what fields are not being encrypted, leaving room for mistakes.

This is where gRPC/Protobufs come in to play, it's the newest addition at the API Proxy level, but is not new to our internal architecture.

Using protocol buffers as our IDL has been a huge help in terms of our data auditing capabilities. Unlike our YAML based solution before, with protobufs we can see every single field that is expected as an input or output of an API call.

We've utilized this feature by creating wrapper message types that indicate when a field contains data that needs to be encrypted/decrypted, and have helper libraries that can traverse an instantiated protobuf message, perform the encrypt/decrypt operation, and swap out the original value(s) for the one(s) returned from the encryption service.

Once we have the library to perform the traversal and the compiled protobufs, integrating it into the API Proxy (along with grpc-gateway to translate to REST/Json) is actually pretty straight forward.

Re:Former employees say Lyft staffers spied on passengers on Thursday January 25, 2018 @10:41PM (#16236880)
You're correct that it does make certain kinds of analysis more difficult.

However that doesn't mean we can't ever get access to the original data. Most of our current BI needs to can be met using the un-encrypted data, but for example, if we did want to answer your phone number question, we could craft a special purpose program to perform the analysis without compromising user privacy.

1. Select all phone number tokens

2. Decrypt

3. Produce counts (total unique, etc)

Said program would have to go through normal code review and approvals, and then deployed into the secure zone (so it could access the encryption service).

Re:Former employees say Lyft staffers spied on passengers on Thursday January 25, 2018 @10:33PM (#16236852)
It's a custom Go application that proxies rest/json based APIs, as well as gRPC/Protobuf.
Re:Former employees say Lyft staffers spied on passengers on Thursday January 25, 2018 @10:32PM (#16236846)
In addition to the points already raised by ryan_j_naughton, another consideration is how to perform key rotation and cipher upgrades over time. This is a lot more complex if every application is doing it on their own, but is quite easy if you centralize all (or most) crypto to a single service.

See my reply to another comment: https://news.ycombinator.com/item?id=16236798

Re:Former employees say Lyft staffers spied on passengers on Thursday January 25, 2018 @10:25PM (#16236803)
The service is written in-house, but we do use Vault for things like provisioning temporary credentials.
Re:Former employees say Lyft staffers spied on passengers on Thursday January 25, 2018 @10:23PM (#16236798)
I can elaborate on these points.

The service acts more like a key value store (this is a simplified explanation, but for your questions it will do).

You give it a value, it gives you back a token, which you can later exchange for the original value.

This means the real value is stored in the encryption service, not in the receiving applications database. This gives us the flexibility to perform key rotation (and even upgrade our ciphers as the crypto landscape evolves) at any time without having to worry about where the the encrypted value is being used, as the only data stored outside the service are opaque tokens.

As for de-anonymizing, the service is not designed to take an encrypted value and return its token. If that were possible, we wouldn't have done a very good job encrypting it ;)