https://danluu.com/wat/ apparently this is normal: Facebook famously let all employees access everyone’s profile for a long time, and you can even find HN comments indicating that some recruiters would explicitly mention that as a perk of working for Facebook. And I can think of more than one well-regarded unicorn where everyone still has access to basically everything, even after their first or second bad security breach. It’s hard to get the political capital to restrict people’s access to what they believe they need, or are entitled, to know. A lot of trendy startups have core values like “trust” and “transparency” which make it difficult to argue against universal access.

It doesn't need to be normal. There's no reason companies couldn't build a system that required approval from your manager before being able to access customer data. Any time a manager granted access, that could be audited by some second tier.

"Okay, stay on the line. My manager went to the bathroom 15 minutes ago, he should be back any minute, and then we can proceed with..."

After-the-fact accounting for all 'sensitive' actions would probably be more practical for most business needs.

I'd put a wizard in front of the thing that grants the access token to figure out the purpose and scope of the token needed.

Information request: "Rider History"

User: current caller

Scope: Between 9 AM and 11AM today

Reason: Lost an item this morning, need to lookup driver

If you were fancy you might even be able to convert the wizard's contained information into a request against the backend. Select trip.driver, trip.time from trips where user_id={caller_user_id} and time={9:00-11:00 today}

It actually seems easy to argue against: the company might trust me, but surely not all the customers trust every employee. You're simply putting the customers first.

> I can think of more than one well-regarded unicorn where everyone still has access to basically everything

Hard to be sure, but GitHub support signed up for a friends-only beta of one of my products before I had told anyone outside my immediate circle about it.

What Unicorns would those be?

Have you tried to debug the same two problems when you do not have access to the infrastructure, versus when you do have infrastructure access? It's a radically different experience.

In addition, even if no one needs access, a bad actor who writes the code can always put a backdoor in place and read whatever they want.

I agree with you. I worked at a BlueCross and when debugging front-end apps, and I'd often dig through the database to check to see what is in each table and what could be throwing an error. I'd often bright up a debugging tool called TeaLeaf that let me watch the entire web session (this was back in 2009; so the full DOM tracking we complain about now has been available for quite some time).

Technically, we couldn't see any Group44 information (BlueCross employees) without special permissions. But we had to build that logic into all the debugging tools, and if I wanted to, I could query the database directly.

We were all covered by HIPPA, so we could face serious fines over leaked personal medical information. And honestly, the security and audit team were actually pretty good; often identifying Group44 issues and they'd have SSH and su log scanners to constantly check for unauthorized access on servers.

It's all about trade-offs. Lyft is still a growing company and they probably just need to implement some more automated auditing. And to be fair, these aren't medical records. It's going to become an issue if a customer service rep gives anything to the police without a warrant though.

That you casually talk about breaking the law with incredibly sensitive data is shocking. And worse still, you built in a system that circumvented the official policy, simply because you weren't capable of doing your job.

You basically built in an illegal back door. If you were my employee I would fire you on the spot for gross negligence when I found that. Or maybe if you were a junior you'd be given a massive dressing down, put on notice, re-sent on all the mandatory company tutorials about sensitive security of data and an urgent fix put in to remove the backdoor.

This is just bad engineering, if you can't recreate bugs without having to constantly resort to live data, get someone in who can. It's not particularly hard, it's a learn-able skill of our profession.

And if you're really having a problem, and it does happen once in a blue moon, you should have the ability to create an anonymized version of the live db that strips out all the personally identifiable info. That you can still only get with a signed, approved request and a full justification of why you needed it.

For the record I have worked somewhere that had highly sensitive data. And they took it seriously, unlike you two. None of the engineering team had access to the live db, the live site or the passwords, apart from some senior team members.

> This is just bad engineering, if you can't recreate bugs without having to constantly resort to live data, get someone in who can. It's not particularly hard, it's a learn-able skill of our profession.

In any large enterprise, you're probably working with data spanning multiple in-house databases, vendor applications, message buses, responsible teams... I can develop a greenfield system that makes identifying bugs trivial without accessing production data. Unfortunately, the real world is rarely that simple—especially for any enterprise that has grown organically over time (i.e., all of them).

My employer recently retired some systems that were older than me (I'm in my mid 30's), and the organization is simply too large to be understood by any one individual.

> You basically built in an illegal back door.

Wat? No, we build the system to block group44 access to all our own diagnostic tools for downstream users. I'm just saying we could also get around it because we build it and had direct db access. We didn't though, or at least I didn't.

I did not violate the law at all. We might have had to look at peoples' health claims information to debug issues, and that's fine, so long as we never disclose that data.

That was a decade ago, so they might have tightened things up. Still, at some point, some people are going to have access to all the data (A DBA is not going to be very effective at his/her job if they don't have admin access on production).

The best approach is to make it the exception not the rule. This ultimately protects everyone and provides a nice paper trail in the event something does happen and needs to be audited. You can partially mitigate the bad actor problem by putting safe guards in place to make sure all changes have to be peer reviewed before shipping.

What's the need for anything? Your argument seems to be predicated on a very specific definition of what need is which is somewhat at odds with most of the rest of the world.

Can you actually refute the parent comment's argument? Because it seems more than reasonable to me.

Analytics and Engineering definitely don't need this level of data access for any sort of day-to-day work. I work on analytics tools, and at best, anonymized and generalized data is needed, but never specific customer data. We specifically strip out any PII on data that might reach developers and need to request permission to access any sort of customer data (and I haven't had to do that in a long time).

CS/T&S/Insurance might need per-customer data, but they also don't need blanket access to all customers' data. If they need to handle a specific customer's data, there should be a request made that is logged and audited, and ideally with customer approval.

It is probably possible to design systems to avoid access, but they will get more complex. Engineering has to debug bugs. For example, suppose there is bug where the rate calculations aren't working for certain types of routes. The engineers will want to look up those routes to understand what is causing it. If you are designing an algorithm to detect to fraud, you are going to want to look at cases of fraud to understand how to design the algorithm. Further, if you want to do usability testing you are going to need to test with. You might want to check the different types of names used in the system to make sure they display properly. You may also want to sample the list of customers to user-test with live data or survey customers.

> For example, suppose there is bug where the rate calculations aren't working for certain types of routes. The engineers will want to look up those routes to understand what is causing it.

Then show routes without names.

> If you are designing an algorithm to detect to fraud, you are going to want to look at cases of fraud to understand how to design the algorithm. Then show names without routes.

> Further, if you want to do usability testing you are going to need to test with. You might want to check the different types of names used in the system to make sure they display properly. You may also want to sample the list of customers to user-test with live data or survey customers.

Use a library that can generate realistic but fake data. I feel there is no excuse to not compartmentalize. If data security is not important to the business...well it only takes one bad article like this to cast doubt on the whole company.

The trouble is, if you're at this level of engineering, you're probably going to be writing the isolation layer. So you already have access to the raw data and you're just going to either make things harder for yourself or only be designing for downstream security.

You can and should restrict customer service reps (only allow them to access routes/users/drivers who they have active tickets on), but at some point you're going to need to trust your developers since they can usually just query the database directly.

As I said in another comment, there's absolutely no need for an engineer to have access to live data in normal circumstances.

Learning how to recreate bugs without trace stacks or live data is a debugging skill you can learn. Often it's as simple as following the steps described in the ticket, something some developers seem to not realize, or their reading comprehension is bad.

For really complex bugs, you might need some sort of access to see the specific conditions, but it should be attempted with an anonymized version of the db that you had to request and get signed off on.

If you then really need to put in tracing, if should be temporary, the data access should be heavily restricted and it should be deleted/removed once the bug is fixed.

> Often it's as simple as following the steps described in the ticket, something some developers seem to not realize, or their reading comprehension is bad.

That sounds like a pretty simple set of bugs you are dealing with. I don't think anyone is arguing that they need this level of information to solve "When I click this button it crashes".

For debugging, you generally can access things like server logs, which would already have a lot of the data you might need. It'd be behind a session ID or some other type of anonymizer, but the customer can provide that and other debugging information so you can look up their entry. It might be a bit harder for mobile apps, but there's plenty of telemetry products that provide crash dumps (with many of them supporting the stripping of PII). You definitely don't need always-on access to all customer's data to do this role, or even access to their PII data for the purposes of debugging.

Rate calculations 100% do not need specific data access. All you need to do that job is to have a generalized set of data based on the routes in question. You don't need to see that Joe Smith in SF took a route from A to B and what went wrong with it, you just need to see what all routes/rates from A to B were, and then from there look for anomalies.

Fraud is a bit more tricky, but if you already know what you're looking for, then you don't need the specific customer data set, you just need to know what deviations from the norm a general fraud request had.

Usability testing should be done with completely fake data, preferably created by someone with knowledge on how to do that specific job. This one is by far the easiest one to argue against needing access to real customer data since most places already have this type of fake data created specifically for this purpose.

Overall, none of your examples really need data access. Sure, it'd be nice to have access to it for some of these points, but it'd also be nice to have a million dollars. It doesn't mean you can't do your job if you didn't have it.

A balance has to be made. The parent remark does not show an understanding of the realities of working on a service on all levels. There are many times when developers need access to specific customer data. Pretty much every company has in their TOS verbiage to note that employees may have to access customer data without their specific consent. That said, companies can and should have multiple layers to help prevent abuse of customer information:

1. Access levels: Different employees need access to different types of information. Access at a specific level should be approved by at least one individual, often the employee's manager, but may require multiple layers of approval, depending on the sensitivity of the data. In some cases, background checks may be required. Access should need to be reviewed any time your job duties change.

2. Separation of networks: Customer data should not be on your normal company network except for specific approved instances (such as when a customer sends a file to support that they can use for testing and pass to developers if needed). It should not be possible to pull information from the network containing customer data to your company network, but it might be necessary to be able to push data over.

3. JIT access: Access to the network and systems containing customer data should require elevation. Access to systems and data on this network should still be subject to your access level (an example would be being able to access unscrubbed logs that may include some private information, but not be able to otherwise access data a customer has entered or uploaded.

4. Auditing: There should be an audit trail of who approved an employee's access levels and when. Access to customer data should always be audited, as should access to unscrubbed log files. Other access may need to be audited based on legal requirements or a company's stated commitments to their customers. --That part isn't an absolute, and while more audit logs are generally better, in some cases it may be too much noise.

5. Encryption at rest: Customer data should be encrypted when it is not in use. This might be file level encryption, database encryption, something else, or some combination.

6. Encryption in transit: Customer data should be encrypted while it is being transported to or from the customer, between networks, and preferably within networks.

This stuff can be difficult to do, but by the time you're as large as Facebook, Uber, Lyft, etc., there's no excuse not to be doing it. You can bet that while the implementation differs, Amazon, Google and Microsoft are also doing this.

But I do have understanding. Because I've actually worked in a company with these safeguards. It handled extremely sensitive data of tens of millions of Britons. Normal engineers had no access to live data, 2 senior managers were the only ones in the engineering team with access to live passwords, etc.

It is practical to put extremely heavy restrictions in place between engineers and the live data and they can still do their job. Our normal day-2-day was not impeded in any way.

I only worked there 3 months for other reasons, but regardless of my view on other parts of their operations, their dedication and practical solution to protecting customer data impressed me.

Why does anyone have "access to live passwords"? That point alone calls into question all of their practices, to me...

It seems to me "need" is very specific almost by definition. One needs access to only that which is fundamentally required to perform work. That need in most cases is actually quite constrained. Often it is far smaller in scope that some might like.

I think it's more that the absolute "need" has not been demonstrated.

> What's the need?

It isn't needed all the time but the one scenario I can think of is for debugging. Sometimes it's hard to dissect or even replicate a bug without the original data.

Discussed elsewhere in the thread, it's definitely not needed day to day.

The number of devs popping up in this thread who think routine access is needed to live data just to be able to debug is quite worrying.

In the rare instances it is, an anonymized version of the live db should be used to recreate the bug. If this doesn't work, then you might put in temporary logging accessible to only approved people, which is removed after the bug is fixed and the logs deleted.

To be a bit more explicit, I think small companies have a justifiable reason to be a bit more relaxed about live db access, but once you get to having larger teams of people, you're under a huge risk of a malicious actor. Which is easily mitigated by restricting access.

From reading all of your comments here, it occurs to me that you might just not be talking about the kind of software that involves actually processing user data as the service. When customer data is incidental to the core engineering work, then yes you should be able to abstract it away. When the service is "upload your data and we will perform analysis on it and give you results", then it really does require that data to figure out why this customer says their results aren't making sense.

> it’s definitely not needed day to day

Yes I agree which is why that was the first sentence of my comment, which was in response to your comment which stated, “No-one needed access... Engineers never did.” It’s great that you changed your position, but please don’t make it seem like I was advocating for unfettered, 24/7 access to customer data when I clearly didn’t write that

This is a pretty absolutist approach.

>What's the need?

How about plain old abuse? People using services to break the law, particularly crimes with victims? Safety risks?

- An uber passenger sees their driver has a gun in the cup holder; they report it to uber.

- A Square merchant is using Square to launder serious money and Square catches it.

- A Dropbox user is uploading child pornography that indicates active child abuse.

In these situations, you think the company should consult the user first before they take a look at PII? Or ask a senior manager? The former is laughable, and the latter is not scalable. Senior manager clearance might work at a small or even midsize company, but for a large tech company abuse happens thousands of times a day. Review must be operationalized.

I would love to see a law that could strike the right balance, but I don't see how it's possible. Accommodating both small and large companies would be very challenging If you mandate the kind of structure that big companies would need, it could cripple smaller companies operational budgets. If you mandate the rules suited for small companies, it fetters large companies at scale. And then how would you enforce it? Another regulatory agency with audit authority?

This kind of regulation is better served by the market, imo, and we've seen that with Uber. In regions where Lyft is a viable option, it has seen significant business increase in the wake of Uber's many scandals. The key is sunlight, which is usually cast by 1. journalists 2. whistleblowers 3. EFF and other watchdogs.

Pretty simple: some departments have access but everything gets logged and audited. If you cannot connect a request to a ticket, you'll get questioned. If you abused your access, you'll be fired immediately. Other industries handle it that way (e.g. banks). I know enough people in banking and know that there's no chance they would ever risk looking up my accounts.

Of course there are exceptional circumstances where access would be granted by a manager without customer approval or routinely by extremely specialised abuse teams.

What I'm talking about is general day to day access to customer data. The analytics, customer service, engineering, etc. teams. It's a different discussion you're trying to have.

These laws already exist as part of Sarbanes-Oxley but aren't as strict as what you're proposing. Certain public companies are required to implement safeguards that prevent most employees from having access to customer PII(personal identifiable information). Non-public companies don't have to comply with SOX regulations but maybe some of them should be expanded to include large private companies.