Re:Facebook Stored Hundreds of Millions of User Passwords in...
by dtrailin on Thursday March 21, 2019 @04:37PM
(#19457043)
If you read the Google Dapper paper from way back in 2010 it has this to say about sensitive information and logging:
> Logging some amount of RPC payload information
would enrich Dapper traces since analysis tools might
be able to find patterns in payload data which could explain performance anomalies. However, there are several
situations where the payload data may contain information that should not be disclosed to unauthorized internal
users, including engineers working on performance debugging.
Since security and privacy concerns are nonnegotiable, Dapper stores the name of RPC methods but
does not log any payload data at this time. Instead,
application-level annotations provide a convenient opt-in
mechanism: the application developer can choose to associate any data it determines to be useful for later analysis with a span.
Given by how influential this paper was and how it likely influenced FBs own tracing system it's crazy that they would choose an opt out model. To me this system design is their biggest mistake and one could have easily been prevented.
> Logging some amount of RPC payload information would enrich Dapper traces since analysis tools might be able to find patterns in payload data which could explain performance anomalies. However, there are several situations where the payload data may contain information that should not be disclosed to unauthorized internal users, including engineers working on performance debugging. Since security and privacy concerns are nonnegotiable, Dapper stores the name of RPC methods but does not log any payload data at this time. Instead, application-level annotations provide a convenient opt-in mechanism: the application developer can choose to associate any data it determines to be useful for later analysis with a span.
Given by how influential this paper was and how it likely influenced FBs own tracing system it's crazy that they would choose an opt out model. To me this system design is their biggest mistake and one could have easily been prevented.