Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by Nitramevfank on Tuesday January 07, 2020 @02:59PM
(#21984118)
Here's probably a silly question: Shouldn't this work automatically? I just assumed they would have an intermediate CA or whatever it's called and have that certificate be signed by some widely trusted CA.
Or have they done it in a different way for security reasons?
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by inopinatus on Tuesday January 07, 2020 @06:07PM
(#21986230)
Not silly at all. Certainly there are some database clients that will access a system default CA trust store. ISTR the PostgreSQL JDBC client may have this behaviour, although perhaps depending on which Java runtime & packaging is in use. And there'll be others besides.
And behold, Amazon are themselves a globally trusted root CA! (Have a peek in /etc/ssl/certs, on must current 'nixen I would expect to see the four Amazon roots). So why not have the Amazon root sign the RDS CA certificate as an intermediate CA, so that clients who could trust an RDS-issued cert that way, will magically trust it?
I've met some of the RDS team and they are smart cookies. If you will agree to my use of cookies, then I can think of at least three good enough reasons (and just to be clear, this is speculation and extrapolation, not insider chapter and verse):
1. Changes in behaviour during another potentially disruptive change are the province of the unwise, the unwary, and the otherwise soon to be fired for cause.
2. Not all clients can use a default trust store; even different clients for the same database may differ. For example, the PostgreSQL C client (as used by the command-line tools and many language bindings) cannot access the system certificate store; the bundle must be configured explicitly. For one client to have a magic behaviour that others do not, in the realm of cryptography and trust, is going to make security admins and auditors very unhappy. Inconsistency is the enemy of policy, and policy makes the DevSecOps world go round.
3. Not all clients support a certificate chain, so the RDS team already has to maintain a bundle of the regional intermediates. Making the RDS CA an intermediate just adds more complication to that activity, more special cases to document, and more potential for confusion in client configurations.
So these are security reasons, although they're more from a policy and human factors and software inadequacy perspective than anything to do with, say, large prime numbers. It's quite possible the service team have more reasons besides; I would certainly agree to their use of cookies or indeed any baked product. For me, those reasons alone would suffice, and I am not surprised to see that the new RDS root is another standalone self-signed CA.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by securityty2020 on Tuesday January 07, 2020 @09:46PM
(#21987797)
Side point: +1 on not silly at all.
Re: anything w/ security (esp how it works), if you don't know ASK. ASK. ASK.
PLEASE ASK.
Security is incredibly complex and minor/subtle mistakes can destroy companies and other potentially catastrophic consequences (especially to your employment.) If you're not sure about something, ASK. End-users are bad enough when it comes to security, but engineers should always feel empowered to "make sure" about all matters of security, large or small.
Obviously it's best to to be familiar with best practices, but if you're not a security "expert" and you're unclear about something, ASK.
There no dumb questions in security, just dumb practices. The dumbest thing of all is not asking your question because you think it's "obvious."
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by scurvy on Tuesday January 07, 2020 @11:52PM
(#21988473)
> Security is incredibly complex and minor/subtle mistakes can destroy companies
For better or for worse, that's not the case. These days, security mistakes just end up in 2 years of free credit monitoring. Ironic case in point, Equifax.
I'm not saying you shouldn't take security seriously. But it's not fair to say that insecurity could destroy your company. Mainly, most end users just really don't care (or have a choice).
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by spenczar5 on Wednesday January 08, 2020 @11:42AM
(#21993005)
It depends on the company. What you say is true for a consumer website, but much less true for a SaaS product which depends on customer trust. For example, imagine an AWS or Salesforce or Github security breach that released very private business data.
As ever, the #1 tip to maximise confidence in your configuration is to automate it, write a failing test case, then fix the test.
> Any specific examples?
Some of the new-wave SSL implementations (including yaSSL, with which mysql-client can/could be built) didn't always support them (not sure where yaSSL is at now), and the AWS documentation hints that it's cropped up in some circumstance in GovCloud, but I think the most likely circumstances will be enterprise folks using appliances and/or appliance-like software that's configured only via a web UI, one in which they just didn't think chains were necessary (or that only supports a small bundle etc etc; the sky's the limit in this class of software gore). I've seen this exact issue, albeit for an LDAP client not a RDBMS.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by jontro on Tuesday January 07, 2020 @03:54PM
(#21984707)
For mysql at least you need to specify what CA or cert to trust. There is no default CA which mysql will try to use. So once a CA expires you will have to rotate it
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by takeda on Wednesday January 08, 2020 @01:24AM
(#21988889)
> Here's probably a silly question: Shouldn't this work automatically? I just assumed they would have an intermediate CA or whatever it's called and have that certificate be signed by some widely trusted CA.
Yeah, that would work if someone at AWS would make a bit of sense and would sign them using their pubic CA, but they decided not to and instead generate new RDS CA every 5 years.
Anyway at least with PostgreSQL clients that rely on libpq (the main postgres client library) by default while it prefers SSL when available it won't verify the certificate unless you explicit use verify-ca or verify-full sslmode.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by dperfect on Wednesday January 08, 2020 @02:11AM
(#21989061)
In the spirit of asking silly questions (as encouraged in some comments here), here's mine:
My small SaaS company's PostgreSQL RDS instance and app servers are in a VPC with security groups configured to only allow connections from the app servers to the DB (no public access to the RDS instance). My client (ruby-pg) on the app servers is connecting via SSL, but not currently with certificate validation (though I believe the cert date still needs to be valid [?], hence the need to rotate the PostgreSQL server's certificate).
In this scenario, how important is certificate validation? I understand the theoretical risk of clients not being able to fully trust that they're connected to the database I intend, but from a practical standpoint, it seems that if an attacker is able to poison the VPC's DNS and trick the app servers into connecting to something else, I'm already hosed and cert validation wouldn't do much to help me. Am I missing something obvious and very dangerous?
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by luhn on Wednesday January 08, 2020 @02:31AM
(#21989146)
AWS claims that VPC traffic is immune from MiTM attacks. [0] (Although they strangely don't talk about this much, and I can't even find official documentation of it.) So within a VPC, you can forgo certificate validation.
However, in general certificate validation would protect you. The attacker wouldn't have a valid certificate, so your application would refuse to connect to the malicious endpoint.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by stingraycharles on Wednesday January 08, 2020 @06:55AM
(#21990580)
So basically they intercept all ARP communication and add VPC header information to all Ethernet packets, which should rule out any MITM scenario.
Of course the keyword here is “should”, and depending on your threat mode you may or may not want to have an additional defense layer of SSL certificate validation.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by sturgill on Tuesday January 07, 2020 @11:18PM
(#21988283)
I created a new RDS instance via Cloudformation just last week and immediately had the notice that I should update the cert. Looks like the new cert will be the default on 1/14, but I thought that was really weird.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by mabbo on Tuesday January 07, 2020 @06:26PM
(#21986407)
> Next, I update my client applications to use the new certificates. This process is specific to each app and each database client library, so I don’t have any details to share.
This is how I lost the last two days of work. Learned a lot at least.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by fovc on Tuesday January 07, 2020 @07:16PM
(#21986798)
Is it possible to simply concatenate the two `bundle` files (2015 and 2019) and feed that to client apps? That would allow updating the client first without changing the db, then updating the db without breaking the client. (Then step 3 would be to update the client to just use the 2019 bundle)
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by insomniacity on Tuesday January 07, 2020 @03:44PM
(#21984590)
> Regions – Rotation is needed for database instances in all commercial AWS regions except Asia Pacific (Hong Kong), Middle East (Bahrain), and China (Ningxia).
I wonder why?
Are these CAs subject to... shall we say, additional goverment oversight?
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by jon-wood on Tuesday January 07, 2020 @05:11PM
(#21985614)
Yeah, Ningxia is a weird region. It’s run under license by a Chinese company, and KMS isn’t available, which by extension means most encryption isn’t either since that’s all built on top of KMS these days.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by jon-wood on Wednesday January 08, 2020 @02:45AM
(#21989257)
Oh, fair enough, last time I looked into the China regions that didn't exist. Reading the announcement I'm loving the seeming contradiction between "
AWS KMS is a managed service that makes it easy for you to create and control the encryption keys used to encrypt your data, and uses Chinese government-approved hardware security modules to protect the security of your keys." and the very next sentence "The service is designed so that no one, including KMS service operators, can retrieve your plaintext master keys from the service."
Something tells me those master keys aren't quite as secure as you'd expect in other regions.
Re:Urgent and Important – Rotate Your Amazon RDS, Aurora, an...
by insomniacity on Tuesday January 07, 2020 @03:49PM
(#21984649)
Yes, but if they're already in the network (although I've nothing to suggest they are), MITMing the database connection means they get the data immediately, rather than at the next quasi-judicial request.
Or have they done it in a different way for security reasons?
And behold, Amazon are themselves a globally trusted root CA! (Have a peek in /etc/ssl/certs, on must current 'nixen I would expect to see the four Amazon roots). So why not have the Amazon root sign the RDS CA certificate as an intermediate CA, so that clients who could trust an RDS-issued cert that way, will magically trust it?
I've met some of the RDS team and they are smart cookies. If you will agree to my use of cookies, then I can think of at least three good enough reasons (and just to be clear, this is speculation and extrapolation, not insider chapter and verse):
1. Changes in behaviour during another potentially disruptive change are the province of the unwise, the unwary, and the otherwise soon to be fired for cause.
2. Not all clients can use a default trust store; even different clients for the same database may differ. For example, the PostgreSQL C client (as used by the command-line tools and many language bindings) cannot access the system certificate store; the bundle must be configured explicitly. For one client to have a magic behaviour that others do not, in the realm of cryptography and trust, is going to make security admins and auditors very unhappy. Inconsistency is the enemy of policy, and policy makes the DevSecOps world go round.
3. Not all clients support a certificate chain, so the RDS team already has to maintain a bundle of the regional intermediates. Making the RDS CA an intermediate just adds more complication to that activity, more special cases to document, and more potential for confusion in client configurations.
So these are security reasons, although they're more from a policy and human factors and software inadequacy perspective than anything to do with, say, large prime numbers. It's quite possible the service team have more reasons besides; I would certainly agree to their use of cookies or indeed any baked product. For me, those reasons alone would suffice, and I am not surprised to see that the new RDS root is another standalone self-signed CA.
Re: anything w/ security (esp how it works), if you don't know ASK. ASK. ASK.
PLEASE ASK.
Security is incredibly complex and minor/subtle mistakes can destroy companies and other potentially catastrophic consequences (especially to your employment.) If you're not sure about something, ASK. End-users are bad enough when it comes to security, but engineers should always feel empowered to "make sure" about all matters of security, large or small.
Obviously it's best to to be familiar with best practices, but if you're not a security "expert" and you're unclear about something, ASK.
There no dumb questions in security, just dumb practices. The dumbest thing of all is not asking your question because you think it's "obvious."
tl;dr: ASK!
For better or for worse, that's not the case. These days, security mistakes just end up in 2 years of free credit monitoring. Ironic case in point, Equifax.
I'm not saying you shouldn't take security seriously. But it's not fair to say that insecurity could destroy your company. Mainly, most end users just really don't care (or have a choice).
I just put my root certs in ~/.postgresql/root.crt and it all works, right? (unless the root itself changed, but this should be stable)
> Not all clients support a certificate chain
Any specific examples? psql and PostgreSQL JDBC both support chains.
What would even cause that? Handrolling an SSL library?
Also ensure sslmode is set to verify-ca or verify-full, either as a set parameter or via the PGSSLMODE environment variable. See https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/ssl-c...
As ever, the #1 tip to maximise confidence in your configuration is to automate it, write a failing test case, then fix the test.
> Any specific examples?
Some of the new-wave SSL implementations (including yaSSL, with which mysql-client can/could be built) didn't always support them (not sure where yaSSL is at now), and the AWS documentation hints that it's cropped up in some circumstance in GovCloud, but I think the most likely circumstances will be enterprise folks using appliances and/or appliance-like software that's configured only via a web UI, one in which they just didn't think chains were necessary (or that only supports a small bundle etc etc; the sky's the limit in this class of software gore). I've seen this exact issue, albeit for an LDAP client not a RDBMS.
Yeah, that would work if someone at AWS would make a bit of sense and would sign them using their pubic CA, but they decided not to and instead generate new RDS CA every 5 years.
Anyway at least with PostgreSQL clients that rely on libpq (the main postgres client library) by default while it prefers SSL when available it won't verify the certificate unless you explicit use verify-ca or verify-full sslmode.
My small SaaS company's PostgreSQL RDS instance and app servers are in a VPC with security groups configured to only allow connections from the app servers to the DB (no public access to the RDS instance). My client (ruby-pg) on the app servers is connecting via SSL, but not currently with certificate validation (though I believe the cert date still needs to be valid [?], hence the need to rotate the PostgreSQL server's certificate).
In this scenario, how important is certificate validation? I understand the theoretical risk of clients not being able to fully trust that they're connected to the database I intend, but from a practical standpoint, it seems that if an attacker is able to poison the VPC's DNS and trick the app servers into connecting to something else, I'm already hosed and cert validation wouldn't do much to help me. Am I missing something obvious and very dangerous?
However, in general certificate validation would protect you. The attacker wouldn't have a valid certificate, so your application would refuse to connect to the malicious endpoint.
[0] https://kevin.burke.dev/kevin/aws-alb-validation-tls-reply/
https://twitter.com/0xdabbad00/status/995833462660612096
Of course the keyword here is “should”, and depending on your threat mode you may or may not want to have an additional defense layer of SSL certificate validation.
This is how I lost the last two days of work. Learned a lot at least.
I wonder why?
Are these CAs subject to... shall we say, additional goverment oversight?
Something tells me those master keys aren't quite as secure as you'd expect in other regions.