Re:Packets per Second Limitations in EC2
by kureikain on Sunday October 28, 2018 @04:27AM
(#18319669)
I have got hit by this several time :(. AWS actually have well document about this DNS issue: https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.htm...
This issue can easily get amplifier if you're using Kubernetes on AWS and some library that didn't cache on DNS on its own. Imagine you have a healthcheck every 3 seconds, do a bunch of DNS to its dependencies services, and a single server may have 10 pods.
Re:Packets per Second Limitations in EC2
by greglindahl on Friday October 26, 2018 @11:15PM
(#18314166)
I wonder if this is related to connection tracking?
Re:Packets per Second Limitations in EC2
by john37386 on Saturday October 27, 2018 @04:13AM
(#18314830)
By default, Amazon uses stateless firewall. It means that by default it's not tracking connections.
Re:Packets per Second Limitations in EC2
by spydum on Saturday October 27, 2018 @07:35AM
(#18315307)
I think you may be mistaken. Security Groups are stateful:
https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Securit...
As suggested, it's very likely they hit the connection tracking limitation: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-ne...
I've personally witnessed teams hit this specifically for DNS (usually for internal, where you have explicitly permitted src/dst).
Re:Packets per Second Limitations in EC2
by john37386 on Saturday October 27, 2018 @08:41AM
(#18315472)
Yep I mistaken Security groups with Network Acl. Thanks. It's in the best practices to not track dns for big systems.
From powerdns https://doc.powerdns.com/recursor/performance.html
https://www.sparkpost.com/blog/undocumented-limit-dns-aws/