I really hope that the website itself is running serverless on AWS, because that would be a great learning opportunity!

It's quite easy and common to wreck a database when you have it hooked up to serverless infrastructure. Depending on the DB, there can be a significant amount of overhead for each client connection. Postgres is particularly bad at this, since it spawns an entire process for each client that connects. This is why connection pooling is so important for applications.

Serverless is a different beast tho; every function is isolated, so having a client pool is much harder/impossible. If you get a flood of traffic, and your function interacts with a DB, you suddenly have a ton of non-multiplexed client connections, which is all it takes to bring many databases to their knees.

If this is a serverless website, I can't wait for the author's next article, describing this issue and how they addressed it. If it's not serverless... add a client pool.

We solved it by having an EC2 connection pool instance :P it just sits there and ensures all the lambdas don’t blow up the database.

Kind of defeats the point though :/

You could run pgbouncer in a fargate container in ECS. Still serverless technically?

Google cloud functions are able to keep connections between invocations: https://cloud.google.com/functions/docs/bestpractices/networ...

It solves this exact problem and enables pooling (disclosure: I worked on related infrastructure)

As a lover of irony, I was disappointed to see the "maintenance page" source references this url:

"https://sinxloud.com/wp-admin/admin-ajax.php"

PHP running Wordpress. Sorry guys, it's not serverless.

On top of everything you said, if you put your database inside your VPC, your lambda function has to create a network interface inside the subnet. That increases the cold start time.

DynamoDB is a much better solution for serverless since it just uses a REST interface.

I’m a big fan of lambda but not for user facing API end points. It works well for eventually consistent event processing.

Setting up a Load Balancer and an Autoscaling group is not rocket science. Neither is automated patch management.

Aws serverless Aurora... API interface, fully mysql / postgres compatible, cheap to run - billed on a cpu minutes used basis, massively scalable, auto backup and its much faster than typical dedicated server setups. Amazon's best performing aws product ever they were saying.

Aurora is not the issue. The issue is a lack of connection pooling and the creation and destruction of ENIs when using Lambda with anything inside of a VPC.

If you want to do anything with user facing APIs, suck it up and spend $10 a month for two or three micro Linux instances set up in an autoscaling group behind a load balancer.

Elastic Beanstalk makes it dead simple.

In Java you can create a shared connection pool for AWS lambda functions, e.g. using DI and initialising your injector statically when classes are loaded by the JVM

It relies on AWS using the same container/JVM for your invocations though which isn't guaranteed depending on what your load is.

Yes you technically can do the same with any language...configure your connection statically.

I find most sites can just be setup as an S3 bucket with Cloudfront as the CDN. Any special logic can then be housed in some Lambda expressions behind API gateway. That probably would have prevented this outtage. But I haven’t read the article.

Heck, even CloudFront is super easy to setup and works like a charm.

So do you proxy your calls from S3 to a serverless service? And do you handle cross origin calls via CORS? Just curious because I've never used serverless before.

You can use JavaScript on the frontend to call a Lambda function that you 100% control (including CORS, etc). You can use this to do things like authentication, authorization, and rate limiting, as well, versus just giving open access to an internal datastore.

CORS can be set on the S3 bucket - wide open is the default setting when you open the CORS properties in the S3 console.

The shoemaker's children go barefoot

But it's "serverless", how could it go down?

(and I say this as someone that runs a web app built on the Serverless platform that I'm quite happy with - I just think it's a dumb name)

financial divide by zero error or wallet capacity underflow.

Just speculating, but my guess is the bandwidth demand exhausted the available capacity and author wasn't willing to pay for more.

My question of how it could go down was firmly tongue-in-cheek. My own serverless web app has gone down a few times during AWS outages, for example.

Upvoted for the irony of it.

Man, that's hilarious