News for hackers. Stuff that matters.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @09:41AM (#38240443)
You have to manage a VM. For example, ensuring that the VM has an up to date OS. If you don't care about that, ok, but that's something that Lambda offers.

Ephemerality is a plus as well. Just from a security standpoint, having an ephemeral system means persistence is not possible.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by QuadrupleA on Sunday November 12, 2023 @09:50AM (#38240524)
sudo yum update, once a week
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @09:56AM (#38240591)
If it's one VM for a personal website that'll be fine. Good luck explaining that to a SOC2 auditor, or managing that across a fleet.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by declan_roberts on Sunday November 12, 2023 @10:29AM (#38240970)
This really doesn’t work in practice. We brought down our whole compute cluster once when someone ran update and it changed some stupid thing.

It will happen to you sooner or later also. Updates are always out of band for this reason.

That’s why everybody does builds and isolates isolates updates to that process.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @12:11PM (#38242055)
> We brought down our whole compute cluster once when someone ran update and it changed some stupid thing

I've seen this happen twice now, as well.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by buffet_overflow on Sunday November 12, 2023 @05:29PM (#38244762)
I feel like we’re close to that with automated testing and rollbacks. Still, it seems like a ton of complexity for what feels like a fundamental need
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by flavius29663 on Sunday November 12, 2023 @10:12AM (#38240743)
that is not nearly enough. You need to make sure the system still works after the update, so you need to carefully control all the versions that go in, test them in lower environments. Also, for any serious application, you will need to do this multiple times, for hundreds or thousands of machines even for small companies.

I am on your side, actually, I think managing machines is better than serverless, but it's not that easy.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by joshuanapoli on Sunday November 12, 2023 @10:29AM (#38240973)
a recipe for getting exploited by a supply chain attack
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by steveBK123 on Sunday November 12, 2023 @09:58AM (#38240617)
Nooo I can't run that, I need to pay for Bezos 3rd yacht!
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by dig1 on Sunday November 12, 2023 @09:58AM (#38240610)
These things can be easily automated, and many Linux LTS distros come with a preconfigured system auto-update.

> Just from a security standpoint, having an ephemeral system means persistence is not possible.

You still have to persist something somewhere, and there is a higher chance someone will figure out SQL injection or unfiltered POST request through your app than hack SSH access to the box. If someone wants to do any real damage, they'd just continuously DDoS that serverless setup, and the cloud provider will kill the company with the bill.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @10:00AM (#38240626)
> These things can be easily automated,

Is this something people are out there believing? That patching is something that's easy to automate? I find that kind of nuts, I thought everyone understood that this is, in fact, the opposite of easily automated...

> You still have to persist something somewhere, and there is a higher chance someone will figure out SQL injection or unfiltered POST request through your app than hack SSH access to the box.

"This entirely separate attack exists therefor completely removing an entire attack primitive haves no value" - how I read this comment.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by tg180 on Sunday November 12, 2023 @11:56AM (#38241892)
> "This entirely separate attack exists therefor completely removing an entire attack primitive haves no value"

It has value, but it's also true that trusting cloud providers serveless infrastructure introduces additional sets of vulnerabilities due to various reasons.

eg: https://sysdig.com/blog/exploit-mitigate-aws-lambdas-mitre/

Reading your comments, I get the impression that you are used to dealing with clients whose infrastructure management skills are lacking, and they are making a mess of things.

While serverless infrastructures certainly eliminate a range of vulnerability classes, it is adoption is unlikely to be sufficient to secure platforms that are inadequate for the threats they face.

At the end of the day, someone has to put in the work to ensure that things are patched, safe, and secure, whether the computing model is serverless or not.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @12:56PM (#38242492)
> Reading your comments, I get the impression that you are used to dealing with clients whose infrastructure management skills are lacking, and they are making a mess of things.

I mean, I worked at Datadog when this happened: https://www.datadoghq.com/blog/engineering/2023-03-08-deep-d...

Multi-day outage because of an apt update.

Not the only one I've seen, and it's by no means the only issue that occurs with patching (extremely common that companies don't even know if they're patched for a given vuln).

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by sgarland on Sunday November 12, 2023 @06:46PM (#38245348)
Patching is easy to automate if you’re familiar with managing Linux boxes at scale, and have a test environment.

Ansible on a cron, and the pipeline goes to prod if the test environment passes.

Or unattended upgrades in test, that fires a job to prod if it passes.

Or a continuous build process with Packer to replace running instances once they pass.

If you have certain things that can’t tolerate sudden downtime (a DB, etc.) then you need to know how to mask/hold those.

All this to say, it’s easy if you already know the footguns. But IMO, if you don’t know them, you don’t really have any business running Linux boxes in prod.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @07:43PM (#38245662)
I guess this answers my "do people seriously believe this" question.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by estebarb on Sunday November 12, 2023 @12:34PM (#38242299)
That scares me from using things like Firebase or serverless. I'm trusting the who bills me from protecting me from overbilling.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by kikimora on Sunday November 12, 2023 @04:05PM (#38244064)
I disagree, it is much much more likely that a former dev would leak a SSH key than someone would care to find an SQL injection.