News for hackers. Stuff that matters.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by dig1 on Sunday November 12, 2023 @09:37AM (#38240425)
> Serverless architecture promises flexibility, infinite scalability, fast setups, cost efficiency, and abstracting infrastructure, allowing us to focus on the code.

The only thing I know that serverless architecture promises are big bills and a steady income for a cloud provider. I'd be happy to see a serverless setup that won't be blown away with a (way cheaper) small/medium-sized VM.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @09:41AM (#38240443)
You have to manage a VM. For example, ensuring that the VM has an up to date OS. If you don't care about that, ok, but that's something that Lambda offers.

Ephemerality is a plus as well. Just from a security standpoint, having an ephemeral system means persistence is not possible.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by QuadrupleA on Sunday November 12, 2023 @09:50AM (#38240524)
sudo yum update, once a week
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @09:56AM (#38240591)
If it's one VM for a personal website that'll be fine. Good luck explaining that to a SOC2 auditor, or managing that across a fleet.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by declan_roberts on Sunday November 12, 2023 @10:29AM (#38240970)
This really doesn’t work in practice. We brought down our whole compute cluster once when someone ran update and it changed some stupid thing.

It will happen to you sooner or later also. Updates are always out of band for this reason.

That’s why everybody does builds and isolates isolates updates to that process.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @12:11PM (#38242055)
> We brought down our whole compute cluster once when someone ran update and it changed some stupid thing

I've seen this happen twice now, as well.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by buffet_overflow on Sunday November 12, 2023 @05:29PM (#38244762)
I feel like we’re close to that with automated testing and rollbacks. Still, it seems like a ton of complexity for what feels like a fundamental need
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by flavius29663 on Sunday November 12, 2023 @10:12AM (#38240743)
that is not nearly enough. You need to make sure the system still works after the update, so you need to carefully control all the versions that go in, test them in lower environments. Also, for any serious application, you will need to do this multiple times, for hundreds or thousands of machines even for small companies.

I am on your side, actually, I think managing machines is better than serverless, but it's not that easy.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by joshuanapoli on Sunday November 12, 2023 @10:29AM (#38240973)
a recipe for getting exploited by a supply chain attack
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by steveBK123 on Sunday November 12, 2023 @09:58AM (#38240617)
Nooo I can't run that, I need to pay for Bezos 3rd yacht!
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by dig1 on Sunday November 12, 2023 @09:58AM (#38240610)
These things can be easily automated, and many Linux LTS distros come with a preconfigured system auto-update.

> Just from a security standpoint, having an ephemeral system means persistence is not possible.

You still have to persist something somewhere, and there is a higher chance someone will figure out SQL injection or unfiltered POST request through your app than hack SSH access to the box. If someone wants to do any real damage, they'd just continuously DDoS that serverless setup, and the cloud provider will kill the company with the bill.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @10:00AM (#38240626)
> These things can be easily automated,

Is this something people are out there believing? That patching is something that's easy to automate? I find that kind of nuts, I thought everyone understood that this is, in fact, the opposite of easily automated...

> You still have to persist something somewhere, and there is a higher chance someone will figure out SQL injection or unfiltered POST request through your app than hack SSH access to the box.

"This entirely separate attack exists therefor completely removing an entire attack primitive haves no value" - how I read this comment.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by tg180 on Sunday November 12, 2023 @11:56AM (#38241892)
> "This entirely separate attack exists therefor completely removing an entire attack primitive haves no value"

It has value, but it's also true that trusting cloud providers serveless infrastructure introduces additional sets of vulnerabilities due to various reasons.

eg: https://sysdig.com/blog/exploit-mitigate-aws-lambdas-mitre/

Reading your comments, I get the impression that you are used to dealing with clients whose infrastructure management skills are lacking, and they are making a mess of things.

While serverless infrastructures certainly eliminate a range of vulnerability classes, it is adoption is unlikely to be sufficient to secure platforms that are inadequate for the threats they face.

At the end of the day, someone has to put in the work to ensure that things are patched, safe, and secure, whether the computing model is serverless or not.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @12:56PM (#38242492)
> Reading your comments, I get the impression that you are used to dealing with clients whose infrastructure management skills are lacking, and they are making a mess of things.

I mean, I worked at Datadog when this happened: https://www.datadoghq.com/blog/engineering/2023-03-08-deep-d...

Multi-day outage because of an apt update.

Not the only one I've seen, and it's by no means the only issue that occurs with patching (extremely common that companies don't even know if they're patched for a given vuln).

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by sgarland on Sunday November 12, 2023 @06:46PM (#38245348)
Patching is easy to automate if you’re familiar with managing Linux boxes at scale, and have a test environment.

Ansible on a cron, and the pipeline goes to prod if the test environment passes.

Or unattended upgrades in test, that fires a job to prod if it passes.

Or a continuous build process with Packer to replace running instances once they pass.

If you have certain things that can’t tolerate sudden downtime (a DB, etc.) then you need to know how to mask/hold those.

All this to say, it’s easy if you already know the footguns. But IMO, if you don’t know them, you don’t really have any business running Linux boxes in prod.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by insanitybit on Sunday November 12, 2023 @07:43PM (#38245662)
I guess this answers my "do people seriously believe this" question.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by estebarb on Sunday November 12, 2023 @12:34PM (#38242299)
That scares me from using things like Firebase or serverless. I'm trusting the who bills me from protecting me from overbilling.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by kikimora on Sunday November 12, 2023 @04:05PM (#38244064)
I disagree, it is much much more likely that a former dev would leak a SSH key than someone would care to find an SQL injection.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by _fat_santa on Sunday November 12, 2023 @09:57AM (#38240604)
Personally I found serverless incredibly useful for very simple tasks, tasks where even a t2.micro would be overkill. I have a couple of static websites that are mostly static but still occasionally have to do "server stuff" like sending an email or talking to a database. For those instances a Lambda is incredibly useful because it costs you nothing compared to an EC2 and it less maintenance. But for bigger setups I agree it would be easier to just host on a small-medium VM (and I say that as someone whose got an entire API of like 200+ endpoints deployed in lambda)
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by willio58 on Sunday November 12, 2023 @12:15PM (#38242100)
I’ve seen many claim this but I just haven’t experienced it in production and I’m not sure why. We use lambda at my work and we serve several million users a month. Lambda bill comes out to around 200$ a month, not exaggerating. API gateway ends up being more expensive every month than lambda.

I’m asking not to seem snarky, I truly want to know what is making people hit high prices with lambda. Is it like functions that are super computationally intensive and require queued lambda functions?

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by joshuanapoli on Sunday November 12, 2023 @01:21PM (#38242729)
In my business, we heavily use FaaS, and I agree that it seems economical. It's a little surprising, though. AWS Lambda is 5 to 6 times higher price per second than an EC2 instance for a given level of memory and CPU. Our application simply doesn't need much CPU time. Other aspects (database, storage) are more expensive.

The main advantage, though, is predictability of operations. The FaaS services "just work". If we accidentally make a change to one endpoint to consume too much resources, it doesn't affect anything else. It's great for allowing fast changes to new functionality without much risk of breaking mature features.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by toomuchtodo on Sunday November 12, 2023 @01:01PM (#38242533)
A use case where you’re executing arbitrary code provided by users and you don’t want to have to maintain the environment for doing so (reliability, security boundaries, etc).
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by timenova on Sunday November 12, 2023 @01:17PM (#38242681)
A reasonably priced serverless (kinda) setup is possible on Fly.io.

Fly charges for a VM by the second, and when a VM is off, RAM and CPU are not charged (storage is still charged). They also allow you to easily configure shutting down machines when there are no active requests (right from their config file `fly.toml`), and support a more advanced method which involves your application essentially terminating itself when there's no work remaining, which kills the VM. When a new request arrives, it starts back up.

Here are the docs [0]. And here's a blog post on how to terminate the VM from inside a Phoenix app for example [1].

So essentially, you can write an app which processes multiple requests on the same VM (so not really serverless), but also saves costs when its not in use (essentially the promise of serverless).

[0] https://fly.io/docs/apps/autostart-stop/

[1] https://fly.io/phoenix-files/shut-down-idle-phoenix-app/

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by cyanf on Sunday November 12, 2023 @01:27PM (#38242782)
That’s an on-demand server, not serverless.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by kikimora on Sunday November 12, 2023 @04:04PM (#38244054)
Your devs would need one such VM each for testing. It at least you’ll need 2 VMs - staging and production.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by jlokier on Monday November 13, 2023 @08:48AM (#38250061)
You can run efficient VM-like distro containers inside VMs (eg. LXD), and if necessary VMs inside VMs. So you only need one rented VM at the top, if your performance needs are low.

It probably doesn't make sense economically, due to the cost of managing the infra vs the cost of more top-level VMs from a provider, but you can certainly segregate prod, staging, and independent spaces for multiple developers (or yourself on different projects) on a single rented VM if you want to.

Re:Serverless at Scale: Lessons from 200M Lambda Invocations by kikimora on Friday November 17, 2023 @05:27AM (#38301681)
Of course this can be done in many different ways. The point is with serverless it is much easier to achieve.
Re:Serverless at Scale: Lessons from 200M Lambda Invocations by thraxil on Monday November 13, 2023 @06:44AM (#38249172)
I once migrated a monitoring task from a smallish VM to GCP Cloud Functions triggered by the Cloud Scheduler. Nothing super fancy; imagine some code that runs on a cron once per hour, collects a bunch of metrics on various things then writes the results somewhere. Straightforward but takes a couple minutes to execute and needs a good chunk of memory while it's running. Costs went from about $100/month for the VM size we needed to $0.12/month for the Cloud Functions version. Plus there was no longer a VM that needed to be secured, updated, monitored, etc. That aspect was arguably the much bigger savings than the basic VM cost.

My current company is running entirely on Cloud Run. Not quite as "serverless" as pure Functions or Lambdas, but we have zero VMs or hardware that we manage, so I feel like it counts. We don't do huge amounts of traffic (and don't need to), but it's not trivial either and it's very spiky. The Cloud Run part of our setup is almost negligable (dominated by the database and storage/network costs by several orders of magnitude). With that we get easy deploys and rollbacks, auto-scaling, ephemeral preview environments for every PR, and a simple security story (when the security questionnaire spreadsheets come around, I get to just say "not applicable" and skip entire sections on host-based security, SSH keys, OS updates, etc). And it's basically just a standard Docker image for the app, so if we ever felt like it would be more cost effective to run it on a VM or K8s cluster, it wouldn't be that difficult.

I agree that not everything is better with serverless, but there are some things where it's just a vastly better fit.