News for hackers. Stuff that matters.
Re:Serverless Horrors by joshstrange on Sunday September 07, 2025 @08:43AM (#45157680)
I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs. I scrolled aimlessly through the site ignoring most posts (bandwidth usage bills aren’t super interesting) but I did see this one:

https://medium.com/@maciej.pocwierz/how-an-empty-s3-bucket-c...

About how you make unauth’d API calls to an s3 bucket you don’t own to run up the costs. That was a new one for me.

Re:Serverless Horrors by sherburt3 on Sunday September 07, 2025 @08:50AM (#45157731)
I believe they changed that shortly after that blog post went viral: https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3...
Re:Serverless Horrors by Dunedan on Sunday September 07, 2025 @10:47AM (#45158637)
I raised that exact same issue to AWS in ~2015 and even though we had an Enterprise support plan, AWS response was basically: well, you problem.

We then ended up deleting the S3 bucket entirely, as that appeared to be the only way to get rid of the charges, only for AWS to come back to use a few weeks later telling us there are charges for an S3 bucket we previously owned. After explaining to them (again) that this way our only option to get rid of the charges, we never heard back.

Re:Serverless Horrors by jsheard on Sunday September 07, 2025 @10:38AM (#45158540)
You have to wonder how many people quietly got burned by that in the 18 years between S3 launching and that viral post finally prompting a response.
Re:Serverless Horrors by franktankbank on Sunday September 07, 2025 @09:32AM (#45158012)
Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!
Re:Serverless Horrors by pooper on Sunday September 07, 2025 @09:51AM (#45158143)
> Seems an interesting oversight. I can just imagine the roundtable, uhh guys who do we charge for 403? Who can we charge? But what if people hit random buckets as an attack? Great!

It is amazing, isn't it? Something starts as an oversight but by the time it reaches down to customer support, it becomes an edict from above as it is "expected behavior".

> AWS was kind enough to cancel my S3 bill. However, they emphasized that this was done as an exception.

The stench of this bovine excrement is so strong that it transcends space time somehow.

Re:Serverless Horrors by franktankbank on Sunday September 07, 2025 @09:57AM (#45158192)
Even pooper is upset about the stench. Tech is fuckin dumb in the corps, the only logical explanation to me is kickbacks to the CTO or similar.
Re:Serverless Horrors by pooper on Sunday September 07, 2025 @10:39AM (#45158550)
Pooping at the job is one thing but pooping at the job and trying to sell it as a favor to the customer is a whole different game.
Re:Serverless Horrors by cyanydeez on Sunday September 07, 2025 @11:08AM (#45158855)
you don't need kickbacks at this level. They're all judged by their 6-month outlook on revenue and their market shares.

This is just obfuscating grift justified by the "well, you own the severless functions!"

Re:Serverless Horrors by small_scombrus on Monday September 08, 2025 @05:26AM (#45166233)
> I can just imagine the roundtable

That's the best part!

The devs probably never thought of it, the support people who were complained to were probably either unable to reach the devs, or time crunched enough to not be able to, and who as a project manager would want to say they told their Devs to fix an issue that will lose the company money!

Re:Serverless Horrors by 63stack on Monday September 08, 2025 @06:18PM (#45174782)
BS, I can't imagine that not a single developer entertains this scenario and voices concerns.
Re:Serverless Horrors by thousand_bats on Sunday September 07, 2025 @09:02AM (#45157821)
> I thought this would be about the horrors of hosting/developing/debugging on “Serverless” but it’s about pricing over-runs.

Agreed about that. I was hired onto a team that inherited a large AWS Lambda backend and the opacity of the underlying platform (which is the value proposition of serverless!) has made it very painful when the going gets tough and you find bugs in your system down close to that layer (in our case, intermittent socket hangups trying to connect to the secrets extension). And since your local testing rig looks almost nothing like the deployed environment...

I have some toy stuff at home running on Google Cloud Functions and it works fine (and scale-to-zero is pretty handy for hiding in the free tier). But I struggle to imagine a scenario in a professional setting where I wouldn't prefer to just put an HTTP server/queue consumer in a container on ECS.

Re:Serverless Horrors by fishmicrowaver on Sunday September 07, 2025 @09:28AM (#45157988)
I've had similar experiences with Azures services. Black boxes impossible to troubleshoot. Very unexpected behavior people aren't necessarily aware of when they initially spin these things up. For anything important I just accept the pain of deploying to kubernetes. Developers actually wind up preferring it in most cases with flux and devsoace.
Re:Serverless Horrors by Ekaros on Sunday September 07, 2025 @10:37AM (#45158529)
I recently had customer who had smart idea to protect Container Registry with firewall... Breaking pretty much everything in process. Now it kinda works after days of punching enough holes in... But I still have no idea where does something like Container registry pull stuff from, or App Service...

And does some of their suggested solutions actually work or not...

Re:Serverless Horrors by trillic on Sunday September 07, 2025 @10:47AM (#45158633)
Convince them to add IPv6 and you’ll be set for life
Re:Serverless Horrors by jiggawatts on Sunday September 07, 2025 @10:57PM (#45164275)
They did!

But they network address translate (NAT) IPv6, entirely defeating the only purpose of this protocol.

It's just so, so painful that I have no words with which I can adequately express my disdain for this miserable excuse for "software engineering".

Re:Serverless Horrors by mikepurvis on Sunday September 07, 2025 @09:29AM (#45157994)
Is that what people do is test/develop primarily with local mocks of the services? I assumed it was more like you deploy mini copies of the app to individual instances namespaced to developer or feature branch, so everyone is working on something that actually fairly closely approximates prod just without the loading characteristics and btw you have to be online so no working on an airplane.
Re:Serverless Horrors by icedchai on Sunday September 07, 2025 @10:07AM (#45158274)
There are many paths. Worst case, I've witnessed developers editing Lambda code in the AWS console because they had no way to recreate the environment locally.

If you can't run locally, productivity drops like a rock. Each "cloud deploy" wastes tons of time.

Re:Serverless Horrors by tonkinai on Sunday September 07, 2025 @09:34AM (#45158025)
Mocks usually don’t line up with how things run in prod. Most teams just make small branch or dev environments, or test in staging. Once you hit odd bugs, serverless stops feeling simple and just turns into a headache.
Re:Serverless Horrors by mikepurvis on Sunday September 07, 2025 @03:08PM (#45161218)
Yeah, I’ve never worked at one of those shops but it’s always sounded like a nightmare. I get very anxious when I don’t have a local representative environment where I can get detailed logs, attach a debugger, run strace, whatever.
Re:Serverless Horrors by joshstrange on Monday September 08, 2025 @08:12AM (#45167332)
SST has the best dev experience but requires you be online. They deploy all the real services (namespaced to you) and then instead of your function code they deploy little proxy lambdas that pass the request/response down to your local machine.

It’s still not perfect because the code is running locally but it allows “instant” updates after you make local changes and it’s the best I’ve found.

Re:Serverless Horrors by com2kid on Sunday September 07, 2025 @01:51PM (#45160511)
Every time I've done a cost benefit analysis of AWS Lambda vs running a tiny machine 24/7 to handle things, the math has come out in favor of just paying to keep a machine on all the time and spinning up more instances as load increase.

There are some workloads that are suitable for lambda but they are very rare compared to the # of people who just shove REST APIs on lambda "in case they need to scale."

Re:Serverless Horrors by siva7 on Sunday September 07, 2025 @08:48AM (#45157722)
How to destroy your competition. Love it. Also why i dislike AWS. Zero interest to protect their SMB customers from surprise bills. Azure isn't much better but at least they got a few more protections in place.
Re:Serverless Horrors by icedchai on Sunday September 07, 2025 @09:41AM (#45158063)
Same, I was hoping for tales of woe and cloud lock-in, of being forced to use Lambda and Dynamo for something that could easily run on a $20/month VPS with sqlite.
Re:Serverless Horrors by kijin on Sunday September 07, 2025 @09:50AM (#45158131)
The webflow one at the top has an interesting detail about them not allowing you to offload images to a cheaper service. Which you can probably work around by using a different domain.
Re:Serverless Horrors by zahlman on Sunday September 07, 2025 @01:13PM (#45160060)
> Imagine you create an empty, private AWS S3 bucket in a region of your preference. [...] As it turns out, one of the popular open-source tools had a default configuration to store their backups in S3. And, as a placeholder for a bucket name, they used… the same name that I used for my bucket.

What are the odds?

(Not a rhetorical question. I don't know how the choice of names works.)

Re:Serverless Horrors by Andrex on Sunday September 07, 2025 @09:11PM (#45163834)
> I reported my findings to the maintainers of the vulnerable open-source tool. They quickly fixed the default configuration, although they can’t fix the existing deployments.

Anyone wanna guess which open source tool this was? I'm curious to know why they never detected this themselves. I'd like to avoid this software if possible as the developers seem very incompetent.