News for hackers. Stuff that matters.
Re:Netlify just sent me a $104K bill for a simple static site by rafram on Tuesday February 27, 2024 @01:58AM (#39520941)
Eh, I wouldn’t say that’s necessarily the case. AWS support, for example, tends to be really good about waiving charges for things that are clearly your mistake, like an unused instance that you forgot to turn off for a couple months. That’s not because hosting instances doesn’t actually cost Amazon anything! It’s because they want to keep you as a customer even if it loses them a bit of money right now.

In the Netlify case, though, insisting that this person still pay 5% is downright insulting. I’m sure they’re taking a hit already - just waive the whole thing.

Re:Netlify just sent me a $104K bill for a simple static site by onion2k on Tuesday February 27, 2024 @02:34AM (#39521189)
AWS support, for example, tends to be really good about waiving charges for things that are clearly your mistake, like an unused instance that you forgot to turn off for a couple months.

This is an admission that their UX sucks and makes it hard to know what state your account is in and what you're paying for. They waive the fees because a few high profile cases of people paying thousands due to the AWS console being awful would drive a lot of customers away.

Re:Netlify just sent me a $104K bill for a simple static site by mewpmewp2 on Tuesday February 27, 2024 @02:15AM (#39521059)
I would say it is a scam, because you can't set a budget limit.
Re:Netlify just sent me a $104K bill for a simple static site by szundi on Tuesday February 27, 2024 @02:33AM (#39521174)
Stunning
Re:Netlify just sent me a $104K bill for a simple static site by toomuchtodo on Tuesday February 27, 2024 @02:04AM (#39520988)
Our AWS TAM says they don’t do this anymore, and we spend tens of millions of dollars with them annually. n=1, ymmv
Re:Netlify just sent me a $104K bill for a simple static site by hnlmorg on Tuesday February 27, 2024 @02:10AM (#39521027)
You presumably already have enterprise pricing discounts agreed though?
Re:Netlify just sent me a $104K bill for a simple static site by toomuchtodo on Tuesday February 27, 2024 @02:11AM (#39521037)
Yes, but nothing novel, and this is a recent development (within the last few weeks).
Re:Netlify just sent me a $104K bill for a simple static site by hnlmorg on Tuesday February 27, 2024 @04:01AM (#39521790)
I wonder if you’ve hit some kind of internal limit. I don’t know if such things exist but I’ve noticed a pattern around how discounts and credits are allocated.

I do feel your pain though. Managing AWS costs can be a full time job itself.

Re:Netlify just sent me a $104K bill for a simple static site by gcbirzan on Tuesday February 27, 2024 @07:59AM (#39523512)
Is it, though? We've been getting a lot of pushback for months, even for things that weren't really completely our fault (and were made worse by the horrible lag of cost explorer), or even for things that were aws bugs. Maybe now it's official policy, but definitely it's been hard to get refunds for a while now. They were throwing tens of thousands of dollars of credits at us to just play with new services a year and a half ago.
Re:Netlify just sent me a $104K bill for a simple static site by d1sxeyes on Tuesday February 27, 2024 @05:00AM (#39522229)
Nowadays for customers spending millions of dollars you'd expect (at least, Amazon would expect) that the customer has a FinOps department who are already working on getting the most 'bang for their buck' out of what they're paying for and minimising their spend, and they would jump to another platform in a heartbeat if they thought they could save money. It's not unreasonable to think that you don't need to do these customers any favours to keep their business, because those customers are big enough to look after themselves.

For smaller customers, the friendliness of customer support and the flexibility to help them if they make mistakes is much more likely to be a retention consideration. And who knows when a company spending 3 digits a month becomes a customer spending 6 digits a month? You want to be the provider of choice in case the company grows.

Re:Netlify just sent me a $104K bill for a simple static site by Repulsion9513 on Tuesday February 27, 2024 @10:07AM (#39524959)
AWS will save us so much money! We don't have to pay for people to look after hardware! ... just pay for people to set up AWS, and maintain AWS, and make sure we're not paying thousands extra for AWS...
Re:Netlify just sent me a $104K bill for a simple static site by rafram on Tuesday February 27, 2024 @10:28AM (#39525196)
Yeah, exactly. I’m talking “I got billed $15 for an instance I haven’t used for the last few months. Can you refund me?”, not “You guys mind writing off a million or two?”
Re:Netlify just sent me a $104K bill for a simple static site by Johnny555 on Tuesday February 27, 2024 @02:07AM (#39520998)
That’s not because hosting instances doesn’t actually cost Amazon anything

Except it doesn't cost them anything. The marginal cost of keeping your single instance running is $0 (unless they were 100% out of capacity and they could have sold that instance to someone else either at full price or spot price)

Re:Netlify just sent me a $104K bill for a simple static site by johngladtj on Tuesday February 27, 2024 @02:17AM (#39521073)
Electricity costs money
Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Tuesday February 27, 2024 @02:41AM (#39521239)
The electricity overhead of keeping an idle VM on an already running host is nearly zero though.
Re:Netlify just sent me a $104K bill for a simple static site by tsimionescu on Tuesday February 27, 2024 @03:03AM (#39521392)
Sure, but the electricity overhead of keeping one host running just to run an idle VM is not as trivial.
Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Tuesday February 27, 2024 @05:44AM (#39522509)
But that's not what's happening: they aren't keeping a full host for you.

Your argument is like saying that a bus traveler costs the gas needed to power the bus, but it's never the case: the bus would be cruising no matter what. And symmetrically the VM host would be up no matter what you did with your instance.

Re:Netlify just sent me a $104K bill for a simple static site by tsimionescu on Tuesday February 27, 2024 @08:57AM (#39524142)
I'm not sure that's true. Shutting down unneeded hardware seems like a very simple but major optimization.
Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Tuesday February 27, 2024 @09:35AM (#39524586)
You assume that the hardware would be unneeded, but that's a very strong assumption.

It would be very bad for any cloud provider to leave hosts with only one VM running on it, and you can be pretty sure only very small minority of their park that end up in that situation where shutting down a single VM would lead to a shut-down of the entire host, because it means that the host was vastly under-used in the first place.

Re:Netlify just sent me a $104K bill for a simple static site by tsimionescu on Tuesday February 27, 2024 @11:01AM (#39525647)
As far as I know, most cloud hosts don't actually support automatically moving live VMs, so I think it's fairly common for a host to be left running a single VM.

At least in AWS, they never supported this, and in fact may require you to reboot an instance occasionally in order for it to be moved to a new hardware host (typically when they are upgrading their hardware).

Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Tuesday February 27, 2024 @12:12PM (#39526568)
But why are you talking about moving VMs?! Looks like you're adding tons of far-fetched speculations at every step of your reasoning.

The way you easily deal with this issue is very simple and does not require moving VMs: you just allocate newly spawned VMs to existing hosts with available room! When you do so (and they obviously all do!) you end up with little unused hardware…

Re:Netlify just sent me a $104K bill for a simple static site by tsimionescu on Tuesday February 27, 2024 @01:09PM (#39527336)
Say you have 3 hosts, each with a capacity of 10 VMs. At some point you have 28 running VMs - 10 on host1, 10 on host2, 8 on host3. Someone then closes down 2 of the VMs on host1, and 7 of the VMs on host3.

Now you have 19 VMs running, but need to keep all 3 hosts powered. If you don't have live VM moving, you are now forced to keep Host3 running only because 1 VM is running on it, even if that VM is idle. So, this one idle VM is responsible for all the energy consumption of host3, and will continue to be so until at least 3 more VMs get started (since you have room for 2 more VMs on host1).

If you did have live VM migration, or if the idle VM were powered down instead of running idle, you could close host3 completely, moving the VM to host 1, and only re-open host3 if needed for new VMs.

This is equivalent to the problem of memory fragmentation. Even though overall usage is low, if host usage is highly fragmented and you aren't allowed to move used memory around (compacting), you can end up consuming far more than actually needed.

Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Tuesday February 27, 2024 @01:58PM (#39528068)
Except you don't have 3 hosts but 3 thousands, and during the time you're stopping the 9 VMs somebody else is starting 5 or 15 new ones!

Yes it is similar to memory fragmentation in some way, but your argument is like saying an integer stored on the heap costs a full memory page! You realize that it's nonsense. Sure in extreme edge cases it can, but that's not a good metric to know the memory footprint of an integer!

Being able to move VMs is nice as it allows more host use, but it's doesn't mean hosts end up with single idle VMs often!

Re:Netlify just sent me a $104K bill for a simple static site by tsimionescu on Tuesday February 27, 2024 @02:47PM (#39528798)
Of course, it's not gonna be common. But it will occasionally happen, in such a large data center.
Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Wednesday February 28, 2024 @04:24AM (#39535772)
Then you need to account for their low share in idle VMs when measuring how much electricity it is responsible for. If it's only the case for 1% of the idle VMs, then you need to count only 1% of the electric power of a host per idle VM (+ the small fraction of a host CPU power that an idle VM consumes). In any case, it's going to be very small (~$20/year)[1] and the “it costs them nothing” is a good approximation of that, or at least a much better one that assuming that the cost they charge you reflects an expense on their side (which is the point that was argued by rafram at the very start of this discussion.

[1]: let say 10W, which at $.2 per kWh[2], ends up costing $17.5 for an entire year!

[2]: electricity prices from [here](https://www.eia.gov/electricity/monthly/epm_table_grapher.ph...) $.2 per kWh is slightly above the rates in California and Rhode Island, which is the highest in the US for industrial use.

Re:Netlify just sent me a $104K bill for a simple static site by JW_00000 on Tuesday February 27, 2024 @02:40AM (#39521230)
But no electricity is used if your instance is up but idle.
Re:Netlify just sent me a $104K bill for a simple static site by tsimionescu on Tuesday February 27, 2024 @02:57AM (#39521347)
What does "idle" mean? Both a Linux or Windows OS not running any active software will still do computation and even network traffic (disk cache wrangling, indexing, checking for updates, NTP clock syncing etc), and requires electricity to do so.

It's very low cost, especially if its on a VM from a host that otherwise runs other VMs, but it's not 0. And if it happens to be the last VM preventing a hardware server from completely powering off, then it's actually quite far from 0.

Re:Netlify just sent me a $104K bill for a simple static site by pxeger1 on Tuesday February 27, 2024 @02:33AM (#39521181)
“There is a good chance it costs them $0” = “in expectation it costs them >$0”
Re:Netlify just sent me a $104K bill for a simple static site by littlestymaar on Tuesday February 27, 2024 @02:42AM (#39521259)
This exactly.