News for hackers. Stuff that matters.
Re:Cloudflare's Disruption by GordonS on Thursday September 30, 2021 @03:19PM (#28710189)
The big 3 have gotten away with crazy egress pricing for too long - I'm really hoping that Cloudflare's R2 puts a huge spotlight on egress bandwidth price gouging by AWS, Azure and GCP, and further hoping that they reduce pricing in response.

With the huge margins they must have on egress bandwidth, I'm not holding my breath though.

Re:Cloudflare's Disruption by nindalf on Friday October 01, 2021 @01:49AM (#28715075)
Like others have pointed out, it’s not about the margins on egress. Cloudflare claims that AWS marks up egress by 4-17x depending on the region, but this isn’t about making money off egress directly. It’s about creating a moat around all AWS services. If you’re already using some services inside the moat, you’re strongly encouraged to use the other services within the moat.

If your data is in S3 or DynamoDB, egress fees will encourage you to process the data within AWS. You’re not going to use BigQuery for that. If you want to add a search index on that, you won’t go shopping around for search products on other clouds, you’ll likely use AWS elasticsearch. That’s where AWS makes money - through this soft lock-in.

They’ll survive and maybe even thrive if they lose the revenue from egress. But who knows what’ll happen if the moat is destroyed? Every AWS service would need to compete on merit with every service on other clouds. They won’t be chosen simply by default.

Re:Cloudflare's Disruption by NicoJuicy on Friday October 01, 2021 @02:57AM (#28715469)
> Cloudflare claims that AWS marks up egress by 4-17x depending on the region, but this isn’t about making money off egress directly.

Euh, 4-80x , the 80x is for EU and the US.

https://blog.cloudflare.com/aws-egregious-egress/

Re:Cloudflare's Disruption by datavirtue on Thursday September 30, 2021 @04:41PM (#28711057)
Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests? It is not cheap, at all. Your home and/or business internet connection is a joke compared to that type of service.

I don't think they have huge margins on egress at all. There needs to be some incentive for customers of cloud services to minimize bandwidth usage. It is a limited resource.

Re:Cloudflare's Disruption by windexh8er on Thursday September 30, 2021 @09:20PM (#28713399)
There are, literally, dozens of hosting providers that bake unlimited egress into their offerings - still. Bandwidth is cheap and has only gotten cheaper. Remember that Amazon doesn't always need to pay for Internet access. It's very likely they participate in Internet exchanges tied to other large providers. Given Amazon has a massive footprint they likely participate in these all over the globe to reduce congestion on their Internet connectivity.

They very much have massive margin on egress. Given some of the cost comparisons floated today comparing R2 to S3 egress AWS is likely hitting 1000s of times (likely more) their return on the actual bandwidth cost they pay for month over month.

Re:Cloudflare's Disruption by fomine3 on Thursday September 30, 2021 @09:22PM (#28713412)
Re:Cloudflare's Disruption by sysihyk on Friday October 01, 2021 @10:08AM (#28718605)
CPU isn't cheap, typical PHP+MySQl pages are ridiculously heavy, that's why Cloudflare got themselves into business.

Usually DoS attack doesn't exhaust the bandwidth, come on..

Re:Cloudflare's Disruption by ohazi on Thursday September 30, 2021 @05:14PM (#28711393)
> Have you ever had to pay for your own business internet connection that is offered by ISPs for the purpose of serving unlimited requests?

Come on, Amazon is not serving traffic through a Comcast business connection, they're peering directly with other large operators for free or for next to nothing.

Re:Cloudflare's Disruption by DoctorOW on Thursday September 30, 2021 @07:35PM (#28712572)
I mean that's another vector for savings... Maybe Cloudflare bought their own cable modem instead of paying the $20/month to rent one. :)
Re:Cloudflare's Disruption by gravypod on Thursday September 30, 2021 @04:50PM (#28711145)
In the few times I've priced blended connections at most Colo locations unmetered 10G is ~500/month to ~1000/month.

5Gbps*1 month is ~1.5 PB. AWS is about 0.02/gb or ~30k/1.5pb.

Approximately 30x the cost.

These are old numbers for specific use cases so I'm not sure how much that has changed.

Re:Cloudflare's Disruption by rstupek on Thursday September 30, 2021 @06:37PM (#28712165)
AWS bandwidth is definitely over priced however the comparison to an unmetered colo isn't quite fair. What happens when that unmetered colo gets hit by a 100G ddos attack? Everything you have on it goes down?
Re:Cloudflare's Disruption by mattzito on Thursday September 30, 2021 @07:12PM (#28712414)
In fairness, Amazon also charges for more than basic DDoS protection:

https://aws.amazon.com/shield/pricing/

but regardless, the EGRESS charges are what are absurd. The only logical reason for charging so much for data exiting is to make sure that you can't practically leave the AWS ecosystem.

Re:Cloudflare's Disruption by Dylan16807 on Thursday September 30, 2021 @08:12PM (#28712845)
If the discussion is about what it costs amazon, that's not really relevant. They avoid DDOS simply by being enormous.
Re:Cloudflare's Disruption by jiggawatts on Thursday September 30, 2021 @07:40PM (#28712600)
We have a customer on Azure running a bunch of medium-sized websites. Their mean outbound throughput is just 25 Mbps, but they're paying on the order of $3K/month all up for that Internet egress. (Not just bandwidth but some additional overpriced services on top like App Gateway.)
Re:Cloudflare's Disruption by piperswe on Thursday September 30, 2021 @08:09PM (#28712828)
I have had to. I pay $400/month for the following:

- A 42U cabinet - A 15A 120V circuit - An unmetered 1Gbps IP transit link

All at a proper datacenter, namely Hurricane Electric's fmt2 facility. Includes a /29 of IPv4 and a /48 of IPv6, allows me to announce the /24 and /36 I own, and there's a free internet exchange onsite which I have a 1Gbps port at.

Re:Cloudflare's Disruption by tatersolid on Thursday September 30, 2021 @10:55PM (#28713965)
1800 Watts for a full rack? That’s bordering on useless. You can run maybe two real servers on 1800 W.

You need to provision enough power for simultaneous startup after a power outage, unless you have some really smart PDUs and automation. We have a 10 year old DC with 30A@208V per rack and we have to leave racks half full because modern servers are so power-dense.

Re:Cloudflare's Disruption by carlhjerpe on Friday October 01, 2021 @05:36PM (#28723654)
Gatekeeping server power!? We run 1U on 250-350w (230v). So yes, odd power allocation but servers are real way before 900w. We're running dual Xeon Gold low core count machines (because of Microsoft licensing) and they're pretty decent.
Re:Cloudflare's Disruption by tatersolid on Saturday October 02, 2021 @07:52AM (#28727846)
What’s the startup peak draw of those severs? Have you measured it? Remember everything turns on at once after an outage.

In any case, even at your optimistic 250W per rack unit, a full rack of servers with two switches is more than 10kW!

Re:Cloudflare's Disruption by carlhjerpe on Saturday October 02, 2021 @08:27AM (#28728067)
Absolutely, I'm with you. I just meant that 900w isn't where "real servers" start. If you're at the point where you're experiencing issues with scalable power on you could disable that feature and script ipmitool to power the servers on once ipmi is reachable but in a queue. Could help you eek out a bit more density.

You could possibly run this fictional daemon on your management switch :)

Re:Cloudflare's Disruption by tatersolid on Tuesday October 05, 2021 @09:56AM (#28759537)
My 900W per server number comes from experience with “hyper-converged” infrastructure where each 2U node has two filled CPU sockets, gobs of RDIMMs, and is stuffed with flash-based storage.

I think this is the most common “enterprise” datacenter server type in 2021, mostly due to licensing constraints from VMware/Microsoft/Red Hat/Oracle/etc. Such servers give the most “bang” per dollar when licensing costs are included.

Re:Cloudflare's Disruption by carlhjerpe on Tuesday October 05, 2021 @10:39AM (#28760104)
We're running VMware with vSAN on 1u nodes with 2x Xeon Gold 8 core cpus @ 3.6ghz base, something about the Microsoft SPLA licensing as we're mainly virtualizing Windows machines. 384 GB RAM per node, and we're just pulling about 400.

https://i.imgur.com/Zr5rq4X.png a graph if you're curious :)

Re:Cloudflare's Disruption by piperswe on Tuesday October 12, 2021 @05:48PM (#28845487)
Fair enough, but for that price it's hard to beat. Sadly, getting another rack is cheaper than another power circuit though.
Re:Cloudflare's Disruption by GordonS on Thursday September 30, 2021 @05:05PM (#28711287)
Nope, plenty of server hosts provide bandwidth for free, or near enough.

Just one example is Hetzner, who include 20TB of bandwidth, with anything over that charged at just €1/TB.

Meanwhile, AWS is gouging at €80/TB.

Re:Cloudflare's Disruption by ianlevesque on Thursday September 30, 2021 @04:47PM (#28711109)
You don't think they do, but Cloudflare estimated AWS' markup is 7959% here https://blog.cloudflare.com/aws-egregious-egress/
Re:Cloudflare's Disruption by vbernat on Thursday September 30, 2021 @05:35PM (#28711605)
To move bytes out of network, you need more than a transit contract. You need routers, you need people to operate them. All this is absent from Cloudflare's blog post. With the example provided on South Korea, the conclusion should be that the egress fees are only marginally infuenced by the transit cost.
Re:Cloudflare's Disruption by elithrar on Thursday September 30, 2021 @06:14PM (#28711994)
The blog post on egress speaks to that - but the true scale at which the major clouds buy hardware and deploy it - just changes the dynamics here. We're (collectively) not used to seeing products marked up to such an extent.

The $/Mbps prices there - about $6k/Tbps in the US - are based in reality and are absolutely reflective of what it costs, hardware, software, redundancy and all - for an effectively 1Tbps pipe.

If you're pricing as $/GB on top of that capacity and keep it reasonably heavily utilized—which can be hard given diurnal demand—the margins only get better! Products like Glacier (S3) exist to fill exactly those gaps.

(Note: currently work at Cloudflare, but wasn't part of this blog and I've been around a bit...)

Re:Cloudflare's Disruption by selcuka on Thursday September 30, 2021 @08:08PM (#28712825)
Yet CloudFlare is able to do it for free up to a point, and at a lower-than-AWS fee after that. It might not be the exact figure but there is most definitely a huge markup there.
Re:Cloudflare's Disruption by seabrookmx on Friday October 01, 2021 @01:53AM (#28715105)
And your "business internet connection that is offered by ISPs" is a joke compared to the backbone AWS uses.

Remember those ISPs need to make margin, that's why they charge what they do.

AWS _could_ make it's margin on services alone. They just choose not to.