News for hackers. Stuff that matters.
Re:EC2 Instances with 6, 9, and 12 TB of Memory by mmt on Thursday September 27, 2018 @08:29PM (#18090058)
> $802,538.64 for the 3-year term. I wonder how that compares to building your own on-premise host with that much RAM

This was discussed a couple days ago in a different thread [1], when 4TB was the EC2 limit.

$400k for the current-gen (224 cores) or, as a sibling comment [2] notes, $250k for the previous CPU generation (which can use twice as many DIMMs of half the density and only 192 cores of that).

> obviously, there's operational costs to consider as well

We can estimate an upper bound on this, given that the current-gen system has N+2 5x1600W 96% efficient PSUs, so 5kW max. If you're paying a colo $.50/kWh, that's another $66k over 3 years worst case.

Realistically, though, CPUs with a max TDP of 1640W, DIMMs (generously) 700W, leaves plenty of room for fans, SSDs and other overhead before getting to 3kW or $40k, and that's still assuming running full-bore the whole time.

There are obviously also ancillary costs to AWS, such as data transfer and EBS.

If you're already running your own hardware, it sure does look more attractive to pay $400k now and $40k over 3 years than to pay AWS $800k All Up Front (now). If not, perhaps it's more attractive to hand-wave away that $360k as saving the "hassle" of hiring someone who knows how to run equipment in a datacenter.

[1] https://news.ycombinator.com/item?id=18041486

[2] https://news.ycombinator.com/item?id=18089267

Re:EC2 Instances with 6, 9, and 12 TB of Memory by packetslave on Friday September 28, 2018 @01:27AM (#18091274)
One thing missing from this analysis is the networking cost. The EC2 instance comes with 14gb/sec of dedicated EBS bandwidth and 25gb/sec of network bandwidth. That implies a 40gb NIC, which will require a 40gb top of rack switch and enough cross-sectional bandwidth in the datacenter fabric to handle however many of these machines they plan for (at whatever oversubscription ratio, if any).

That's a pretty non-trivial amount of expense to setup a network with enough capacity to support a machine like this fi you want to do it yourself.

Re:EC2 Instances with 6, 9, and 12 TB of Memory by mmt on Friday September 28, 2018 @02:21AM (#18091410)
> That implies a 40gb NIC

It may imply that but only outright requires 40Gb/s total, which is provided by the "4-port 10GBase-T Ethernet SIOM (Intel® X550 controller)" of a SuperServer 7089P-TR4T.

The EBS bandwidth can be elided entirely by using local storage instead. Local storage bandwidth can be much cheaper than ethernet, especially considering how many NVMe ports are already included.

As 10GE is maybe $200/port (including SFPs), at scale, network cost doesn't "move the needle" on a $400k server.

Re:EC2 Instances with 6, 9, and 12 TB of Memory by mmt on Saturday September 29, 2018 @12:31AM (#18098723)