> - Cost associated with finding an admin who understands how this thing works

Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent. My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server.

> - Time (and cost) it takes to install software, drivers, etc

This is largely an initial cost of a day or two of setup when you start setting up an on-prem setup + for the first server of a totally new model. If you're buying one, then sure, you need to factor in a bit of time. If you're buying more, then if you use a competent admin the second one should be a matter of inserting boot media or (preferrably) configuring PXE booting, and picking an IP.

> My experience is when I've done consulting, clients on AWS used to end up paying more per instance than on-prem clients ended up paying per physical server.

This has been my experience, as well. Clients who switch to AWS for cost-saving reasons will be disappointed.

I think AWS consultants tell the same. I had a training session from Amazon guy arranged by employer. He said that AWS is not going to save us lot of money. But it's value is to enable us move quickly and experiment. Even the managers of my company were aware of that.

I feel like the main advantage AWS has at this point is that "everyone is using it". While it may not give you a competitive advantage over your competitors, it at least doesn't put you at a disadvantage if you are using the same platform as them.

... unless the use case is weird, e.g. you need massive amounts of processing power infrequently.

There certainly are people with use-cases like that, but it is rare to find people who both have large, sudden spikes that are not better handled in other ways (e.g. combination of CDNs and micro-caching), or where the load can not be trivially smoothed out enough (e.g. there's enough work with relaxed latency requirements or that can be scheduled to off peak times, or where the spikes are not small enough relative to base-load that it still pays off to put most load on dedicated equipment and use AWS or similar to handle spikes only.

People often think they have workloads like that. I've done work on plenty of setups over the years were people were certain they had lots of big spikes, but most of them only had moderate day/night cycles (and usually a lot less spiky than they thought, because they'd never bothered to measure).

Only in a few cases I've seen usage patterns that justified spinning instances up/down very often, and for those cases, ironically the availability of clouds have made dedicated hosting more cost effective, because if you host everything in VMs or containers anyway, you can now get to far higher utilization levels on your on-prem/colocated setups, and be prepared to spin up some cloud instances (automatically, if you wish, it's not that much effort), and tie them into your cluster when your load warrants it. It's not suitable for all workloads, but it works surprisingly well if anything.

I do see that argument a lot for using cloud providers, but the vast majority of cloud setups I've seen end up having very static workloads growing at rates that gives plenty of time to prepare.

> People often think they have workloads like that.

This is exactly what I see on the regular. The AWS model promises huge savings if you have these cyclical use cases and rarely does this ever happen in a small business or startup.

Worse, you can have such needs but wind up overprovisioning because your software is tough to autoscale automatically up or down. In that case you should probably pay for dedicated, mostly static servers in traditional setups until you can’t stomach the costs of being overprovisioned anymore then go to a cloud that lets you autoscale to recover a lot of your costs. Sure, reserved instances help substantially but because it’s a cloud instances themselves can go up or down without the best SLAs (I still say I’ve had more physical servers among the tens of thousands seen go belly up than the hundreds of thousands of EC2 instances I’ve seen in my experience).

Other factors are obviously the ability to get a compute instance up immediately or because you have bad operational situations (horrible legacy DC equipment, genuinely bad admins or vendors, etc) that far outweigh the opex and capex parts of the equation and bleed into SLA, poor reputation / relationships, etc.

This does apply to non prod environments. Development/test environments that have low utilization and only run during business hours are better candidates

> ... unless the use case is weird, e.g. you need massive amounts of processing power infrequently

I worked for a large organisation that had this need. It would require ~10K cores and a large number of GPUs, for a period of a few days, approximately once a month (or less).

That sort of ask appears to be too much for amazon, and they requested for us to pre-book the capacity for the whole month, wiping out any cost savings we might have made over just buying the hardware.

> Managing AWS resources is not "free" either, or the market for people to handle devops work on AWS related to ongoing operations would be non-existent.

The costs of finding an AWS admin is much lower. This is, generally speaking, why many orgs still use SAP for ERP since there are a plethora of blue-chip consultancies that support those systems.

Hi, Lambda engineer here (I'm one of the authors). You bring up some good points, I'd like to address some of them:

Admin:

Time (and cost) it takes to install software, drivers, etc

These tasks aren't made unnecessary by cloud. Yes, with cloud, once you've done the work, it can be forever encoded into an image or container. However, the same applies to on-prem using container solutions like Docker.

Maintenance / Capitalization / Finances:

3-year? What is the useful life of this? When does it seemingly become obsolete?

With GPUs of recent history, obsolescence is not a concern in a 3-year time frame. On AWS, people are still using K80s (released in 2014!). The GTX 1080 Ti, which was released 2.5 years ago, is selling for substantially above MSRP. This may change if competition in the GPU space increases and NVIDIA loses its monopoly.

AWS will continually upgrade their hardware and you keep paying the same

True, but this concern is mitigated by slow rate of GPU obsolescence.

Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes)

Can you say a little more about this? Not sure what you're getting at.

Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

This needs to be evaluated on a case-by-case basis. A poorly-capitalized start-up with an unpredictable compute workload is not a good candidate for buying on-prem. A well-capitalized start-up that consistently uses GPUs is a better candidate.

Returns, breakage, warranty in case of a hardware failure

Any reasonable hardware provider will include an option for 3-year warranty.

>Hardware can be capitalized, which means you can push it to the balance sheet (for tax or valuation purposes) Can you say a little more about this? Not sure what you're getting at.

I think he means this: that hardware goes to your balance sheet, then it depreciates by a certain amount over those 3 years and that loss may be tax-deductible.

In other words: on your bank account it looks like an upfront cost, but because you could sell the servers at any time they really look more like a rental in your books, with capital slowly draining away each year as they become worth less.

Correct. Depending on what accounting principals you use, this is typically 3-5 years. It's akin to an airlines buying a Boeing plane. It'll cost them $1B let's say, but it'll actually hit their income over 35 years ($1b / 35), which means on their income statement, only ~$28M shows up per year (simplified example). Most companies are valued and taxed on their income, so this is important to understand.

> Any reasonable hardware provider will include an option for 3-year warranty.

Did you include the cost for that in your scenario?

For a reasonable vendor, a 3 year warranty is have a negligible cost. (~5-10% of the price of the system)

In fact, for most enterprise grade hardware I've ever purchased, the three year warranty tends to be pretty much built into the cost. Usually where I pay extra is to add the fourth and fifth years.

> Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

I think they used the 3-year reserved instance pricing, meaning you can’t just turn it off after a year and stop the cash flow out (at least not at that price).

It is possible to sell (or buy) reserved instances in a secondary market: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ri-marke...

(Author here)

I agree with a lot of your points, especially the fact that TCO is practically impossible to calculate, is highly subjective, and should ideally include opportunity costs which, almost by definition vary from person to person.

I've definitely thought through some of your points. First I want to say that if your load is spiky you have no business buying hardware. No matter how you slice it, you're simply not going to be able to get hundreds of GPUs of throughput with an 8 GPU machine so it's not possible to get the same "product". So for those doing short bursts of hyperparameter search, stick with cloud.

But, like you said, for those with high utilization and known utilization patterns, it makes a lot of sense to go on-prem. Let's just say pretty much anybody who is buying a reserved instance from AWS should consider buying hardware instead.

>- Cost associated with finding an admin who understands how this thing works

Pricing based on what co location facilities will provide you without much search and is pretty generous. $10k per year per server is silly high and should cover that search cost.

> - Try before you buy

You can try essentially this exact machine on AWS. That instance and hardware are almost exactly the same. Now of course this doesn't apply to many instance types.

> - Time it takes for the box to be built, shipped and sent to the data center

5 business days is our mean lead time for that unit. Most workstations are 2 days.

> - Time (and cost) it takes to install software, drivers, etc

It comes with that stuff pre-installed as mentioned in the article. See Lambda Stack for driver / framework / CUDA woes: https://lambdalabs.com/lambda-stack-deep-learning-software

> - 3-year? What is the useful life of this? When does it seemingly become obsolete?

I'll admit it's a long time but 3 years is about right for GPUs from my experience. I personally purchased new GPUs for my workstations in 2012 (Fermi), 2015 (Maxwell), 2017 (Pascal).

> - AWS will continually upgrade their hardware and you keep paying the same

Yes, but only if you never commit to a reserved instance, in which case costs are 3x higher. If you buy a reserved instance you don't get upgraded hardware.

> - Spending $90k instead of $184k in year 1 with the option to turn it off if you want (no longer need). This could be very valuable for a startup who wants elastic spending patterns.

Yea, most of these are people who are already using GPUs really often for internal training infra and are finding it expensive.

> - Returns, breakage, warranty in case of a hardware failure

Our hardware comes with a 3 year parts warranty. Of course, this doesn't pay for your time lost but it's pretty rare to see parts fail and the $10k / year more than covers in-colo swap outs.

I agree with all of your points. I don't think that we really disagree with much here. TCO is hard to calculate :).

K80s are 2014 and people still use them at scale. So 3 years isn't unreasonable, right?

A major drawback of having this sort of hardware on-prem (and this unit in particular) is that it doesn't include local storage for a 100TB+ scale dataset. There's not just the admin cost of hard drives and RAID, but there's the infra cost of synchronizing data to the machine. An average, reasonable, product-oriented Engineering Manager could easily choose cloud over a $100k savings if it means less headaches. Especially since GCloud will give big discounts if they're trying to outbid Amazon. On the other hand, if Lambda offered some sort of NAS solution plus caching software (Alluxio! :), that would make the offering a lot more competitive. Oh and maybe throw in a network engineer to set up peering and such.. ;)

What might make the comparison more compelling is to take anonymized workloads from your customers, examine how often the workload results in idle (wasted) time, and to factor that into the figures. If a user's workload is elastic (many are, especially in R&D), then the user would be drawn towards ML Engine, Sagemaker, Floydhub, Snark, etc... The downside to these services is that they almost always involve expensive copying of data from NAS to worker machines. But if user utilization is high, and the dataset is static, the on-prem machine is a prime investment.

While the savings is a big and noteworthy number, iteration speed is of principle concern to readers here and likely a good segment of your customers. I love the Lambda offerings, but there are a lot of deep learning hackers who can't make the most of bare metal.

The 1080TI server is probably the killer solution at 1/10th the cost. But NVidia legal blah blah blah :(

Thanks for the response! Yes you're right, it is difficult to calculate. I hope people reading my comment took it into the context of a larger view on the process to calculate TCO, as opposed to just directly scrutinizing your products/article.

> 5 business days is our mean lead time for that unit. Most workstations are 2 days.

So this is an advert? Gotcha.

I think if you called out at the top of the article then it would have been a little more honest.

> - Time (and cost) it takes to install software, drivers, etc

I'd imagine this is part of having a system administrator.