Haha! Back in ~2014 or so my company was spending nearly $30,000/month on an EC2 "compute-optimized" cluster to transcode live video streams to multiple renditions. One of our engineers said hey, why don't we try to colo some real hardware? We did a test with a single bare-metal 8-core Xeon server and it completely destroyed the performance of the EC2 "compute-optimized" cluster!

After that we colo'd 4 big Xeon servers for about $1,600/month total. Looking back on it now it's just so insane...$30,000? no way.

I don't use AWS for a damn thing because of exorbitant costs. I just don't get why people think that it's necessary other than that they're the types to get drawn into marketing hype.

There's just so many better things for your company to be spending the money on.

I agree that cloud compute is usually (too) expensive, but it is sometimes useful and cost-effective. Not long ago, I need to very quickly run a one-off analysis, which required over 200 GB of RAM. It was much cheaper and faster to spin up one VM on GCP for a day than having to order parts, etc.

Similar - wanted to see how much ram gron would need for a particular file and needed much more than the 16GB or 32GB available at home. Quick and easy to spin up a 192GB EC2 to verify it needed ~95GB all told. No way I could have done that without something like AWS.

Depends... set the pagefile to something large, run the program. If it doesn't thrash too hard you can observe it to its largest growth. You'd probably want an SSD. Maybe that would work. Maybe.

What was the price?

No idea about GCP but a 32-CPU Digital Ocean droplet with 256GB of RAM costs $1,280/month, which is less than $43/day.

IIRC, about $120.

The argument is - engineers are expensive so why pay for the expertise to setup and run machines? There's just so many better things for your company to be spending the money on.

What if my company is pretty much all engineers?

We don't let pencil pushers with MBAs anywhere near what we're doing, and it's going great.

I know this isn't the most usual configuration but if undervaluing my skills and trying to bottom dollar on them is going to be their rules, then I'm just going to do my own thing, and they're just going to have to scrape the bottom of the barrel for talent.

I hope the zeitgeist changes any time soon. God knows how many unicorns have been sacrificed with that kind of paradigm which could be successful companies by now.

>What if my company is pretty much all engineers?

I don't know how that changes the equation. No one is undervaluing your skills; it's would you rather spend your time driving to a colo center to replace a RAID array or working on $product.

With AWS you are outsourcing an IT team, not just processors and how you approach pricing should reflect that.

I hear this bad argument often (“replacing hard drives”) and I don’t understand why. It’s as if we’re mentally stuck in a bad hacking movie from 1999.

If you’re doing colocation to save money, you’ve also figured out that going to the datacenter sucks and it’s a terrible place to do work.

You’re not building your own servers from scratch, you’re generally purchasing them from a vendor who offers a warranty and optional on-site service.

Or you’re leasing them from a hosting company who will take care of those pesky RAID alarms for you.

You (or your hosting providers) have likely outfitted your server with remote out-of-band access to allow you to get into BIOS or the RAID controller without physically being in front of the server.

And finally, you have remote access to power cycle the server (or a batphone at your hosting provider to do it on your behalf).

I want to say that these datacenter-visit-prevention techniques have been near standard practice for a decade-and-a-half.

Or is it just me and my circle that do this?

> Or is it just me and my circle that do this?

Nope, this seems to be the norm. I've worked on a couple colo servers that nobody at the company had ever actually seen in person. They figured out colo in Germany was the best deal, so they had some servers delivered straight to the DC and the staff there installed them and plugged them into an IP KVM. Not sure if this is a standard service most providers offer, but I'm sure a big enough cheque would convince most - and considering the cost of transporting both the hardware and engineer to install it, that cheque can be quite large.

So you've just explained why 'the cloud' is better than DIY.

Take all those things you just talked about, and expand them horizontally and vertically up the stack, and you have 'AWS'.

So not just 'a guy to replace the hardware' - but now it's software configurable, has all sorts of other, fancy things.

Time is money, and it's expensive to pay people to mess with things if they don't have to.

It's like this:

If your company needs 3 cars, you rent/lease them. You do not hire your own mechanics, even if technically speaking "we could change the oil for so much less!"

If your company is in the business of transportation, and you have thousands of trucks, you may want your own repair/maintenance team etc. instead of paying some service company a fat margin to change the oil.

The original discussion was around price-performance of physical servers vs cloud VMs. That being the case, it's not a clean a distinction as you describe it. It would be more along buying a few trucks and taking them to the garage when needed (which is rare in small numbers) vs renting many more vans, for higher margin, just to avoid the garage.

We do very occasionally go into our telehouse data centres, maybe one person once every year.

Our in house data centres we visit more often, usually to add new equipment, doesn’t take long to walk down stairs.

I can’t think of the last time a hard drive failed

Seems to me KVMs also save travels. I don't know, one can make productive travels. There's more tools available for that nowadays.

Almost as if redundant colo servers on VRRP or CARP costs less like damn

I'm just really glad not to work in a company under these delusions

Maybe you need an MBA to help you understand that in many cases, it's incredibly more cost effective to use the cloud, because the marginal savings that could be achieved with on prem hardware are dwarfed by the cost of labour, and especially lost opportunity cost.

For most things 'local prem' is an optimization that usually needs on some degree of scale to justify, or, you have a peculiar setup i.e. a couple of well versed hardware and networking guys who have no problem with a bit of a physical setup. Which can be a bonus.

"I hope the zeitgeist changes any time soon."

No, it won't, it's going in the 'other direction' forever, because the 'economies of scale' at Amazon, it's incredibly difficult for individual engineers to compete with those efficiencies.

Just the opposite of 'being a problem for startups' , the 'cloud' has basically made entire swaths of types of startups possible where they wold not otherwise.

Like everything, you have to use think about it a bit but their costs are really, really transparent (imagine Oracle trying to do it ...).

I have never worked for a business with well controlled AWS costs, seems the MBA is failing a lot of people.

I think the big difference that I'm seeing here is that I don't live in a country where engineers demand +100000/yr salaries

I love that you say “demand” like somehow engineers are forcing companies at gun point to pay their salaries. No, stop. It’s the result of market pressure and actual engineering degrees + peng certifications being hard to acquire and desirable.

What a glib and senseless follow up.

Never understood that argument. How exactly an expertise of setting up and running (your cloud provider) instances is cheaper than expertise of setting up a physical machine?

So you saved, $28,400 a month. Did that make a material difference to the company? I often tell people above me I could save us $10k a month at AWS and generally the response is, "Yeaaaaaaah that's great, could you do XYZ instead to help us land an additional $1M in ARR?"

The people above you suck. That's a terrible attitude. Signalling that saving money or just generally improving systems doesn't matter will not build a culture of technical excellence and ambition.

Also, it's not peanuts. How many extra developers are $28400 per month?

> The people above you suck.

No, actually they don't. In the time I've been there they've 10x the size of the company. Maintained majority control through multiple rounds of funding. Significantly increased salaries. Provided a great work life balance. Etc.

Why are developers obsessed with how many additional developers they could hire with hypothetical savings?

no idea of $ but say they'd get 5000€ a month (quite OK for most places europe), the company would need to pay out a bit more here (for some tax/social benefit things), 7000€ would be pretty realistic, so 4 developers could be hired.

This then means that while op wouldn't immediately add 1M ARR, the original dev and the new devs could soon add 5M ARR (stupid extrapolation, it won't be as much in practice, at least normally)

Please note that in many places 5k€/month is a huge sum of money for a salary. Even in western Europe, and in France more specifically, many devs i know are payed 1-2x minimum wage (i don't know a single IT person earning 5k€/month although i'm aware they exist). In France, minimum wage is about 1500€/month (before taxes), to which you add about as much professional taxes and contributions from the employer.

According to my napkin calculus, you could get about 4.5-9 developers (for 1-2x minimum wage) onboard here in France for 28k€/month. I'm betting in other countries with low salaries and a vast talent pool, 28k€/month would get you even more employees.

I'm working currently in Vienna, and 5k is a wonderful salary for Vienna living standards, don't get me wrong, but it's also really not unheard of if you're good and/or working in a senior role.

In Austria, the "IT Kollektivvertrag" [0][1] (think contract for the whole IT collective/unions) demands a minimum of €2503 brutto (before taxes) per month for developers (those are normally falling under ST1 category), and that 14 times a year, and that's for entry level (as in, not first job but starting at a company). Note also the 14 times a year, where the last two extra salaries (Christmas pay for December and vacation pay for June) is taxed much less (note, we can get bonuses on top of that too).

[0]: https://www.wko.at/service/kollektivvertrag/kv-abschluss-inf...

Note for above PDF (only the short money table the full one can be easily found via searching "Austria IT Kollektivvertrag 2022"):

- The ST1 is for devs, and LT1 for leadership roles.

- "Einstiegsstufe" is Junior, "Regelstufe" is "normal" and "Erfahrungsstufe" is Senior

So if you get hired as senior in a leadership role you'd be entitled to €5521 Brutto salary, 14 times a year, or more depending on your experience/knowledge and your negotiation skills.

You need to account for the total cost to the company, not just the before-taxes salary.

In France a young developer easily costs €100k/year to employ, even if he earns a third of that before taxes.

I think you underestimate European taxes as an employer. In general it costs 2x to employ people in my experience. Taxes and social security contributions are massive

No I don't, I'm employed in Europe in a leadership role and thus talk with the CEO about salaries and general money flow, and I'm also able to read my pay slip, which is quite detailed and also lists all "Lohnnebenkosten" (side/extra costs for employer on top of my brutto money), iow. what the employer really pays.

$340,800/year. If that doesn't make a material difference to your company or department you've never worked where cash is tight. More bluntly: you've been spoilt with excess resources. That's a lot of cash to waste.

There's now an entire generation of programmers, all the way up through senior/staff level, that's known nothing but the free-money era.

It will be an interesting next few years.

I know how to multiply by 12, thanks. I asked the previous commenter if the $340,800 actually made a difference and he hasn't replied.

I can't speak for him, so for myself: some years ago I had to almost beg for a new disk for a server. A disk. I didn't even bother to ask if we could have a bigger server, no point trying.

This kind of comment (from your management) always confuses me. Can't they hire another developer and do both?

Savings is pure profit.

ARR could be whatever. They are not interchangeable or mutually exclusive.

For some use cases though, the cloud is just amazingly cheap and fast. We are currently scaling our (computationally cheap) batch re-training jobs on AWS Lambda and it‘s quite incredible that you can train thousands of models in parallel with TBs of RAM. There is no on-premise alternative.

There appear to be slightly weird commercial reasons behind this, because gaming GPUs have great CUDA performance but NVIDIA won’t let you put them in a datacentre. So buying your data scientists gaming laptops (RGB and all) generally works out faster for any reasonable price point. That said, a dedicated server with a decent Xeon and MKL set up correctly generally outperforms CPU-bound stuff.

I think it really depends on your data size. All the benchmarks I can find are on massive datasets, with tens of millions of rows or thousands of columns. I’m sure there are significant performance gains in these situations. Our data just wasn’t big enough.

There was a time when I thought 10s of millions of rows was massive. Now it just send run of mill.

With larger data it really depends on the algorithm. If you must iterate over more than a few GB at a time, GPU memory capacity and bus speeds become prohibitive, while a dead-simple implementation on a single CPU with 100+ cores and TBs of RAM goes brrr.

CPU RAM is generally much slower.

8 channels of DDR4-3200 only provide 200GB/s bandwidth. RTX 3090 has 936GB/s. So even 4 socket Xeon won't catch up.

That would only be an advantage if you had to do multiple passes over the data, otherwise the data would still go through the CPU RAM before getting loaded onto the GPU, no?

Definitely the case in the state of the art stuff like neural networks.

Many if not most other algorithms are iterative. Hell, even sorting is.

When the models get sufficiently big, even a 40GB A100 is not sufficient. Unless you can feed the core quick enough, your performance drops considerably.

GPUs are like heavy flywheels. Getting them up to speed takes some time (copy data, compile and copy the kernels, kickstart everything, etc.), so you need to start them once to get the performance benefits. Otherwise CPU is much more nimble since they're closer to RAM and made to juggle things around.

What prevents gaming GPUs from being used in a data center? Is there some licensing restriction?

Yes, it’s part of the EULA for the driver.

> The updated end-user license agreement (EULA) states: “No Datacenter Deployment. The software is not licensed for datacenter deployment, except that blockchain processing in a datacenter is permitted.” [0]

I guess it's time to invent a blockchain that trains ML models as PoW :)

As a sidenote, I have rented servers with GeForce cards from multiple providers in multiple countries, so this rule doesn't seem to be respected very much. And since it's part of the driver EULA, nvidia can't legally go after server providers, since they don't install any drivers, just build and rent out the hardware. For all they know, all their customers are running noveau.

[0] https://www.datacenterdynamics.com/en/news/nvidia-updates-ge...

My rule-of-thumb is that if you have less than a terabyte of data, you're better off processing it locally, and even that is pretty conservative. Big data is for when you have problem sets that simply will not fit on a single machine. With 4TB hard drives going for about $60, a lot of problems are better solved by simple algorithms in efficient programming languages on a single box.

There are some data sets where you really do need big-data tools, but it's for when you have petabyte-scale data, not megabyte/gigabyte-scale data.

Also depends on the complexity of the algorithm (specifically thinking of large neural networks). We have a model that requires 8 A100s for training due to the size of the activations. No way to replicate that on a local machine and have it train successfully in any reasonable time frame.

However the complexity of the algorithm many times scales with the size of the dataset, either the full corpus or the size of individual examples.

I built this decision tree (LightGBM) compiler last summer: https://github.com/siboehm/lleaves

It get's you ~10x speedups for batch predictions, more if your model is big. It's not complicated, it ended up being <1K lines of Python code. I heard a couple of stories like yours, where people had multi-node spark clusters running LightGBM, and it always amused me because by if you compiled the trees instead you could get rid of the whole cluster.

Wow, very interesting, thanks for this. Daily batch predictions is all we do. I’m the maintainer of miceforest[1], do you think this would integrate well into the package at a brief glance? I’m always looking for ways to make this package faster.

[1] https://github.com/AnotherSamWilson/miceforest

I had a brief look at your package, and my impression was that it's only changing model training. If this is correct then the format of the model.txt (calling `lgbm.save(model, "model.txt")`) is the same as regular lightgbm. This would mean you can use my library for inference.

I found the same thing when doing video transcoding. The VPSs were all woefully underpowered. Netcup bare metal (root servers) ended up getting pretty close and were by far the best bang for the buck of anything I found.

Curious what the setup of VPS' was and why you would expect better than real hardware, video transcoding is quite a beast from what I remember and I just can't imagine there's a VPS solution that expects to keep up

The Intel Xeon processors that cloud providers typically use don't have the Intel Quick Sync core that provides hardware A/V encoding/decoding on typical desktop/laptop CPU SKUs. So the software has to fall back to CPU-based codecs, which are much slower.

AWS EC2 has a VT1 instance family that enables high-speed A/V encoding via a Xilinx media accelerator card.

IIRC Quick Sync encoded with poorer quality than software; is that not still the case?

AFAIK that should still be the case.

This comment reminded me S3 ViRGE, the original "3D decelerator chip".

Oh yeah I love simply avoiding memory allocation at all costs and keeping things to the processor cache and streaming APIs. awk/sed is fantastic for this if you're just working with CSV data, but I've done it in my own custom code processing hundreds of gigabytes of JSON in seconds as well.

I think data scientists just aren't really hugely concerned with programming optimizations or bottlnecks or whatever. Most of them are just intermediate-level python programmers, and that's completely fine until they think they need a hadoop cluster for whatever they're doing and the costs start piling up.

That seems like a problem of matching the workload to the hardware- graph computing isn’t embarrassingly parallel in the regular sense.