News for hackers. Stuff that matters.
Re:Cold Showers by VWWHFSfQ on Saturday June 18, 2022 @06:40PM (#31794453)
Haha! Back in ~2014 or so my company was spending nearly $30,000/month on an EC2 "compute-optimized" cluster to transcode live video streams to multiple renditions. One of our engineers said hey, why don't we try to colo some real hardware? We did a test with a single bare-metal 8-core Xeon server and it completely destroyed the performance of the EC2 "compute-optimized" cluster!

After that we colo'd 4 big Xeon servers for about $1,600/month total. Looking back on it now it's just so insane...$30,000? no way.

Re:Cold Showers by jamal-kumar on Saturday June 18, 2022 @10:16PM (#31795526)
I don't use AWS for a damn thing because of exorbitant costs. I just don't get why people think that it's necessary other than that they're the types to get drawn into marketing hype.

There's just so many better things for your company to be spending the money on.

Re:Cold Showers by m_kos on Sunday June 19, 2022 @12:33AM (#31796189)
I agree that cloud compute is usually (too) expensive, but it is sometimes useful and cost-effective. Not long ago, I need to very quickly run a one-off analysis, which required over 200 GB of RAM. It was much cheaper and faster to spin up one VM on GCP for a day than having to order parts, etc.
Re:Cold Showers by zimpenfish on Sunday June 19, 2022 @02:07AM (#31796564)
Similar - wanted to see how much ram gron would need for a particular file and needed much more than the 16GB or 32GB available at home. Quick and easy to spin up a 192GB EC2 to verify it needed ~95GB all told. No way I could have done that without something like AWS.
Re:Cold Showers by zasdffaa on Sunday June 19, 2022 @05:09AM (#31797611)
Depends... set the pagefile to something large, run the program. If it doesn't thrash too hard you can observe it to its largest growth. You'd probably want an SSD. Maybe that would work. Maybe.
Re:Cold Showers by jamal-kumar on Sunday June 19, 2022 @02:11AM (#31796594)
What was the price?
Re:Cold Showers by selcuka on Sunday June 19, 2022 @04:29AM (#31797405)
No idea about GCP but a 32-CPU Digital Ocean droplet with 256GB of RAM costs $1,280/month, which is less than $43/day.
Re:Cold Showers by m_kos on Sunday June 19, 2022 @07:31PM (#31804568)
IIRC, about $120.
Re:Cold Showers by redredrobot on Saturday June 18, 2022 @11:17PM (#31795825)
The argument is - engineers are expensive so why pay for the expertise to setup and run machines? There's just so many better things for your company to be spending the money on.
Re:Cold Showers by jamal-kumar on Saturday June 18, 2022 @11:20PM (#31795835)
What if my company is pretty much all engineers?

We don't let pencil pushers with MBAs anywhere near what we're doing, and it's going great.

I know this isn't the most usual configuration but if undervaluing my skills and trying to bottom dollar on them is going to be their rules, then I'm just going to do my own thing, and they're just going to have to scrape the bottom of the barrel for talent.

I hope the zeitgeist changes any time soon. God knows how many unicorns have been sacrificed with that kind of paradigm which could be successful companies by now.

Re:Cold Showers by nemothekid on Saturday June 18, 2022 @11:42PM (#31795966)
>What if my company is pretty much all engineers?

I don't know how that changes the equation. No one is undervaluing your skills; it's would you rather spend your time driving to a colo center to replace a RAID array or working on $product.

With AWS you are outsourcing an IT team, not just processors and how you approach pricing should reflect that.

Re:Cold Showers by oogali on Sunday June 19, 2022 @12:26AM (#31796164)
I hear this bad argument often (“replacing hard drives”) and I don’t understand why. It’s as if we’re mentally stuck in a bad hacking movie from 1999.

If you’re doing colocation to save money, you’ve also figured out that going to the datacenter sucks and it’s a terrible place to do work.

You’re not building your own servers from scratch, you’re generally purchasing them from a vendor who offers a warranty and optional on-site service.

Or you’re leasing them from a hosting company who will take care of those pesky RAID alarms for you.

You (or your hosting providers) have likely outfitted your server with remote out-of-band access to allow you to get into BIOS or the RAID controller without physically being in front of the server.

And finally, you have remote access to power cycle the server (or a batphone at your hosting provider to do it on your behalf).

I want to say that these datacenter-visit-prevention techniques have been near standard practice for a decade-and-a-half.

Or is it just me and my circle that do this?

Re:Cold Showers by franga2000 on Sunday June 19, 2022 @01:48AM (#31796470)
> Or is it just me and my circle that do this?

Nope, this seems to be the norm. I've worked on a couple colo servers that nobody at the company had ever actually seen in person. They figured out colo in Germany was the best deal, so they had some servers delivered straight to the DC and the staff there installed them and plugged them into an IP KVM. Not sure if this is a standard service most providers offer, but I'm sure a big enough cheque would convince most - and considering the cost of transporting both the hardware and engineer to install it, that cheque can be quite large.

Re:Cold Showers by jollybean on Sunday June 19, 2022 @12:13PM (#31800661)
So you've just explained why 'the cloud' is better than DIY.

Take all those things you just talked about, and expand them horizontally and vertically up the stack, and you have 'AWS'.

So not just 'a guy to replace the hardware' - but now it's software configurable, has all sorts of other, fancy things.

Time is money, and it's expensive to pay people to mess with things if they don't have to.

It's like this:

If your company needs 3 cars, you rent/lease them. You do not hire your own mechanics, even if technically speaking "we could change the oil for so much less!"

If your company is in the business of transportation, and you have thousands of trucks, you may want your own repair/maintenance team etc. instead of paying some service company a fat margin to change the oil.

Re:Cold Showers by balex on Sunday June 19, 2022 @03:58PM (#31802821)
The original discussion was around price-performance of physical servers vs cloud VMs. That being the case, it's not a clean a distinction as you describe it. It would be more along buying a few trucks and taking them to the garage when needed (which is rare in small numbers) vs renting many more vans, for higher margin, just to avoid the garage.
Re:Cold Showers by midasuni on Sunday June 19, 2022 @04:11AM (#31797308)
We do very occasionally go into our telehouse data centres, maybe one person once every year.

Our in house data centres we visit more often, usually to add new equipment, doesn’t take long to walk down stairs.

I can’t think of the last time a hard drive failed

Re:Cold Showers by Fnoord on Sunday June 19, 2022 @06:15AM (#31797977)
Seems to me KVMs also save travels. I don't know, one can make productive travels. There's more tools available for that nowadays.
Re:Cold Showers by jamal-kumar on Sunday June 19, 2022 @02:13AM (#31796611)
Almost as if redundant colo servers on VRRP or CARP costs less like damn
Re:Cold Showers by jamal-kumar on Sunday June 19, 2022 @02:11AM (#31796600)
I'm just really glad not to work in a company under these delusions
Re:Cold Showers by jollybean on Sunday June 19, 2022 @12:07PM (#31800604)
Maybe you need an MBA to help you understand that in many cases, it's incredibly more cost effective to use the cloud, because the marginal savings that could be achieved with on prem hardware are dwarfed by the cost of labour, and especially lost opportunity cost.

For most things 'local prem' is an optimization that usually needs on some degree of scale to justify, or, you have a peculiar setup i.e. a couple of well versed hardware and networking guys who have no problem with a bit of a physical setup. Which can be a bonus.

"I hope the zeitgeist changes any time soon."

No, it won't, it's going in the 'other direction' forever, because the 'economies of scale' at Amazon, it's incredibly difficult for individual engineers to compete with those efficiencies.

Just the opposite of 'being a problem for startups' , the 'cloud' has basically made entire swaths of types of startups possible where they wold not otherwise.

Like everything, you have to use think about it a bit but their costs are really, really transparent (imagine Oracle trying to do it ...).

Re:Cold Showers by ianbutler on Sunday June 19, 2022 @01:15PM (#31801273)
I have never worked for a business with well controlled AWS costs, seems the MBA is failing a lot of people.
Re:Cold Showers by jamal-kumar on Sunday June 19, 2022 @04:30PM (#31803169)
I think the big difference that I'm seeing here is that I don't live in a country where engineers demand +100000/yr salaries
Re:Cold Showers by 9935c101ab17a66 on Tuesday June 21, 2022 @05:25AM (#31821992)
I love that you say “demand” like somehow engineers are forcing companies at gun point to pay their salaries. No, stop. It’s the result of market pressure and actual engineering degrees + peng certifications being hard to acquire and desirable.

What a glib and senseless follow up.

Re:Cold Showers by 112233 on Sunday June 19, 2022 @06:17AM (#31797989)
Never understood that argument. How exactly an expertise of setting up and running (your cloud provider) instances is cheaper than expertise of setting up a physical machine?
Re:Cold Showers by zimbu668 on Sunday June 19, 2022 @01:32AM (#31796406)
So you saved, $28,400 a month. Did that make a material difference to the company? I often tell people above me I could save us $10k a month at AWS and generally the response is, "Yeaaaaaaah that's great, could you do XYZ instead to help us land an additional $1M in ARR?"
Re:Cold Showers by KSteffensen on Sunday June 19, 2022 @02:51AM (#31796848)
The people above you suck. That's a terrible attitude. Signalling that saving money or just generally improving systems doesn't matter will not build a culture of technical excellence and ambition.

Also, it's not peanuts. How many extra developers are $28400 per month?

Re:Cold Showers by zimbu668 on Sunday June 19, 2022 @09:32AM (#31799208)
> The people above you suck.

No, actually they don't. In the time I've been there they've 10x the size of the company. Maintained majority control through multiple rounds of funding. Significantly increased salaries. Provided a great work life balance. Etc.

Why are developers obsessed with how many additional developers they could hire with hypothetical savings?

Re:Cold Showers by tlamponi on Sunday June 19, 2022 @03:19AM (#31797016)
no idea of $ but say they'd get 5000€ a month (quite OK for most places europe), the company would need to pay out a bit more here (for some tax/social benefit things), 7000€ would be pretty realistic, so 4 developers could be hired.

This then means that while op wouldn't immediately add 1M ARR, the original dev and the new devs could soon add 5M ARR (stupid extrapolation, it won't be as much in practice, at least normally)

Re:Cold Showers by southerntofu on Sunday June 19, 2022 @04:34AM (#31797436)
Please note that in many places 5k€/month is a huge sum of money for a salary. Even in western Europe, and in France more specifically, many devs i know are payed 1-2x minimum wage (i don't know a single IT person earning 5k€/month although i'm aware they exist). In France, minimum wage is about 1500€/month (before taxes), to which you add about as much professional taxes and contributions from the employer.

According to my napkin calculus, you could get about 4.5-9 developers (for 1-2x minimum wage) onboard here in France for 28k€/month. I'm betting in other countries with low salaries and a vast talent pool, 28k€/month would get you even more employees.

Re:Cold Showers by tlamponi on Monday June 20, 2022 @03:42AM (#31807371)
I'm working currently in Vienna, and 5k is a wonderful salary for Vienna living standards, don't get me wrong, but it's also really not unheard of if you're good and/or working in a senior role.

In Austria, the "IT Kollektivvertrag" [0][1] (think contract for the whole IT collective/unions) demands a minimum of €2503 brutto (before taxes) per month for developers (those are normally falling under ST1 category), and that 14 times a year, and that's for entry level (as in, not first job but starting at a company). Note also the 14 times a year, where the last two extra salaries (Christmas pay for December and vacation pay for June) is taxed much less (note, we can get bonuses on top of that too).

[0]: https://www.wko.at/service/kollektivvertrag/kv-abschluss-inf...

Note for above PDF (only the short money table the full one can be easily found via searching "Austria IT Kollektivvertrag 2022"):

- The ST1 is for devs, and LT1 for leadership roles.

- "Einstiegsstufe" is Junior, "Regelstufe" is "normal" and "Erfahrungsstufe" is Senior

So if you get hired as senior in a leadership role you'd be entitled to €5521 Brutto salary, 14 times a year, or more depending on your experience/knowledge and your negotiation skills.

Re:Cold Showers by benhurmarcel on Monday June 20, 2022 @07:55AM (#31809208)
You need to account for the total cost to the company, not just the before-taxes salary.

In France a young developer easily costs €100k/year to employ, even if he earns a third of that before taxes.

Re:Cold Showers by azmodeus on Sunday June 19, 2022 @05:37PM (#31803782)
I think you underestimate European taxes as an employer. In general it costs 2x to employ people in my experience. Taxes and social security contributions are massive
Re:Cold Showers by tlamponi on Monday June 20, 2022 @03:21AM (#31807257)
No I don't, I'm employed in Europe in a leadership role and thus talk with the CEO about salaries and general money flow, and I'm also able to read my pay slip, which is quite detailed and also lists all "Lohnnebenkosten" (side/extra costs for employer on top of my brutto money), iow. what the employer really pays.
Re:Cold Showers by zasdffaa on Sunday June 19, 2022 @05:26AM (#31797706)
$340,800/year. If that doesn't make a material difference to your company or department you've never worked where cash is tight. More bluntly: you've been spoilt with excess resources. That's a lot of cash to waste.
Re:Cold Showers by morelisp on Sunday June 19, 2022 @09:43AM (#31799309)
There's now an entire generation of programmers, all the way up through senior/staff level, that's known nothing but the free-money era.

It will be an interesting next few years.

Re:Cold Showers by zimbu668 on Sunday June 19, 2022 @09:23AM (#31799124)
I know how to multiply by 12, thanks. I asked the previous commenter if the $340,800 actually made a difference and he hasn't replied.
Re:Cold Showers by zasdffaa on Sunday June 19, 2022 @09:46AM (#31799338)
I can't speak for him, so for myself: some years ago I had to almost beg for a new disk for a server. A disk. I didn't even bother to ask if we could have a bigger server, no point trying.
Re:Cold Showers by taneq on Sunday June 19, 2022 @04:47AM (#31797510)
This kind of comment (from your management) always confuses me. Can't they hire another developer and do both?
Re:Cold Showers by antisthenes on Sunday June 19, 2022 @06:37PM (#31804197)
Savings is pure profit.

ARR could be whatever. They are not interchangeable or mutually exclusive.

Re:Cold Showers by davnn on Sunday June 19, 2022 @07:14AM (#31798256)
For some use cases though, the cloud is just amazingly cheap and fast. We are currently scaling our (computationally cheap) batch re-training jobs on AWS Lambda and it‘s quite incredible that you can train thousands of models in parallel with TBs of RAM. There is no on-premise alternative.