News for hackers. Stuff that matters.
Re:We saved millions in SSD costs by upgrading our filesystem by aaronblohowiak on Tuesday November 09, 2021 @01:17PM (#29165232)
The amount of staff you have to have on-hand and amount of pre-planning (and up-front capital commitment) can all make that very unattractive long after the basic per-GB price would seem to make it attractive.
Re:We saved millions in SSD costs by upgrading our filesystem by bbarnett on Tuesday November 09, 2021 @01:21PM (#29165285)
No. You already have 24x7 staff at this scale. Hardware requires thought and skill, but then so does software. It isn't voodoo.
Re:We saved millions in SSD costs by upgrading our filesystem by koolba on Tuesday November 09, 2021 @01:29PM (#29165404)
It is a different skill though. Going from zero to one for physical infrastructure is a significant leap in both cost and operational process. You need to manage inventory, provide 24/7 physical access, and set up supply chains to ensure you have ongoing availability.
Re:We saved millions in SSD costs by upgrading our filesystem by z3t4 on Tuesday November 09, 2021 @01:44PM (#29165624)
You don't have to do everything in house, you can for example buy servers with a on site support agreement. Then you just have to buy new servers at regular intervals, you don't need to have a guy that can fix a server with a soldering pen.

Same for internet connection, you can buy transit, no need to become your own ISP. So you don't need people who deal with peering agreements, etc.

For electricity you can make a support deal with a local electrician company. You don't need a guy who can build and maintain a custom power supply unit.

It does help however to have someone with basic sysadmin and network skills. But if you don't have that, you will sooner or later screw up your AWS infrastructure too.

Re:We saved millions in SSD costs by upgrading our filesystem by tomnipotent on Tuesday November 09, 2021 @02:51PM (#29166574)
> support deal with a local electrician company

Maybe if you're wiring a closet in your office, but no colo facility is letting you within ten feet of their power infrastructure. The best you're getting is a racked-mounted UPS.

Re:We saved millions in SSD costs by upgrading our filesystem by manquer on Tuesday November 09, 2021 @05:41PM (#29168524)
I think OP means setting up your own DC, for colo most of this is anyway offered by the DC partner, you would only go with something else if there is very good reason not to use what they offer.
Re:We saved millions in SSD costs by upgrading our filesystem by tomnipotent on Tuesday November 09, 2021 @06:11PM (#29168847)
> I think OP means setting up your own DC

This is far more fiscally irresponsible for most companies than just using AWS.

Re:We saved millions in SSD costs by upgrading our filesystem by manquer on Tuesday November 09, 2021 @06:36PM (#29169030)
Fiscally, there is a price point where the economics will make sense .

We can argue whether it is billion dollars or few millions , but there are 1000s companies spending 50million + or even 500 million a year on compute globally.

For a lot of these companies it would make economic sense to keep their own DCs , however hiring that talent and retaining is very very hard, especially if you need even 1-100th of what AWS offers in their IaaS services, this is what will cost a company a lot more than cloud.

It is really shortage in devops/sysads that drives cloud . The talent shortage is just getting worse, newer devops engineers who know only k8s blink seeing a bare metal VM let alone know how to do proper cable management.

Re:We saved millions in SSD costs by upgrading our filesystem by tomnipotent on Tuesday November 09, 2021 @08:07PM (#29169868)
> It is really shortage in devops/sysads that drives cloud

No, it's not. I've have never once met a company that went cloud because they couldn't find someone to rack and configure infrastructure. There's considerably more to building and running a data center than just hiring staff.

Re:We saved millions in SSD costs by upgrading our filesystem by manquer on Wednesday November 10, 2021 @01:05AM (#29171587)
They may not have gone to the cloud because it was difficult to hire, but they are likely to stay there because of it.

This thread is full of comments making it sound that running your own hardware is black magic and cloud is the only way to deploy at scale. Yes it is hard, but not so hard that spending $500 Million/year [1] on cloud is cheaper than running your own DC.

It is even harder for companies that already had investments in DC operations to justify.

Leveraging public clouds for workloads that need flexible scaling or use cloud native services which are not cost effective / feasible to develop in-house it always makes sense to shift, however the lift and shift for everything we have been seeing has no justification for companies already running their own infra unless developers and system admins are demanding cloud, either they want cloud in their resumes or don't know how anymore to write traditional applications.

Many enterprise apps even today are not cloud native or leverage anything beyond basic compute and storage IaaS services and do not have high load variations and not even set to auto-scale, they never had technical or economic reasons to be moved to the cloud if you already had sunk costs running a DC.

[1] We can argue what that number is all day , when including everything you need to do run a DC it is cheaper than on the Cloud. There is a definitely a number, as at very top end of usage companies like Apple or Facebook etc are not sitting on top of AWS and it would never make sense for them at their scale, somewhere in between that point does exist.

Re:We saved millions in SSD costs by upgrading our filesystem by tomnipotent on Wednesday November 10, 2021 @01:52PM (#29178735)
This is a forum about VC-backed start-ups, where the cost of setting up even a basic DC will wipe out most Series A/B fundraising rounds.

The rest of your comment makes it clear you're entirely speculating, like hiring being an obstacle to spending tens of millions of dollars to build a DC.

Re:We saved millions in SSD costs by upgrading our filesystem by manquer on Wednesday November 10, 2021 @09:51PM (#29183908)
The actual linked article is about a company(also VC backed startup and no by Y-Combinator no less ) saving few millions on SSD cost alone by 20% performance improvement. It is a reasonable expectation then the discussions would be at that scale not typical HN reader setup .

I am not entirely speculating, I have been part of the recruitment industry for the last decade, but you have already decided to discard statements from a random stranger on the internet (totally reasonable!) so no point in arguing this further.

Re:We saved millions in SSD costs by upgrading our filesystem by Spivak on Tuesday November 09, 2021 @01:37PM (#29165531)
Y'all are making "buy an asset tag printer", "have a rep from Dell/HP" and "use the data center's remote hands if you need it" sound crazy complicated.
Re:We saved millions in SSD costs by upgrading our filesystem by outworlder on Tuesday November 09, 2021 @02:00PM (#29165890)
> No. You already have 24x7 staff at this scale. Hardware requires thought and skill, but then so does software. It isn't voodoo.

Not necessarily. Hardware requires people to physically replace failed drives and otherwise do on-site maintenance.

In the unlikely event that an AWS volume fails, I can (and have) automation to fix that. While everyone sleeps.

Re:We saved millions in SSD costs by upgrading our filesystem by deeblering4 on Tuesday November 09, 2021 @02:11PM (#29166025)
> Hardware requires people to physically replace failed drives and otherwise do on-site maintenance.

This is the premise of colocation (as opposed to building your own server room). A colo is a secure building with round the clock staff. Hardware vendors offer rapid on-site parts replacements and can gain access via the on-site staff, and the colo has services to perform on-site work like "remote hands" as well.

> In the unlikely event that an AWS volume fails, I can (and have) automation to fix that. While everyone sleeps.

Fault tolerant architectures can be be deployed on colocated hardware too.

Re:We saved millions in SSD costs by upgrading our filesystem by outworlder on Tuesday November 09, 2021 @03:04PM (#29166736)
The point is - I can do any changes I need to the underlying resources programmatically and near instantly, without ever having to talk to anyone. Including cloud provider staff. Or rather, automation can.

There may exist some colo where I can get a server(or storage, or network cards or anything else) added in minutes over an API call but I haven't heard of any. That's usually found on the VPS side.

> Fault tolerant architectures can be be deployed on colocated hardware too.

They can, usually requiring that you specifically setup redundancies and the like. Which is something that you already have for many cloud offerings. Your automation and redundancies sit on top of the vendor's existing redundancies.

For instance, the EBS volume I mention. It is not a disk. Its not even just an array. It's a far more sophisticated abstraction. If there are issues, it can automatically fetch blocks from your snapshots(if the blocks are unmodified, something they also keep track of). Not happy with spinning disks and want a SSD? No need to place a service order to your colo provider, just send an API call and this will be automatically migrated to SSDs without your applications ever noticing the difference (other than the response time) and with zero downtime. Your software could even do this if it notices that the workloads require it.

If an AWS datacenter goes up in flames the systems I manage will still function (and will self-heal, assuming they even get affected, which for big zones they might not be). I don't have to talk to anyone. I can be sleeping and this will still happen.

It's a completely different level of abstraction.

If you want to compare a big cloud provider with either your own datacenter or colocation facility, there's a big disparity in scale. At a minimum, you would have to compare with several interconnected datacenters or colos. You still don't get the abstraction layer.

It's all missing the point though - I was pointing out that software doesn't necessarily need to have 24x7 staff, as the parent poster was pointing out, even for exceptional (but predictable) issues. Sure, you need someone on-call to handle completely unexpected events, but I don't think that was the point being made.

Re:We saved millions in SSD costs by upgrading our filesystem by jjav on Tuesday November 09, 2021 @03:54PM (#29167339)
> There may exist some colo where I can get a server(or storage, or network cards or anything else) added in minutes over an API call but I haven't heard of any. That's usually found on the VPS side.

To be fair, you can't get hardware added in AWS via API call either. What you can do is spin up instances/storage/etc via API call, as long as that spare capacity hardware is already set up, available and ready to be allocated to you. Which you can also do on on-prem hardware.

If you're saying that your utilization is so peaky or unpredictable that you end up needing an order of magnitude more, or fewer, resources available day to day, then you are absolutely correct that provisioning so much spare capacity on-prem would be prohibitive. This is an use-case where AWS excels.

But if your utilization doesn't have dramatic peaks and growth is mostly predictable, then it becomes practical to provision for it on-prem and it'll be a lot cheaper.

Re:We saved millions in SSD costs by upgrading our filesystem by aaronblohowiak on Tuesday November 09, 2021 @06:22PM (#29168924)
Comparing a 6 month leadtime to a 6 minute leadtime is... not fun. Even if you have a stable peak to mean, success happens and that can cause firedrills, etc.
Re:We saved millions in SSD costs by upgrading our filesystem by jjav on Tuesday November 09, 2021 @10:00PM (#29170572)
It doesn't take six months to order a new server. You can often have it tomorrow (for some extra fee) or in a week or so. Done this many times since the first ~20 or so years of my career were running our own datacenters.

You're right, hypergrowth (a subset of unpredictable capacity needs) is a perfect use case for AWS or similar.

The vast majority of companies, for the majority of their corporate life, aren't in these categories (spiky, unpredictably) phase though, so for those, AWS is a large cost premium.

Re:We saved millions in SSD costs by upgrading our filesystem by Robotbeat on Tuesday November 09, 2021 @02:29PM (#29166271)
Okay, but it’s not hard to setup up redundancy and warm spares as well to make it automatic. You don’t need someone physically there.
Re:We saved millions in SSD costs by upgrading our filesystem by fwip on Tuesday November 09, 2021 @01:36PM (#29165522)
Not always true. Some data is intrinsically bigger than others.

If you have a petabyte of chatlogs, sure, you have 24x7 obligations to millions of people. If you have a petabyte of astronomy data, you have like 3 research scientists using it.

Re:We saved millions in SSD costs by upgrading our filesystem by Robotbeat on Tuesday November 09, 2021 @02:30PM (#29166290)
The research scientists DEFINITELY can’t afford to run petabytes of astronomy data on AWS. Source: am a research scientist.
Re:We saved millions in SSD costs by upgrading our filesystem by fwip on Tuesday November 09, 2021 @05:21PM (#29168295)
Oh, for sure. Just that you don't usually have the amount of dedicated staff that was implied.