> Hire a good ops team

I feel this is one of those "easier said than done" statements...

I'd argue that's only easier said than done if you insist on doing a roll-your-own open-source science project. If you default to established enterprise vendors, finding ops people isn't THAT difficult.

But then the cloud might be cheaper again.

It isn't, and it's not even close. Unless you're running at less than 40% efficiency the cloud is more expensive for infrastructure running 24/7. If you're a shop with 1 server and no IT guy? Sure, probably makes sense to use the cloud - although I'd argue you'd be better off with managed hosting so you actually have a number to call if something breaks.

An organization of even medium size? Not a chance.

> Unless you're running at less than 40% efficiency the cloud is more expensive for infrastructure running 24/7.

Sure, paying extra for a dynamically scalable service is inefficient if your resource needs are flat 24/7/365, but is that realistically the case?

(OTOH, one must not forget that there is a cost to engineer systems to take advantage of dynamic scaling to minimize costs while meeting requirements.)

>Sure, paying extra for a dynamically scalable service is inefficient if your resource needs are flat 24/7/365, but is that realistically the case?

Yes, the vast majority of apps are not 12 factor architectures designed for scaling so nodes can't be brought online and killed at the flip of a switch.

In most enterprises yes. They've got a consistent flat workload 90% of the time. I'm willing to bet gitlab is the exact same way other than a tiny portion of the environment. And, as you mentioned, writing an app from scratch to take advantage of instant scaling is extremely difficult, and sometimes impossible depending on what statefulness is required.

Established enterprise vendors are increasingly promoting cloud offerings, and their pricing reflects that.

Also can be very expensive. If another IT devops type costs 100k per year. How much will that buy in cloud services and time saved?

150 TB/month of data transfer on AWS will cost you 15k/ month the same thing on dedicated provider will be 500/month. And so on.

That's about the fully loaded cost of one engineer, and you may easily need multiple additional people to run your own hardware instead of using AWS.

That's fairly modest traffic and does not include cost difference for the compute and persistence layers. In case of GitLab their dev and ops office is in Ukraine so that difference on traffic cost alone would pay for 3 good ops people.

> The cloud isn't a money-saving tactic.

Most tactics in business are money-saving tactics, and cloud is clearly one: dynamic scaling means only spending now for what you need now rather than spending now based on anticipated future need, and someone-else's-server means economies of scale in ops.

I'd also argue that "sufficient demand" is a threshold that's been rising very fast over the last couple of years and likely to continue doing so.

This also explains why older tech companies disproportionately use their own hardware, even those that aren't all that big. They all made that decision at a time when it made sense and the switching cost is sufficiently high to prevent them from turning back now.

I doubt there are many rational cases now for newer companies to use anything but the cloud. Even at the scale of Snapchat (>3B$ infrastructure spend over next 5 years) and Netflix, it appears the public cloud is still the rational choice.

You should remember that the early cloud unicorns are getting super low rates for their name clout as well. Their discounts are passed as extra costs to everyone else.

So while it may be worthwhile if you're huge enough to get a big discount, that doesn't mean it applies to small or medium size companies

> The cloud isn't a money-saving tactic.

If it requires no or a smaller ops team then it seems like it might be.

Generally you're hitting the cheap part of your ops team. The rack and stack, and dc technicians. You start seeing the real savings when you move up to using higher services like managed databases, and analytics engines.

I think that's right. We tend to see fragmentation w/ our hybrid approach. Teams doing DevOps type stuff with VMware/vRA on internal hardware and also on AWS/Azure. "Customers" of each environment also tend to be different so priority conflicts arise.

Could be solved by more staffing, I suppose...

Anecdotally, I used to be on a two-man team and switching to Azure made life insanely better.

> If it requires no or a smaller ops team then it seems like it might be.

If you needed a team to manage it before, you still need Ops with "the cloud"

> and the costs of cloud service are zooming up, potentially leaving you without the margin to invest.

What do you mean by this, because it sounds like a straight up lie?

Amazon and Google are in an ongoing price war and cut costs regularly.

>What do you mean by this, because it sounds like a straight up lie?

There is no economy of scale with the cloud. As you scale up your service, Amazon/Google happily scale your bill right up with it. When you roll your own datacenter, things get much cheaper per unit (bandwidth, storage, etc) as you get larger.

Possibly outgoing bandwidth. These costs are directly proportional to traffic and regularly 20-30x the unmetered rate for a colocated box

Yes, cloud bandwidth is expensive. I don't disagree with that. But it's not increasing in price.

Obviously if you're growing your costs will grow, but that's basically a tautology. Bare metal costs also rise as you grow.

Will you stop calling me a liar if I correct that from "the costs" to "your costs"? I am assuming that you are growing, and I think there's a provable assumption that for high levels of service, hiring someone else to do it is more expensive than in-house.

That's an entirely different statement than your original one. Saying "the costs of cloud service are zooming up" disingenuously makes it sound like the price of cloud services are going up.

Thanks for the charitable interpretation of my words.

> hiring someone else to do it is more expensive than in-house.

Except you're gonna need between 10 and 100 people to recreate the service in-house, not just one.

Trying to re-do something from scratch in-house is always more expensive than just buying the product.

I get the sentiment, but it really depends: For some services, it's very true. For other services, careful problem analysis will show that you only need one or two features from the full package, which then can be replaced by a very small shell script.

Can you give an example where a shell script would solve the problem? Curious to know

Last month, I needed to add a periodic job to a simple Rails app to snapshot some data and send it to another system. I discussed the design with a colleague, a Rails expert, who went into the pros and cons of various worker-queue configurations. In the end, I added a hidden API call, and deployed a shell script along the lines of

  while true; do
    curl https://rails.app/do/the/thing
    sleep 3600
  done

The actual version is slightly larger because of error checking etc., but that's the gist. So that's one example of replacing (or rather: avoiding) a distributed system (an asynchronous job queue) with a small shell script.

I don't mean to say that shell scripts always can replace more complex systems, but sometimes they're enough.