Sure, we have most of the day-to-day grunt work for our applications automated. But good operations is just more. It's more about maintaining control over your infrastructure at one hand, and making sure your customers feel informed and safe about their data and systems. This is hard and takes lots of experience to do well, as well as manpower.
And yes, that's entirely a soft skill. You end up with questions such as: Should we elevate this issue to an outage on the status page? To a degree you'd be scaring other customers. "Oh no, yellow status page. Something terrible must happen!". At the same time you're communicating to the affected customers just how serious you're taking their issues. "It's a thing on the status page after an initial misjudgement - sorry for that." We have many discussions ilke that during degradations and outages.
Agreed
> enthusiasts would be better served DIY; put a beige box in a local colo
I mean, like, can I provision a zero ops bit of compute from <mystery colo provider> for $20/month?
Edit: looked up colo providers in my city- “get started in 24 hours, pick a rack and amperage, schedule a call now.”. Yeaaah, no. This is why people use cloud providers instead.
You can pay for a business class fiber link too. It's about twice as expensive but they have guaranteed outage response times which is really what you pay for.
We've been adding a ton more hardware lately to stay ahead of capacity issues and as you would expect this means the volume of hardware-shaped failures has increased even though the overall failure probability has decreased. There's more we can do to help users avoid these issues, there's more we can do to speed up recovery, and there's more we can do to let you know when you're impacted.
All this feedback matters. We hear it even when we drop the ball communicating.
https://www.fujitsu.com/global/products/computing/servers/pr...
I used to use them a few years ago in a local data centre, and they were pretty good back then.
They don't seem to be widely known about though.
That does sound pretty useful.
So for yourselves, you rack them then run hardware qualification tests?
I'm still wondering about their hardware acceptance/qualification though, prior to it being deployed. ;)
Have to admit it's disappointing to hear about the lack of communication from them, especially when it's something the CEO specifically called out that they wanted to fix in his big reliability post to the community back in March.
https://community.fly.io/t/reliability-its-not-great/11253#s...