Take Hadoop. I wasted a lot of time and money on distributed computing once, almost entirely because of this mistake. I focused on the cool scale-out features, and of course I want to be able to scale out. Scale out is an incredibly useful thing!
If, on the other hand, I had focused on the problem - "If your storage can sustain a transfer rate of X MB/s, then your theoretical minimum time to chew through Y MB of data is Y/X seconds, and, if Y is big enough, then Y/X becomes unacceptable" - then I could easily have said, "Well my Y/X is 1/10,000 of the acceptable limit and it's only growing at 20% per year, so I guess I don't really need to spend any more time thinking about this right now."
The service they provide can't be anywhere near the Facebook/Google league of scale, nor Netflix level of data/performance demand.
That said, volume and velocity aren't the only kinds of scale that technical teams have to grapple with. I've spent enough time at a big organization to understand that Conway's Law costs CPU cycles. Lots of them.
I would rather have cool tech on a resume rather than boring tech. Helps if someone decides to jump to a cool company.
This still sounds like one (like a scientist) starts out by looking at something for it being cool, hip and perhaps useful before considering if it solves any of the problems. The proper working-order is to start by having the problem. Hardly any problem requires a state of the art solution.
When one is wiring a house (where I live) the regulation says you should use the same standards for everything on a group on the switchboard. This hilariously means that if you need to extend iron pipes with canvas isolated wires you have to use metal pipes with canvas wrapped wires.(or rip everything out and replace it with something modern)
Before that, don't worry about the tech, go for what you know, and expect it to need totally re-writing at some point. Let's face it, this stuff moves quickly.
If your business requires enough infrastructure that this kind of tooling is a need at all, it may very well be an important variable on whether you can possibly be profitable or not.
What's the ROI on spending time on planning for that, given that you acknowledge that the response will be mess anyway? In my experience companies that planned for scaling don't handle it noticeably better than companies that didn't plan for it, so maybe it's better not to bother?
Let's put it this way: what's the ROI of failing to address any of the failure modes avoided or mitigated by taking the time to thing things through at design time?
Are you planning on getting paid when you can't deliver value because your service is down?
That's mostly the Dunning-Kruger rearing its head. It makes zero sense to state that making fundamental mistakes has zero impact on a project.
This type of obliviousness is even less comprehensible given that every single engineering field has project planning and decision theory imbued in its matrix, and every single intro to systems design book has its emphasis on failure avoidance and mitigation, but somehow in software development hacking is tolerated and dealing with perfectly avoidable screwups resulting from said hacking is ignored as if they were acts of nature.
Companies that are growing fast enough to need Kubernetes are also changing fast and learning fast.
It makes zero sense to presume you don't need to know what you're doing or benefit from automating processes just because you can't hire a full-time expert.
My top level take on this is that the cues to take things up a level in terms of infrastructure diligence are when the risk budget you’re dealing with is no longer entirely your own. Once you have a banker, a lawyer, an insurance policy, an investor, a big customer, a government regulator, shareholders... ultimately once there’s someone you have to answer to if you haven’t done enough to protect their interests.
And risk management isn’t just ‘what will happen if the server catches fire’, it’s ‘how do I guarantee the code running in production is the code I think it is?’ And ‘how do I roll back if I push a bug to production?’ And ‘how can I make sure all the current versions of the code come back up after patching my servers?’
And it turns out that things like kubernetes can help you solve some of those problems. And things like serverless can make some of those problems go away completely. But of course they come with their own problems! But when those things become important enough that you need to solve for them to answer the risk aversion of sufficient stakeholders, the cost of adopting those technologies starts to look sensible.
> Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great. You have the funds to do things “right”. Your fancy zero-click continuous deployment system saves you thousands/millions a year. But at indie maker, “Hey look at this cool thing I built … please, please someone look at it (and upvote it on Product Hunt too, thx)” scale — the scale of almost every single site on the net — that 0.1% is vanishingly insignificant.
I'm yet to see a company with any o line presence where downtime doesn't eat away their profits.
In the very least, downtime means you're paying money to get no service.
Downtime is fine for someone's blog on his personal website that he setup on a weekend while drinking beer. I'm yet to see a business state that they are ok with getting their site 404 or 500 randomly and during a whole workday, which is what 0.1% translates to.
> Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great.
The software world involves way more than your small side project that you managed to cobble together during a weekend. Things do need to work reliably and predictably. Otherwise not only do you not get your lunch but also your competitors eat it from under your nose. Why is this even being discussed, in HN of all places?
It’s like saying ‘the vast majority of people building houses don’t need to dig foundations’ because you build lego houses, and after all the vast majority of houses are made of Lego, right?
It seems all of the articles/posts that get are attention are the "you don't need it ever" opinions, but those are incredibly short sighted and assume that everyone is recklessly applying complex infrastructure.
In general, it would be nice to have more nuanced discussions around these choices rather than the low hanging fruit that is always presented.