The biggest problem with planning for scale is that engineers often have no idea what problems they will actually run into when they scale and they build useless shit that slows them down and doesn't help later at all.

I've come to the conclusion that the only strategy that works reliably is to build something that solves problems you have NOW rather than trying to predict the future.

Exactly this. Not only would they not know the tech challenges, they also wouldn’t know the business/domain challenges.

The flip side of that is that you end up with spaghetti code that is expensive to add features to and is expensive to clean up when you are successful. Then people in the business implement workarounds to handle special cases that are undocumented and hidden.

No it doesn't. Simple and targeted solutions are not bad code. For example, start with a single postgres instance on a single machine, rather than Hadoop clusters and Kubernetes. Once that is maxed out, you will have time and money to solve bigger problems.

In my experience the drag caused from the thinking to plan for scalability early has been so much greater than the effort to rearchitect things when and if the company becomes a unicorn that one is significantly more likely to become a unicorn if they simply focus on execution and very fast iteration and save the scalability until it’s actually needed (and they can hire a team of whomever to effect this change with their newly minted unicorn cachet).

Not only that but the work to rearchitect for scalability is often appealing, energizing work that lots of people want to do. While the work of maintaining overly complex software full of affordances for scalability scenarios (or use cases) that never come up is rarely as productive or motivating.

The success planners almost always seem to be the same ones pushing everyone to "not overengineer". Uhhhh..

There's writing code to handle every eventuality, and there's considering 3-4 places you MIGHT pivot and making sure you aren't making those pivots harder than they need to be.

This is exactly what I try to do and what I've seen successful systems do.

Laying out adjacent markets, potential pivots, likely product features, etc. is a weekend-long exercise. That can help define both where the architecture needs to be flexible, and just as importantly, *where it does not*.

Over-engineering happens when you plan / architect for things which are unlikely to happen.

The exception is when you have people with skills in particular tools.

The suggestion upthread to use awk is awesome if you’re a bunch of Linux grey beards.

But if you have access to people with particular skills or domain knowledge… spending extra cash on silly infrastructure is (within reason) way cheaper than having that employee be less productive.

Nope, if every person does things completely differently, that's just a lack of technical leadership. Leaders pick an approach with tradeoffs that meet organizational goals and help their team to follow it.

> That's exactly what every architecture astronaut everywhere says. In my experience it's completely untrue, and actually "planning for success" more often than not causes huge drags on productivity, and even more important for startups, on agility. Because people never just make plans, they usually implement too.

That's not my experience at all.

Architecture != implementation

Architecture astronauts will try to solve the world's problems in v0. That's very different from having an architectural vision and building a subset of it to solve problems for the next 3 months. Let me illustrate:

* Agile Idiot: We'll stick it all in PostgreSQL, however it fits, and meet our 3-month milestone. [Everything crashes-and-burns on success]

* Architecture Astronaut: We'll stick it all in a high-performance KVS [Business goes under before v0 is shipped]

* Success: We have one table which will grow to petabytes if we reach scale. We'll stick it all in postgresql for now, but maintain a clean KVS abstraction for that one table. If we hit success, we'll migrate to [insert high-performance KVS]. All the other stuff will stay in postgresql.

The trick is to have a pathway to success while meeting short-term milestones. That's not just software architecture. That's business strategy (clean beachhead, large ultimate market), and every other piece of designing a successful startup. There should be a detailed 3-month plan, a long-term vision, and a rough set of connecting steps.

Another way to say that is that "planning for success" is prematurely optimizing for scale.

Scaling up will bring its own challenges, with many of them difficult to foresee.

This. If you plan for the time you'll be a unicorn, you will never get anything done in the first place, let alone being a unicorn. When you plan for the next 3 month, then hopefully in three month you're still here to plan for the next quarter again.

Are there any tech businesses that failed to succeed because they couldn't scale? Myspace maybe? There are dozens (probably hundreds) of famous examples of companies that self destructed because they bogged themselves down with an elective rewrite or by solving imaginary scaling problems (at scale).

A/K/A "premature optimisation is the root of all evil".

<http://c2.com/cgi/wiki?PrematureOptimization>

<https://news.ycombinator.com/item?id=5673015>