I did an interview screen with them at the end of last year for a data engineering position (former coworker was high up on the design side of things and he gave a referral). The first bit of the technical question was just "write a SQL query" and boiled down to doing a group by, count, and limit. The second was something like "what if we needed to return this query from a distributed system without going to the database?" without much context.

As any sane interviewer would do, I asked for context around scale, infrastructure, and what "without going to the database" even meant. The interviewer just reiterated the question. I started talking about different approaches depending on context and all the interviewer did was tell me to program a solution. Turns out all he wanted was a time-windowed hash-map to cache values and the fact that it's a distributed system didn't matter at all.

I think that just about sums up their approach to engineering solutions.

https://github.com/google/guava/wiki/CachesExplained

This is often the problem when hiring too fast. New employees don't have context or direction but generally people try to be productive, so they start inventing work.

Management team is new too, and not built up either direct or handle the bandwidth and they also are lacking the direction. Senior leadership and founders are likely busy hiring more folks and potentially trying to sort through the chaos described before.

IMO, pre-product market fit company shouldn't have 500 employees, maybe not even 10. These things don't become easier with more people, usually everything gets harder and slow. You also shouldn't have massive marketing or sales force before you actually have product to sell and have figured out your sales motion.

It seemed that Fast was just trying to shortcut their way to be the next Stripe by hiring people instead of actually working on the fundamentals like the product and business.

If you have a 500 person org that has operated for years you can probably add 10%-20% new people each year without it breaking too bad. But it can be very hard to build a complete function from 0 to 100 people within a year because there is no infra, culture or understanding how things work. The new people cannot be absorbed/aligned because there is no central gravity so people easily end up pulling different directions and spending their time on internal coordination vs actual useful output.

The parent mentioned 5 teams instead of 2. That's director or higher level decisioning. Which is where you frequently end up having someone far enough away from the actual work that needs to be done, starting early to come up with a roadmap. Given fixed deadline and fixed scope, you lean on your PM triangle knowledge and try to get as much headcount as possible to ensure success, because you don't have enough knowledge yet to really decide at what point adding headcount leads to a negative gain. It's usually less about empire building and more about trying to plan things early rather than growing in response to need.

I've been in this situation, as a manager, being told to hire three teams (two being teams of contractors), pushed back to say I should hire only one (the FTE team) as we didn't have enough work for them, was rebuffed and told I needed to hire (the roadmap said we had the work!), hired, and then had leadership panicking about the fact the teams were idle with nothing to do. After a couple months had enough work we could split it (unnecessarily finely) between them so they could all look active, which lasted about a month before the FTE team just started doing it all, as that was more efficient. I didn't stay there long.

This may be true about Amazon but it is not a given in general. Managers can operate in the grey area where they deliver just enough to justify more headcount "in order to deliver more" where the underlying motivation is increasing headcount. Not atypical in political environments like banks etc.

For your huge hires, 1/3 of the people were hired to be fired in the stack ranking, which of course is a huge morale hit.

The ones that aren't setup to fail lose 20-50% of productivity doing machiavellian backstabbing, defensively or offensively, for promotions other culturally ingrained/highly encouraged organizational sociopathy.

With 50% turnover per year (IT stats, not warehouse worker stats), it'll be hard to sustain development as well.

So it makes sense that Amazon would overhire.

Get out as soon as you can.

Heavy stack ranking basically means for a life event, a worker is very likely to get axed. Any health event, having a child, partner has hard times, or management reorg that puts you with a boss that doesn't like you, and the writing is on the wall.

Companies with a good long term focus should concentrate on acquiring and developing long term workers.

Amazon in particular structures its pay (payoffs come from bonuses and vesting 3-4 years in) in such a way that if one takes a perverse sociopathic view of how they manage workers, that management is incented to rug-pull workers three-four years in so the vesting doesn't happen.

Amazon is DEFINITELY sociopathically and perversely managed to do this.

Once that fundamental disdain of your work force becomes ingrained, then all other organizational abuses (stack ranking, hire for fire, backstabbing) is all fair game.

It starts with stack ranking and the fundamental management sociopathy. Which anyone that looks into accounts of Jeff Bezos becomes apparent of the source.

So why has Amazon been successful? Amazon's advantage in the overall corporate competition landscape was simply that they built an integrated business+IT strategy while every other company treated IT as a cost and annoyance.

But they treat their people like utter shit, and as Amazon transitions from an explosive growth company to a more sustained presence, and from AWS from explosive growth to "utility", the now-huge middle management and workforce devolves from stack ranking and sociopathic disdain into a cesspool where only the wicked survive.

https://en.wikipedia.org/wiki/Toxic_workplace

Unfortunately, Amazon continues to ride a growth momentum, so the toxic workplace will be viewed as a positive. Nothing will change until serious incidents occur.

Of course Amazon is huge, so there are workers who may be in less troubled areas. Just keep in mind that toxic areas of companies that derive from corporate practices will inevitably spill over to the "good" parts. Amazon is acquiring massive amounts of pure sociopaths in their management culture, and those sociopaths will "eat" other idealistic parts of the corporate management.

Corporate america is adapting. Better integration of IT and management is going to come to Amazon's competitors. Amazon's apathy to its product fraud and fake reviews, now stretching into a three year ongoing problem shows that it is faltering. Its lies during its AWS downtimes show a blame culture in action.

My company's strategy is get one extremely solid person across all our verticals (Android / iOS / Backend / Infra) to minimise communication overhead and maximise iteration speed (since we throw out half the code we write anyway).

I would be very hard pressed to hire more than 2-3 people in each role even if we were offered 10s of millions.

but ... but ... that makes sense ...

I have found that having fewer good engineers is much better than lots of bad ones.

That approach is treated as heresy, in today's tech industry.

I kept engineers (really good ones) for decades. They had many "life problems" (like divorce, cancer, etc.) during that time, and I kept them on.

It's entirely possible. I have done it. I ran a team that was all "top-shelfers."

Depending on key roles is something that every company in history has had to do. It’s a risk, but one that has been paying off for hundreds of years, for millions of companies.

There are ways to reduce “bus factor,” yet still maintain high Quality work. The corporation I worked for, managed this, but the price is extreme levels of overhead and rigidity. That has its own risks. I was treated as a “cowboy” in our company, and was considered to be borderline “reckless.” Most folks here, would have considered me to be destructively conservative and risk-averse. I got the job done, though. Our team consistently delivered high-Quality solutions to difficult problems, for decades.

The “easy answer” is to keep expectations and demands on labor low. Keep quality to a minimum. Rely on “black box” dependencies. Don’t use advanced development techniques. Squeeze your coders like lemons. Treat every employee the same as every other one, and control for the lowest common denominator.

That’s not how I worked.

It's OK. The industry is safe from me, and my radical views. It was made abundantly clear, years ago, that no one wants people like me in their company, so I have been forced to work with people that can't afford even crappy programmers.

Poor bastards are living the nightmare.

I feel as if I was a very “human” manager, and that seemed to make the difference.

My employees were loyal to me, not the company. Not an ideal situation, but it WFM.

It has changed a bit, but a large part of it is due to HR policies around salary bands. You can have one person do 10x or 5x and another do 1/3x but their salaries are often max .5 factor apart. Ive been in situations where i'd rather reward (perhaps with a vest period) one person with 3x the salary and just skip all the inter-person communications bs, but HR makes it hard.

We were able to do this at hedge funds, and i'm sure small startups can do this, but as companies grow, HR will generally not allow this.

In the 90's there was a joke that they'd take the engineering headcount and multiply it by 10M (or something), subtract a multiple of management and there's your valuation.

The "shiny toys" in this case are employees and the task of managing them.

As a (future?) lead/manager, having more people on your team bolsters your resume. A lot of people would benefit from the company hiring more too as it generates demand in other parts of the business - HR, IT support, etc (which in turn causes more hiring if you need more people to deal with the increased workload). As a higher-up in the company, it looks better on your resume if it was a "big" company with 100+ employees rather than a scrappy garage startup with 3 people, and likewise for investors.

Basically, to move development velocity faster we "load everything". Seems like a really bad idea on the surface. In practice, we only ran into an issue with a single customer that was literally 1000x as large as the other customers. It took about two weeks by 1 engineer to rework a select few calls that were egregiously slow and we were back at it.

----

In other words, even with some arguably inefficient technical decisions, I've rarely if ever seen performance issues at early stage startups.

If you can make the problem go away by spending more in servers always do it.

For startups, the life and death is product market fit and you only get there by iterating on the business as many times as possible, not making something more efficient.

Put engineers on business iterations.

Massive servers are laughably cheap these days. E.g. 16 cores, 128GB RAM is €112 from Hetzner.

In the meantime have you looked at leaseweb?

https://www.leaseweb.com/dedicated-servers#US

I'm planning on running nimbus web services[0] there and there are a few other smaller providers I have earmarked depending on how brave you are:

https://www.defendhosting.com/usa-unmanaged-servers/

https://cc.delimiter.com/cart/dedicated-servers/

https://greenserver.io

[0]: https://nimbusws.com

Spending time on needless infra (scale you don't need, AI you don't need, infra you don't need, things could have been postgres as in this case, ...) both prevents building useful stuff ($-generating features/experiences/...) and increases operational cost (maintenance, debugging, ...). Both the lost revenue growth and the increased costs are compounding. Imagine a maniac steadily piling on a bit more technical debt every day and eat a bit more of the seed corn every day instead of growing it.

A good phrase for this is "playing house": http://www.paulgraham.com/before.html

More like, sounds like management was trying to get as much money as quickly as they could from their VCs before it all imploded. The engineers aren’t at fault that management didn’t have a real product or vision.

I mean, the engineers held up their side of the bargain. They were hired for webscale, the company got webscale…

Now, if this was a more strategic startup, that had tried to hire pragmatic engineers, that insisted on webscale, I would blame the engineers. However, in this case, I think it’s pretty clear that the engineers did exactly what they were hired to do.

It's reasonable to argue that shelling out the US$800 for a server with 128 gibibytes of RAM (or renting one from a cloud provider for US$150 a month) amounts to buying nifty toys for the engineering department. But if you're paying each engineer US$200k per year in salary and a similar amount in benefits, that's US$190 an hour, so each engineer's workday buys you almost four such servers. Your RAM has a minimum transaction time of about 100 ns, while an SSD is more like 10000 ns. If you have 100+ GiB of data to query, the alternative to buying nifty toys is paying the engineering department to do optimization work that wouldn't be necessary if your data was stored in memory that was 100x faster.

The rough equivalency here is that adding one more engineer to your team costs as much as adding 16 dedicated Xeon servers at Hetzner with 128 GiB of RAM each.

Sometimes the best answer is still a relational database, of course! It's often easier to write your queries in relational terms than by looping over hash tables, though probably not if you're using an ORM, and Postgres is delighted to cache all your data in RAM.

At some point, you may grow beyond being able to run your production environment on a single server, at which point splitting things into tiers is worthwhile. (In some cases you start at that point, for whatever reason, but a lot of web services can serve a million users on a single server. It sure sounds like Fast's service would have worked fine on a single server.)

I've done something similar before and it saved countless hours wasted on configuring databases, writing queries, fighting impedance mismatches. Postgres brings a lot of work. With a file you just need a couple of lines to write your memory out to a file and a couple more to read it back. Simple and effective.

Until it stops scaling, but at that point you've established that your business is successful enough that it warrants the effort of doing something shiny. Postgres has a place in this world, but no need to put the cart before the horse.

Note that this basically works 99% of the time now and is part of a learning process that had the whole thing taking 2 - 5 hours every day for several weeks, down to 2 hours a day for several weeks after that, and now down to the nearly inescapable 20 - 40 minutes a day I have now.

Yeah, not doing that would be my own form of playing with nifty toys.

Programmers love to add the latest tech to their resume then move on to a new position within ~18 months

If/when a developer builds deep knowledge of a problem domain, things like framework-of-the-month may become an unwelcome distraction.

Well the best way to reap the benefits of your experience is to run your own company. Not screwing yourself over with BS tech is a competitive market advantage. However you might need to hire people so you can’t pick something too old.

Working for other people it’ll mostly net you less on call headaches.

I've seen a few startups that seemed to overengineer things (or build custom solutions rather than use something off the shelf) in expectation of massive growth.

I also thought it might have been due to resume driven development and the need to keep engineers engaged so they didn't leave.

I've also seen successful startups with a monolithic rails or PHP application that ran on heroku with next to zero custom architecture components.

My experience has been that companies who will customize integrations for you have poorly designed products. By that, I mean, when we ask for a feature, and they bring on an engineer who does some requirements gathering then comes back with some invisible solution. That's a major redflag.

The success of an integration, in my experience, is directly proportional to the amount of the work I'm able to handle as a client.

I'm feeling a bit called out right now (not an owner but that's my experience, too)

I've seen this repeated on HN, but never saw the alleged flak.

They went from a JSON file, to etcd, to SQLite. etcd seems a little misplaced, but presumably it was already in their infrastructure and they thought they could save time leveraging it. The file-based approach seems appropriate for their particular use-case, though. It's not like it's a Rails app.

I’m still of the opinion that whatever you are storing, if your alternatives are JSON or Sqlite, etcd is a really strange/unconventional choice.

Tootally worth the time /s

If your boss's boss or peers signed a contract for $1m a year from Oracle, you're going to use Oracle even if Postgres is a better product. And the only way to use Postgres in this situation is if they don't know you're running it. Which puts you in the uncomfortable situation of wanting to say, "But Oracle sucks, look how successful we've been at using Postgres?".

Basically if you don't already know that someone higher up the food chain is comfortable with saying, "I've made a huge mistake", you aren't going to 'help' them learn to do it by pointing out they fucked up. What they're going to hear is that you're asking them to say, "I just wasted a million dollars of company money on something everybody hates," which not only are they not going to do, but they're going to start thinking about how to shoot the messenger.

As many conversations go in a couple of my hobbies and talking to friends and acquaintances about health concerns, the answer often starts with, "first, get a time machine" and ends with, "or learn to live with it," often with some useful compromises in between.

One of the important things to do early in your tenure at a place is to bid various people in the management chain and those who seem to be on a management track to see how open they are to having their minds changed. Because you want to know this when the stakes are low (also the blowback on low stakes things seems to be less problematic). Then you know if or when a boondoggle is in the making whether you can stop it before it starts, or amplify all of the problems and kill it along the way, or whether you should keep your head down and conspire to keep the 'doggle at arm's length with those who agree with you.

What you'd do in that situation is make the best of that $1M contract you're stuck with anyway, while being flexible enough to migrate away from that solution when it actually makes sense to do so. Just because you've made a huge mistake doesn't mean that making a different choice after-the-fact is more sensible. Hindsight is always 20/20.

Otherwise it’s just a sunk cost fallacy.

I absolutely hate waste and was always proud of myself when I found ways to shave a thousand here and there off our AWS bill, but it was a bit depressing when I went through the thought exercise of "What would it be like for the company if I managed to get everything onto the AWS free tier?" And then compared that to what it would be like for the company if we found a way to make 1 or 2% more in revenue.

Early on, the math heavily favors worrying less about infra spend and more about product enhancements that can build revenue.

(Although often reducing the size of your infra for other reasons - usually around complexity for ops or ease of development can also have the side benefit of reducing cost and it seems worthwhile in those cases.)

YES! This is VERY true

I'm curious to learn why, if there's any more you can share. What about vertical scaling, i.e. bigger machines instead of more of them? Can't that go pretty far?

For everyone that will say "well why..."

1. real products need a generally available shipping unit (SKU) when your customers are enterprise / telco types.

2. for item 1, what is the lowest common thing this type of customer would take - a VM they can run on their existing virtualization setups. the reality of K8s, containers in this type of customers at that point was near zero.

3. there was thoughts about a SKU of a HW appliance, ie, beefy server and just toss CPU/RAM in it and ship that. shipping some stupid HW appliance that we controlled would just be a distraction and a waste of money also requiring 2 shipping artifacts, 2 test setups, etc. not to mention the need to sort for support for HW failure (you can contract with Dell for example to have them label their Kit as your appliance).

I don't understand the issues with using a bigger VM, but I have no reason to doubt you.

So in my now close to 30 years of experience later, working on systems expected to run 24/7/365, I can't tell you what percentage of failures and outages were caused by the redundancy/fail-over/etc software and hardware layered into a system, but its definitely a very significant part of the problem. Frequently because its just that a "layer" someone bolted on, even if that layer was some software engineer designing for "web scale". All the edge cases are frequently not completely thought out, and when you hit one its a lot harder to recover, than simply restarting a single service running on a single server.

I’ve had massive problems with fancy (HPE, IIRC, but it wasn’t called HPE then) raid arrays caused by the controller being a POS. Using plain mdraid would have been far more reliable.

Sometimes I wonder if all the effort people put into protocols that run Paxos and Raft could be better spent running a small number of coordinator servers with manual failover. I’m suspicious that, in many use cases, leader elections cause failures more often than the leaders themselves fail.

Sounds like someone didn't understand redundancy...

There has to be a smart reply to that, like: 100% of pilots of single engine planes don’t make it home after a single engine failure. 100% of pilots of dual engine planes do make it home after a single engine failure.

pushes glasses

So the odds are you don't. Keep it very simple, very small. I can't highlight this enough. A few boxes will get you very far. If you ever actually hit the limits, throw a big party to celebrate success and only then start to think about making your infrastructure a bit (just a bit) more complex.

> [sales] signed up a large number of smaller businesses on the platform. [...] However, integrating these smaller businesses was challenging thanks to several customizations needed for each new customer.

The ratio of revenue per each small customer vs the total cost of integration (and very likely on-going maintenance) was probably at least an amber flag somewhere for those who had visibility of it.

But maybe there was on-going hope that they would be able to sign up a big customer and integrate them before they ran out of money?

At the last company I worked at (won't call them a startup, just a 10-year old small business that somehow kept raising funding but has never made a profit) I was astonished to learn from a new CPO that in our tenth year we didn't know if we were making money from a customer.

We had no idea how individual customers used the application, which was very data intensive, or what that was costing us. The information was there, but no-one ever looked. Every sale was considered a success, even if it turned out to cost us 3x in data costs and customer support. Often customer support alone would make a sale into a loss.

We spent a relatively large portion of our revenue running very expensive servers to respond instantly to searches that no-one ever performed. I created detailed monitoring to show this. I talked about them to everyone, including Product and the CEO. No-one disputed those facts. But instead we had several people essentially dedicated to downsizing application servers and deleting unused S3 buckets, trimming three zeros a month when we could have trimmed five.

I have learned not to assume that warnings are visible, that people pay attention to them if they are visible, or that anyone cares.

They're still circling the drain, still raising money (somehow!) and I doubt anything's changed.

The last 7 years of my life as an SRE. It's maddening.

It was the pet project of another team-lead, that no-one else wanted. Group decision making is bizarre.

Cost can be a difficult problem to solve! I've been working on a project for about six months trying to identify just how much it costs to deliver a unit of the thing we sell. There's just so much variability, one customer might have widgets that are 40kb in size and rarely ever run widget analytics workloads, while another customer has 40mb widgets and can't stop looking at them.

I feel for the whole "how do customers use the application" thing too. Product analytics at most places seems to be a concern long after features are developed.

"So how many customers have been using that feature we spent two years and $20MM developing?"

"Good question."

"Soooo, mywittyname, can you figure out how many people at companies with over 200 seats look at dog pictures after opening an email with a 'C' in the title?"

I think what saved us was the unlimited license we had negotiated for the database server we were using (which wasn’t bad technology), expired and then the company was going to charge us through the nose for any new servers.

That triggered a rearchitecture of our system, into micro services that made sense for the kinds of queries, volume, and load we were dealing with. We did have a lot of data, but many of the queries could be satisfied by a key value store, for example, which didn’t require the same amount of hardware resources.

The original project did lead us to products our customers wanted to buy. Then we just had to change the architecture to fit those actual products.

Places I've worked yearly contracts for B2B have been in the $100k-$5M/yr range.

The customers paying $100k/yr barely get any integrations and feature changes... they go for the ride. The customers paying $1M+/yr get features on the double.

If Fast was doing integrations and customization for customers that were paying $1000 or less a year something was very very wrong.

Unfortunately this can lead to the outcome that the company just loses more money.

from the article: "The average volume of Fast customers was low because most customers were from small businesses. And yet, every small business needed custom engineering work to be done, making integration slow. Several engineers mentioned how they did not understand how spending lots of engineering effort for each small client resulting in little revenue would result in building a company that could be worth $12B one day."

I think their primary problem was a huge valuation—with correspondingly huge expectations—but no real revenue. If you're pulling in $600K revenue, but valued at $500M... you're gonna have a bad time.

You're describing most startups here.