Like, from work, hosting postgres. At this point, I very much understand why a consultant once said - "You cannot make mistakes in a postgres 10GB or 100GB and a dozen transactions per second in size". And he's right, give it some hardware, don't touch knobs except for 1 or 2 and that's it. The average application accessing our postgres clusters is just too small to cause problems.
And then we have 2 postgres clusters with a dataset size of 1TB or 2TB peaking at like 300 - 400 transactions per second. That's not necessarily big or busy for what postgres can do, but it becomes noticeable that you have to do some things right at this point and some patterns just stop working hard.
And then there are people dealing with postgres instances 100 - 1000x bigger than this. And that's becoming tangibly awesome and frightening by now, using awesome in a more oldschool way there.
I'm sure there are many teams that could design such a network with nearly unlimited resources, but it is entirely different when you have profit margins.
The needle keeps moving doesn’t it? A tremendous breadth of difficult problems can be effectively addressed by pulling together libraries and calling APIs today that weren’t possible before. Today’s hard problems are yesterday impossibilities. The challenge for those seeking to make an impact is to dream big enough.
Anecdotal, but most of the people I've worked with as ICs couldn't give a damn about that. They want dollarydoos.
One of the 10X-ers I know (they exist and are real), told me repeatedly how he'd much rather be doing his own thing. He hates the business needs. But income is important and that's why he's dedicated to doing it. I'm surprised at how focused and good he is given his disposition, and I want to hire him when I scale my business more. Drive and passion are sometimes just spontaneous.
An old CEO of mine even quipped that we were not family and that we were there to do a job. All true. Most of the people doing that job were only there for the money.
Most jobs that drive sales and revenue simply aren't fun or rewarding. There's lots of infrastructural glue and scaling. Tiring, boring, monotonous work. 24/7 oncall work. The money is good, though.
In a way it's the opposite. Things scale up by removing complex interactions. Software gets faster by solving the same problem with less steps.
Any beginner can write to much code and pile up too many software and hardware components.
It takes knowledge to simplify things.
I'd take it a step further, the beginner has no choice but to write too much code/cruft. By definition they lack the experience required to know the options before them, many of which are simpler.
Removing complexity and saving CPU cycles does not feel as good to beginners.
That’s one of the fundamental problems in todays Open Source world: everything’s being optimized for a very rare case.
This is all in stark contrast to services like HBO Max and Disney+ which still stutter and crash multiple times a day. Amazon for some reason treats every season of a TV show and HD/SD versions of movies as independent items in their library. I still haven't been able to download a HBO Max video for offline viewing on iOS without the app crashing on me at 99%.
The problems you mention with Netflix are real, but they have more to do with the business side of things. Netflix recommendations seem crap because they don't have a lot of third party content to recommend in the first place. Their front page layout is optimized to maximize repetition and make their library seem larger. They steer viewers to their own shows because that's what the business team wants. None of these are problems you can fix by reassigning engineers.
Wait, what? Netflix is the absolute worst at this. Every time I log in the interface is different! Netflix could not care less about users having a consistent seamless experience.
But as far as performance goes, I totally agree with you. The performance is impressively good and noticeably better than the other streaming apps I use.
The UX is just so bad in so many ways (UI churn, autoplay, useless ratings, useless categories, recaps that can be watched exactly once, and so on...) it mostly ruins the app for me. The actual video quality is great though.
- by the time HBO Max finish loading, I've already lost interest
- Amazon Prime constantly gives me errors, and it's often hard to find what you paid for and what you have to pay for
- Paramount+ often restart episode from beginning instead of resuming.
- Many leave shit in your queue with a few seconds left for you to "Continue Watching". I still have shows in Paramount+ that I've finished months ago in the queue, and there is no way to delete them without watching end credits. - HBO Max only allows you FF in small fixed intervals
- Plex...used to be okay, now it's pushing its streaming services and works very bad offline
- Apple TV has awful offline experience compared to netflix in terms of UX
Nah, I will take netflix constantly changing rows over shit others do.
Their recommendations used to be world class until they got rid of the user feedback mechanisms.
I don't know how it is in the US, but in Japan, it's even worse than that. Japanese dub and English with Japanese subs are different items (although the UI has a way to choose between audio and subtitle channels), and Japanese subtitles are burned into the video.
Very often, despite all the config being in Japanese, it will complain in Japanese, when starting a video, that there isn't an English dub for the show, as if I had asked for it, which I never have.
In related info for Anime that is definitely not an English dub, it shows the names of the US voice actors and their related works more often than the Japanese voice actors.
Multi-season shows in the Prime Video app might be either separate items or not, depending on the title. Through the Fire TV home, though, it's a huge mess. You may have Season 1, 2, 3 under an item for $Show, with Season 2 being the second half of season 1, but on netflix and season 3 really being season 2, on netflix as well. At the same time, the real season 2 is also a separate item which has 2 seasons, with season 1 being really season 2 on Prime and season 2 being... season 2 on another service. Confused?
It's often super hard to get those non-season 1 items as search results too...
Things are however a little better if only using the Prime video app.
The issue is the business polices surrounding it. The UI itself is user-hostile.
Today I rarely use Netflix, and I wouldn’t pay for it. Periodically I open the app and add to my list those titles that are immediately visible which I know I want to watch, like comedy specials of comedians I’m familiar with, but I don’t inspect further, because if I haven’t already heard about a title from some external sources I trust, it’s not worth my time to check. That list just grows and grows, though titles are often removed as they become unavailable, but I never prune the list, because when I remove one title, I’m taken all the way back to the beginning of the list. Trying is just a wast of time.
It is baffling to me that anyone could interact with Netflix’s current UI and conclude that it was anything but a raging dumpster fire.
Admitedly, it follows the same pattern as Netflix, but I like how it's more responsive and feels way simpler/lighter.
Jargon BS is invading people's heads and it has to stop.
Hulu did that big redesign, and it's extremely pretty to look at, but even after a few years of trying to use it, I still struggle to do anything other than "resume episode". Finding the previous episode, list episodes, etc is always an exercise in randomly clicking, swiping, long pressing, waiting for loading bars, etc.
One thing Netflix really got right as well: the "Watch It Again" section. So many times I want to rewatch the episode I just "finished" (because either my wife finished a show when I leave the room, the kids fell off the table, I fell asleep or wasn't paying attention, etc), and every other platform makes this extremely difficult to find.
Back to Hulu--the only way I know how is the search feature, which is a PITA with a remote.
Which devices are you referring to? I’ve only used the PC and mobile interfaces both of which are quite pleasant.
It's the kind of thing that people run at home on a raspberry pi, docker container or linux server and it consumes almost no resources.
But at our organization this needs to scale up to millions of users in an extremely reliable way. It turns out this is incredibly hard and expensive and takes a team of people and a bucket of money to pull it off correctly.
When I tell people what I work on they only think about their tiny implementation of it, not the difficulty of doing it at an extreme scale.
[1] https://www.foxbusiness.com/technology/5-things-to-know-abou...
It's closer to a live streaming problem than pre-encoded video like Netflix.
Having worked at Netflix I can say that the YouTube problem is much more complex.
I do think there is some temporal logic already present in Google’s algorithm which wasn’t part of the challenge.
Where the CDN boxes go, you can't always just throw more hardware. There's a limited amount of space, it's not controlled by Netflix, and other people want to throw hardware into that same space. Pushing 800gbps in the same amount of space that others do 80gbps (or less) is a big deal.
Most companies maintain internal calculations of these sorts of things, and make rational decisions.
When you say that companies maintain internal calculations of the benefits, would you say that it’s (extremely roughly) something like: $10M benefit, need 5 core engineers + benefits + PM + testing lab etc etc -> we can spend up to $500k per eng give or take.
Or is the $10M one number (that would be held somewhat secretly internally at the company) and the salaries mostly represents where the market is? Does the (salary) market take into account the down-the-line $10M value?
Basically, could those engs negotiate to be paid more, or are they already sort of paid close to exactly what the group they’re part of generates in terms of revenue?
Thanks!
—
I see that you said $10M per person, not for the “network optimization group”. Hmm. So it would be fair to say that the engs are definitely not paid according to the value they generate..? I wouldn’t be surprised by that but just to confirm.
Secondly, yes $10M per employee of revenue or cash flow is pretty reasonable for similar companies. The prioritization is NOT “how many employees per $MM.” The allocation is “what opportunity is the highest $MM return per available employee.”
In terms of negotiation, it really depends on how differentiated your skills are. Short answer is that if you can convince management that it would be difficult to find other engineers who could deliver the optimizations you're delivering, yes, you have leverage.
Very highly skilled engineers in specific niches can basically price themselves like monopolists, because the company can easily figure out how much money they are leaving on the table by not hiring them. This is not like "feature work" engineers, whose value is very nebulous and unknown.
Is that all of Disney or just Disney+?
It doesn't seem like that would be a useful statistic if that includes completely unrelated positions (e.g. does that 20x statistic include Disney employees working at Disney Land/World serving up hotdogs? Because they probably don't contribute much to the streaming service)
Walmart's market cap per employee is probably much, much lower than Disney or Netflix, too. That doesn't mean Walmart is doing anything wrong.
I've done a lot of video processing professionally (the server side stuff, exactly what Netflix does) and Netflix is by far the worst of all the streaming providers. They absolutely sacrifice the quality of the video to save bandwidth costs in aggregate and it shows (or more accurately it doesn't show, all the fidelity is lost).
https://www.wsj.com/articles/SB10001424052702304834704579401...
But it's like, every discussion today must end with something about the pay and head count of engineers.
So it turns out decentralized multi routed is not a good solution for video streaming.
Works amazingly well for watching something front to back if your download speed is fast enough; you'd never know it wasn't being streamed. The hardest part is finding a good torrent for what you want to watch. Ironically the Netflix catalog is one of the most easily available to pirate since people rip it directly from web.
And I'm not sure most people actually care if their home hardware is being used for whatever by the service they're using, or else there'd be pushback on electron apps from more than just HN.
The sense I always got from Netflix's P2P work was that it was heavily tied into the political battles wrt the BS arguments that Netflix should pay for peering with tier 2 ISPs. Did this work there continue much after that problem went quieter?
Good quality, barely any buffering.
The niche content may be too difficult for a "live" streaming experience.
Strictly speaking, the Internet was supposed to help some servers survive and continue working together despite some others being destroyed by a nuke. That is more-or-less the case today: we see how people use VPNs to route around censorship. Whether you were supposed to stream TikTok videos directly from the phones of their authors or through a centralized data hose - i'm not sure that was ever the grand idea.
Also "decentralized" and "monetize" don't go well together because innovation is stimulated by profit margins and rent-free decentralized solutions by definition have those margins equal to zero (otherwise the solution is not decentralized enough).
It's still impressive. A 5x increase at that scale can be a phenomenal challenge. Where do you source the ingredients? Where do you build the factories (plural because at that scale you almost certainly have multiple locations in different geographic locales subject to different regulatory structures). Where do you hire the people? How do you manage it? What about the storage and shipping and maintenance of all the equipment and on and on? How much do you do in house how much do you outsource to partners? What happens when a partner goes belly up or can't meet your ever increasing needs?
Your comment is a great example of what the OP pointed out.
> The menu team comes up with interesting ideas like including kale in salads. The procurement team and suppliers then try to get the menu team to understand the challenges. How do you bring kale to 14,000 restaurants? As one example, when they introduced Blueberry Smoothies in the U.S., McDonald’s ended up consuming one third of the blueberry market overnight.
https://www.forbes.com/sites/stevebanker/2015/10/14/mcdonald...
I couldn't find any other source to back it up, but still wow! That's an absurd number.
In the case in animal product, there are almost certainly major operations worldwide that have been built and financed purely to serve McDonalds demand. They probably have to even build these out well before entering some markets.
They raise a lot of chickens in the US. I’ve hauled chicken nuggets or chicken breasts for McDonald’s in the past quite often.
I can’t even tell you where they grow blueberries.
Taking the software example, you can easily scale from 1 to 100 users on your own machine. You can handle thousands by moving to a shared host. Using off-the-shelf web servers and load balancers will help you serve a million+. From there on you'll have to spend a lot more effort optimizing and fixing bottlenecks to get to tens, maybe hundreds of millions. What if you want to handle a billion users? Five billion? Ten billion? It always gets harder, not easier.
Pushing the established limits of a problem takes exponentially more effort than reusing existing solutions, even though the marginal improvement may be a lot smaller. Getting from 99.9% to 99.99% efficiency takes more effort than getting from 90% to 99%, which takes more effort than getting from 50% to 90%.
You never pierce the scaling wall. It only keeps getting higher.
To add something useful as well besides snark, first of all, there are hard physical limits, which are sometimes well within context (you really shouldn’t try to outcompete light speed for example, relevant in some high-freq trading, infrastructure projects). Then you can try to increase headcount to any number, you won’t produce for example a better compiler. There are simply jobs that are more “serial” - the only way to win at those is to try to employ the very best of the field in a small team.
1M users and 10k employees is not in the range where you have crushingly impactful logistics.
“Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway.“ -Andrew Tannenbaum
Each employee adds some overhead, which requires more employees... which requires more employees.
That's a 2X increase. Now do it again and a half for a 5x. Crazy to say there's a "scaling wall" that once you "pierce" it's easy to scale up. It's the opposite, McDonald's already knows how to supply and sell X McRibs a year, there's no company that's ever sold 5X those McRibs so they have to figure it out themselves.
Anecdotally I experienced this when scaling my software product from 1 --> 10 --> 100's --> 1000's etc. of users.
Thats not to say 2x can't be a substantial challenge, as you pointed out. It gets harder (and IMO more fun) when you're at the bleeding edge of your industry.
This is absolutely not true. The closer you are to peak performance, the harder it is to scale, and the returns diminish heavily. At many major tech companies, there's a huge amount of effort into just 1% - 5% optimizations -- these efforts really require creative thinking and complex engineering (not just "scaling existing processes".) At the volumes these companies operate, even a 1% optimization is quite significant.
If you're on 100M users you're probably scaling vertically. So adding 5x more hardware shouldn't be a problem.
But when you're at 500M all of a sudden it makes sense to optimize further since the capital saved will be the same percentage(ish) but the money is worth peoples time all of a sudden.
I know that we don't care particularly about power savings in the DCs I've worked in, because they're relatively small. While bigtech will do all kinds of shenanigans to save a couple watts here and there, because it's worth it across your hundreds of thousands of servers.
If only things were that simple. https://en.wikipedia.org/wiki/Amdahl%27s_law
Building five "Netflixes" with identical content is possible; the amount of content wouldn't change (it would decrease, the cynic says); you just need parallel copies of everything (servers, bandwidth, etc).
The fun would come in syncing usernames, etc through the system.
It's an entirely different class of problem compared to "acquire resource, convert it, sell it".
Sorry, you gotta overhaul majority of your architecture and its components to scale by every 10x. It's not a single "scaling wall" to break through but it's more of a relentless stream of uphill battles. And this gets even more interesting when you reached to the point where there's no prior art for your problem, usually at hundreds of billions of users.
Making 5 things and making 5,000 things is as different as making 50,000 things and 1m things. There are always cost constraints at each level, and each design can only go so far.
I don't think that's a popular sentiment about Netflix. Twitter, Reddit, Facebook, yes, but Netflix, YouTube, Zoom, not so much.
This is the sort of impressive work that I've never seen scale.
With that said, we are standing on the shoulders of giants. There are tons of other optimizations not mentioned in this talk where removing any one of them could tank performance. I'm giving a talk about that at EuroBSDCon next month.
Believe me or not, I was in a company doing web file streaming in 2009 using Nginx, sendfile and SSL offloading on the NIC.
It was installed by one dude. A standard Linux distro, standard kernel and no custom software. Just compile the SSL offloading kernel module once.
It's strange that you assumed otherwise.
I would like to hope nobody asks that. Video is the one of the, if the not the hardest data plumbing use-case on the internet.
A lot of these tricks being discussed here cannot be applied to Skype calls.
Pre-recorded video streaming is, under the hood, really just a high-volume variant of serving up static web pages. You have a few gigabytes of file to send from the server its stored on to the device that wants to play back the video. As this presentation demonstrates that isn't trivial at scale, but the core functionality of sending files over the internet is what it was designed to do from day one. Because you can generally download video across the internet faster than it can be played back its possible to build up a decent sized buffer which allows you to paper over temporary variance in network performance without the customer noticing.
Realtime video streaming has two variants. One to many Twitch style video streaming is relatively simple, since you can encode video into files and upload them to a server for people watching to download those files. This is how HLS streaming works, and most of the techniques Netflix use to optimise video delivery can also be applied here at the cost of adding latency between the event being streamed and people consuming it. That latency will often sit at about 30 seconds, and people generally find that acceptable.
Skype style realtime video streaming is much harder. You're taking video from one person's camera, and then sending it over the internet to one or more people's device. You can't do any sort of pre-processing on that, or stage the video on servers closer to the consuming users, because you have no way of generating that video until the point your users decide to start talking to each other. Because you can't pre-stage that video you need to be able to establish a network route between the people on a call, potentially in an environment where none of the participants have any open connection from the internet directly to the device they're streaming from. Slight fluctuations in network performance can potentially degrade video delivery to the point of it being unusable. The most common route to deal with that is systems that attempt to establish a direct connection (ideally over a local network) between participants, and if that doesn't work going via relay servers operated by the software provider. These servers provide a single point on the internet all parties can connect to, and then allow passing packets as if they were all on the same network.
I agree but since some optimization is theoretically impossible, perhaps it can be said that optimizable area is much smaller than other services. YouTube (many many videos, high quality, many livestreams, and supports VR) seems to most difficult service to fully optimize.
I'm not a telecommunications guy, but I had some professors back in college explain how difficult and fundamental the research of "ma-bell" was from the 60s through 80s. I'm talking Erlang, C++, CLOS circuits, etc. etc. The innovations from Bell Labs are nearly endless.
Telephone communications is one of the biggest sources of fundamental comp. sci research over the 1950s through 2000s.
Compression removes redundancy from your message. Error correcting codes introduces carefully controlled redundancies, so your message survives transmission errors, so your recipient can read it. And cryptography deals with 'scrambling' your message, so that no one else can read it (and also deal with authentication etc).
A lot of innovations did come out of Bell Labs. But I'm pretty sure Erlang wasn't one of them.
I'm not sure what you mean? Real time communication, both video and audio-only, have much lower latency requirements. You can't just buffer ahead when you have some spare bandwidth, like Netflix or YouTube can.
My point was intended to be that there's the same challenges and more - but it's not something I've thought about in depth (and certainly not had to work with), it maybe wasn't a very good characterisation because it's not the same on the other side either, no large file to serve because at the start of the call it doesn't exist yet for example, so perhaps I take it back.
It is much much harder because you can't do the "cache everything on the edge" solution. If storage was infinitely cheap and small, Netflix could run their entire business by sending a USB stick with every single movie/tv show they have on it encrypted to you, and everything would play locally. This is basically what they do with their edge servers/CDNs.
You can't do that with video calls, because the video/audio didn't exist 1 millisecond ago.
In any case, it's hard to say what the 'greatest' engineering challenge is. You can make almost any kind of engineering really challenging, if you (or the market..) sets yourself a very low cost-ceiling.
Scaling cost-effectively is.
(Just like sending a human to Alpha Centauri is hard, even if you had unlimited funds.)
If Netflix built out more slower servers, that would be acceptable scaling. I don't see any plausible scenario where that becomes too difficult. Even if they had billions of subscribers.
Eg enriching a tiny bit of uranium is 'relatively' easy. Enriching enough of the stuff to build an atomic bomb is a question of scaling. And, yes, in principle you can just use more and more centrifuges etc.
Humanity has already built some small infrastructure off-earth, like the ISS or satellite networks. Also various unmanned probes that toiled away for years in space and on Mars.
Building a habitat big enough to plausibly keep humans alive for the decades or more it would take to get to the Alpha Centauri would involve enormous scaling efforts. And that's not even touching on propulsion, yet.
However, yes, grand feats require more than just scaling.
But we're not scaling for scaling's sake. We're scaling with the number of customers.
If it was profitable to enrich some milligrams by hand, then you'd be able to get more scientists and engineers and better equipment and automate some parts and make even more profit. If you go the centrifuge route, it's not because you had to do the harder methods, it's because it's massively cheaper per gram of product.
Such a scenario is very different from "we can't do it at all, and we need to build complex infrastructure just to make it possible".
* after a certain scale
What I've read they burned a lot of money and hat large problems scaling nevertheless. Which I don't find too surprising, not because they are unable, but because it isn't easy to scale.
From my experience and from what I read scaling people roughly a power of ten is a larger change in an organisation and therefor likely a challenge. For _any_ technical process the boundaries might not be strictly a power of ten but i would say that scaling a power of a hundred is a challenge if this value is not already reached on any process in your organisation.
Scaling to - say - Paramount+ size should not be difficult if you're willing to pay AWS / Azure / GCP 10-100x what it would cost to serve it yourself (which in many cases actually makes sense).
It's possible at Netflix's size, they couldn't just run on AWS anymore. Though, given enough lead time and a realistic growth curve - I'm sure it's feasible.
Obviously scaling manufacturing is not a solved problem like (realistically) scaling network and compute usage.
Is this claim based on some example I should know? Countless companies never achieve product/market fit, but very few I can think of fail because they weren't able to handle all their customers.
I call them asinine because these kinds of architectures aren't written out in one day (let alone 60 minutes) and it's silly to not build an evolving setup (unless you are replacing a legacy system in a company with scale, but then again you have more than 60 minutes...). I wish system design interviews would not just think about if you know how to write high scale designs, but if you know how to build tiny designs that minimize footprint before scale and give maximum flexibility for scaling when its indicated ...
It's such a previous generation thing to be angry at the way that modern development is done. Of _course_ dev is now web heavy and people are pulling in all sorts of libraries to make their lives easier.
But pretending that those same devs wouldn't also be capable of development on lower/less abstracted levels (well sure, maybe not _all_ of them) is insulting.
Serving large content has been solved for decades already. It's much easier and reliable to serve from multiple sources, each at their maximum speed. Want more speed ? Add another source. Any client can be a source.
Netflix artificially restrains itself by only serving from their machines. It is a very nice engineering feat, but is completely artificial. As a user it feels weird to think of them highly when they could just have gone the easier road.
We ended up switching to Fastly for CDN. There's something hidden here though that becomes a problem at Netflix size. We were willing to pay the cloud provider tax, and we didn't dig down into kernel level or storage optimizations because off the shelf was good enough. At Netflixes scale, that adds up to millions of extra server hours you have to pay for if you don't do the 5% optimizations outlined in the article.
The solution I'm talking about is bittorrent. The more people watch your content, the less your servers bear load. That is using the internet to its best potential instead of reverting back to the centralized model of the big shopping mall and its individual users.
They work like garbage in practice for mainstream users.
By creating this optimized system, it makes serving that much video profitable.
Bittorrent is built towards "offline" viewing. Try Peertube for a stack that is more built for streaming and has bittorrent sharing built-in (actually webtorrent, because the browser doesn't speak raw TCP or UDP, but the idea is the same)
Just use Nginx and a backend lang of your choosing.
I wouldn't bother. Unless you use storage at the CDN - which is probably very not cost effective for you.
It's just that they succeeded in streaming market with low competition and great success bring in lot of post facto justifications on how outrageously great Netflix tech infra is.
I mean it may be excellent for their purpose but to think their solution can be industry wide replicated seems not true to me.
In my personal experience lots of companies (admittedly all large companies, but many of which sell their services / software / hardware to smaller companies) have a use for serving hundreds of Gbps of static file traffic as cheaply as possible. And the slides for this talk seem exactly on the money (again from my experience slinging lots of static data to lots of users).
Umm, those solutions exist (from places like AWS and Azure) because Netflix was able to do it without them. The cloud platforms recognized that others would want to build their own streaming services, so they built video streaming offerings.
You have the cart in front of the horse. The out-of-the-box solutions of today don’t exist without Netflix (and YouTube) building a planet scale video solution first.
And your timeline is all wrong too. Netflix didn't even engage with the ISPs about bandwidth until long after moving out of our own datacenter. We started the OpenConnect program specifically to make it easier for ISPs, there was no bullying. The spat you're thinking of is that Comcast didn't want to adopt the OpenConnect but also didn't want to appropriately peer with other networks to give their customers the advertised speeds.
And hardware cost is a hugely relevant parameter. Being efficient with hardware is the difference between profitable streaming at that scale and not profitable.
Someone still has to do the R&D for edge cache? These slides are about Open Connect - their own edge cache solution that gets installed in partners racks (i.e. ISPs and Exchanges). Before things that Netflix and Nginx implemented in FreeBSD, hardware compute power was wasted on various things they discuss in slides.
Yes, you can throw money at the problem and buy more hardware.
How do you think this compares in complexity with real time distributed transactions that spans across several financial partners across the globe?
Building software and productionalizing/scaling it are two very different problems, and the latter is far more difficult. Running a successful company always requires an unlimited number of very smart people who are willing to get their hands dirty optimizing every aspect of the product and business. Too many people today think that programming starts and ends at pulling a dozen popular libraries and making some API calls.