Fun trivia: for the longest time, lichess.org - which by any measure is a high traffic, hugely interactive web app - was run on... a single server with 8 CPUs and 24Gb RAM[0]

[0] https://lichess.org/forum/lichess-feedback/hardware-of-liche...

Agree! One can still Keep It Simple and Stupid (KISS).

Our latest MVP - https://www.code-scope.com lets one interactively analyze and document GBs of source such as linux kernel.

It runs on Digital Ocean's $5 instance, build just with Jquery - bootstrap4.

The app binary that runs code-scope.com website is < 10MB including all the frontend and backend code.

Good old tech such as jquery works very well for us.

Complex js front-end tooling is a choice, not a requirement for develop web app.

KISS remains one of the most under-appreciated principles out there.

I still annoy people by talking about my time at the high frequency trading firm, because working there was a revelatory experience. They were operating at a scale and velocity far beyond any place I've worked at, with lower tolerance for downtime, and very little hardware, relatively speaking. The traders had fun, of course, but it was frankly kind of boring to work there as a techie, what with everything ticking along so smoothly all the time.

And the way they got there was through an absolutely fanatical devotion to the KISS principle.

Too often I see developers not thinking about KISS, produce very complex and slow systems. And as a solution to improve the performance, they decide to throw 10x more cloud infra against it instead of thinking a bit harder.

Especially with the current popularity of scale out as a golden hammer.

I've come to think that scale out is a self-fulfilling prophecy. If you choose a solution that's designed to support scale out, then all the mechanisms and limitations (yes, limitations) it needs to adopt in order to make distributed processing work will impose a burden of overhead that you will need to offset by scaling out.

It's actually sad to see that scale-up almost disappeared, especially considering the insanelly powerful HW we have now, just NVMe SSDs are game changer for anything I/O bound. I remember how few years back there were some people in our company completely floored when they learned that StackOverflow ran on 1 SQL server (quite beefy though, 1TB RAM IIRC). Similar 'cloud' (aka 'someone else's datacenter') setup would cost significantly more, but that seems to be forgotten/forbidden knowledge these days. I guess people need to justify those cloud technologies certificates.

Yes, exactly. I don't want to call it the 'new' generation but many have seem to forgotten how efficient systems can be if you build them properly.

I haven't seen any certificates that replace sweat and hard work yet, except in organizations where they cannot keep track of individual developer performance and simply count certificates.

Do you happen to know of any best practices guides that might help others achieve similar results? I’m working on an app that might have to scale very quickly at launch, and I want to do my best to help our team do it right.

No, I don't. And the specific practices at this company would be unpalatable to almost anyone. Package managers for dependency management, for example, were banned, because they made it too easy for the dependency list to grow in an uncontrolled manner.

There was a lot of homegrown software where others would use a library. That was deliberate. If you need five features, but choose an off-the-shelf package with 5,000 features, pretty soon you'll find that the list of features you're using has grown to 50, then 500, and so on, even if you don't actually need any of it. If a toy's in the box, the kids will play with it.

Maybe one thing to try is to focus on imposing constraints, and making people live within them. RAM, # of CPU cores, whatever. Software is like a gas. It will naturally expand to fill all available space.

Simple advise. Start with a 5 dollar droplet and increase ram / disk speed when required. If you reach 64 gigs consider spliting.

What I do is put my database on a droplet and my files on another and it can handles millions of requests. Put them on the same machine and I need to increase ram higher than the cost of multiple small droplets.

If you ever reach those limits consider setting up a load balancer.

In https://www.code-scope.com, the source code search engine database is in the same $5 droplet that run webserver - We are cheap. It is part of <10MB app which hopefully anyone can download soon. We figured the SSD can do 1-3GBs/second transfer, no point to slow it down with separate process or worst, separate network node.

All the web queries are responded in <<< 1 second - so far....

This is good, and it works well. The issue (as others may have pointed out) is not that simple apps _can't_ be produced, it's that they _won't be accepted_ when they are. The same thing happened to TCL/Tk ... it "doesn't look right" so is dismissed. By and large, the driving force seems to be those who want to pay for "sexy dashboards", usually but not limited to those whose primary motivation is initial impact. Aesthetics are everything - functionality is secondary. Snake oil sells. Things that "look old" are laughed out of the room. It's unhealthy.

I built a static site served by S3 & cheap Lightsail boxes that serves a $5B/yr business.

That was 4 years ago and beyond basic patching they've barely touched their infrastructure and it runs around $100/mo.

"that serves a $5B/yr business."

Ok but is that static site primarily generating the $5B in revenue ? That's more relevant to know. Having a static site for a business where the revenue is generated elsewhere is not that big of a deal in my opinion. Yes I get your point that the $5B company is hosting a static site on S3 which is cool but what will be cooler is if THAT static site is the primary revenue driver. I bet it isn't.

I think it's more along the lines that a $5B/yr business would typically have gone to a large contracting firm, and the resulting solution would have been over-engineered with all of the front and back-end stuff and resulted in a higher cost to solve the same problem.

^ You got it completely. Thank you.

Actually probably the output would be worse, because there's a significant amount of domain knowledge in building an effective website in this space. The only companies that are effective at it do it in house or go to specialists in the space.

Car dealership advertising is actually extremely similar in this regard. There are just a few big players working with hundreds to thousands of dealers.

I tried to imagine what kind of website could generate $5B without requiring more than $100/month of hardware. Seems impossible.

Just take a look at the top 10 adwords keywords and let your mind wander. Our business touched 4 out of 10...

Found in a random blog post, top 10 keywords:

- Insurance

- Loans

- Mortgage

- Attorney

- Credit

- Lawyer

- Donate

- Degree

- Hosting

- Claim

Still can't see it. But just to be sure, you're saying that the $100/month website was directly generating most of that $5B?

And not, for example, being just a bridge between users and an army of attorneys?

Qualified leads are king in that business.

Keep in mind that a large number of law firms in this space do not try cases and just generate and sell leads.

Yes, obviously there's a lot more that goes into it, but my point was that websites can be extremely simple in their infrastructure and still be extremely effective.

The online property only accounts for about 10% of the total ad spend (over $200M/yr). The website accounts for exactly 0% of the bread and butter (high volume, low profit) business, but easily 60% or more of the the rest of it (low volume, absurd profit).

I've talked at length about it before in comments. Not really looking for the repeat discussions.

This use case is about where AWS cost effective scalability ends, anything more complicated and you are into multiple thousands a month for a redundant solution for a critical web-app with a database. We used to run prod and stage envs in aws but as our stage environment grew we realized we could on-prem some pretty bad-a* hardware for the cost of a few months AWS billing. Now we run prod on aws, staging and dev locally.

There's a cost maintaining a staging environment on-prem while prod is on a cloud. Slightly different versions of databases, different networking, etc... It could create issues. This does depend on how much you buy into the AWS ecosystem, if it's strictly EC2-only it's not as much of a problem.

Not that I wouldn't do it, a lot of times it does make financial sense.

Second that.. and hardest thing is you don't really know where you are getting charged. i mean you can look at the billing but boy they do add up to 1000s before you know and you are so hooked into auto scaling, s3, rds with multi-az what not..

You don't exactly need to tell me. I have 10^4 machines in multiple clouds right now plus more n datacenters. Cost effective scaling is hard, but you have to design your applications around it.

Tbh, I find most engineering orgs don't have the competency to pull it off.