News for hackers. Stuff that matters.
Re:Scaling to 100k Users by majkinetor on Wednesday February 05, 2020 @12:47PM (#22249230)
This is relevant only for multimedia apps.

I have fintech systems in production with 100k+ users with complex Gov app for entire country that runs on commodity hardware (majority of work done by 1 backend server , 1 database server and all reporting by 1 reporting backend server using the same db). Based on our grafana metrics it can survive x10 number of users without upgrade of any kind. It runs on linux and dot net core and Sql Server.

Most of the software is not multimedia in nature and those numbers are off the charts for such systems.

Re:Scaling to 100k Users by bob1029 on Wednesday February 05, 2020 @03:26PM (#22250650)
Thank you for this post. I read "10 Users: Split out the Database Layer" and about had an aneurysm.

I also work with fintech systems built upon .NET Core and have similar experiences regarding scaling of these solutions. You can get an incredible amount of throughput from a single box if you are careful with the technologies you use.

A single .NET Core web API process using Kestrel and whatever RDBMS (even SQLite) can absolutely devour requests in the <1 megabyte range. I would feel confident putting the largest customer we can imagine x10 on a single 16~32 core server for the solution we provide today. Obviously, if you are pushing any form of multimedia this starts to break down rapidly.

Re:Scaling to 100k Users by collyw on Thursday February 06, 2020 @05:46AM (#22255031)
As someone that uses Python and Django most of the time (a "slow" language), I haven't seen much in the way of performance problems at the application level. Usually some database optimization will do the trick.

Actually thinking about it- I have inherited plenty of performance problems at the application layer, but that's because the joins were effectively being done at the application level (and making huge numbers of database calls) when they should have been done at the database layer.

Re:Scaling to 100k Users by thinkmassive on Wednesday February 05, 2020 @04:51PM (#22251344)
It seems like they recommend splitting out the database at the start because using a managed service is much easier than properly managing your own production database.
Re:Scaling to 100k Users by throwaway5752 on Wednesday February 05, 2020 @05:52PM (#22251762)
I can't speak for high performance/near or realtime system people but I would not trust a managed database service for those needs. My experience is that the managed offerings lag behind upstream versions substantially and usually are economical because they are multitenant. So you have a bit less predictability in io wait / cpu queue, lose host kernel level tunings (page sizes or hugepages, share memory allocation, etc), and - not naming names - some managed db services are so behind they lack critical query planner profiling features. That's not even going into application workload specific tuning for various nosql stores. This is a nice article but its audience is people that haven't scaled up a system but are trying to cope with success. It's not great generalized scaling advice.
Re:Scaling to 100k Users by jschmitz28 on Wednesday February 05, 2020 @10:30PM (#22253221)
Those all sound like concerns that probably aren't your top priority when you have 10 users unless your product is performance critical, and then none of this guidance applies anyway. When you have 10 users, your database concerns are more likely to be: Is my database up? Is my database secured? Do I have automated backups? Am I otherwise at risk of losing data? Is the database fast enough for now to not be a blocker to my business? All of which are concerns that tend to be well covered by managed database services.
Re:Scaling to 100k Users by bob1029 on Wednesday February 05, 2020 @07:13PM (#22252253)
All I will say is that our latency getting a business entity in or out of a SQLite database (running on top of NVMe flash) is on the order of tens to hundreds of microseconds. There will never be a hosted/cloud offering that can even remotely approach this without installing some "bring the cloud to you" appliance in your datacenter.
Re:Scaling to 100k Users by throwaway5752 on Wednesday February 05, 2020 @08:40PM (#22252744)
It almost kills me, just 10 years ago it was nearly six figures to get a RamSan that had 2TB storage and did 100k/25k r/w iops (https://www.networkworld.com/article/2268291/less-expensive-...)

Now a WD Blue NVMe does 95k/84k r/w iops at $215 and that's just off their website (https://shop.westerndigital.com/products/internal-drives/wd-..., may be more depending on shipping method...)

That said, it's not a fair comparison and I wouldn't want to run a big service on a single sqlite/nvme setup for more reasons than are worth mentioning, but not prematurely optimizing can take you really far - scale and money - with good design.

Re:Scaling to 100k Users by jbergens on Friday February 07, 2020 @07:24AM (#22265294)
I think we also should be more clear with "10 Users". What does he/she mean? What do we mean?

It could be 10 users that each views a few web pages once every month or it could be 10 heavy users for an internal system that uses the system most of the working day and creates thousands of calls per user to the system each day.

It is as a lot of people have mentioned also very different with information that can be easily cached and things like monetary transactions where you don't want to see old data in reads and therefore often can't cache or use read replica db.

For systems with heavy users I have seen the need to use a few servers even with just 1000 - 2000 users. For a web system the same can happen if you have 1000 users active on the site at the same time (within a minute or so).

Re:Scaling to 100k Users by karambir on Wednesday February 05, 2020 @03:21PM (#22250613)
Yeah, for normal web app, we can easily have 100x users for each step mentioned in article.

For our company, we had more than 500k users with 1 small nginx + 1 medium appServer(with autoscaling, though never needed it) + 1 small cache server and RDS till now. We just added a aws managed load balancer into the setup and think it might be overkill.

For a client with NewsFeed needs, I used a dedicated server(64GB, 2TB space) to run nginx, app, cache, huge elasticsearch and postgres. It was great(and cheap) option for an MVP and let them validate the product for few months with >10k users.

It was awesome to learn a few years ago, how much compute power we don't use.

Re:Scaling to 100k Users by holoduke on Wednesday February 05, 2020 @09:11PM (#22252880)
Yes exactly. I have a an app with about 1m concurrent users at peak hours. For years it is running perfectly fine on a single 8 core machine. Linux - > Nginx -> memcache -> nodejs -> filesystem. Yes filesystem. All content is fetched from files. Another process is populating all files. I pay 40 dollars/m for this self managed machine.
Re:Scaling to 100k Users by mwcampbell on Wednesday February 05, 2020 @09:19PM (#22252918)
What do you do when that one machine goes offline? Say there's a power outage, or a networking issue, or you have to reboot with a patched kernel. Now your 1M concurrent users can't use your application. This is why we go to the trouble to use multiple machines, ideally in multiple availability zones if we're using one of the big three cloud providers.
Re:Scaling to 100k Users by zrobotics on Wednesday February 05, 2020 @10:35PM (#22253232)
What is wrong in this case w/ multiple VPS's? For instance, I run a shipping system that our entire operation depends on. I can't afford downtime either, as that would shutdown our entire shipping department. However, 1 server running on-prem, 1 remote in shared hosting and 2 failover servers on DigitalOcean costs ~$120/month.

What would a managed cloud like AWS or Azure provide besides higher expenses? Last I checked, AWS would be ~$550/month based on their calculator.

IT departments managed redundancy long before cloud was a thing, and it's certainly possible to achieve acceptable uptime without the managed services. I realize that I still rely on DO for their uptime, and I'm pricing renting rackspace for that reason. However, uptime is currently within acceptable levels (ie I haven't had to explain to the CEO why all his employees are standing around). We had tried 2 SaaS vendors before our current on-prem solution, and both had had downtime that cost us money. Both were hosted in AWS, so clearly that isn't the panacea for ultimate reliability. If our on-prem server goes down, I get a call. However, I got calls when our SaaS vendor went down as well; sitting on line with their tech support wasn't any great comfort.

The same graceful failover processes are accessible on machines you control, there's no reason that you have to run on one machine if you avoid the cloud. The biggest hurdle for me to implement this was database concurrency with multiple servers, but a cloud solution wouldn't have done anything to solve that problem.

[edit:] typos and miss-typed AWS pricing due to mobile keyboard.

Re:Scaling to 100k Users by Kiro on Thursday February 06, 2020 @01:59AM (#22254041)
I tell the users to try later. Not every app needs a SLA.
Re:Scaling to 100k Users by jjeaff on Wednesday February 05, 2020 @09:53PM (#22253062)
That could be solved by mirroring the system to another $40/mo machine in another data center.

Redundancy was around long before cloud compute was a thing.

Re:Scaling to 100k Users by tempestn on Wednesday February 05, 2020 @05:46PM (#22251726)
Even though the absolute numbers may well not apply to other types of apps, the general concepts of how to scale do. We have 100k+ users on effectively a single box too (actually two for redundancy, ease of upgrades, etc., but one can handle it), but this is a great overview of how to think about scaling beyond that, however many users that's done at. Honestly when I was reading the article I read that 1/10/100 as more of a unitless degree of scale than actual numbers of humans.
Re:Scaling to 100k Users by Kiro on Wednesday February 05, 2020 @02:18PM (#22250096)
Yeah, it really depends on the kind of app. My app with a couple of hundred thousand users runs on a single $5 Digital Ocean droplet with standard PHP and MySQL.
Re:Scaling to 100k Users by leeoniya on Wednesday February 05, 2020 @04:17PM (#22251066)
> My app with a couple of hundred thousand users

per year? per month? per day? simultaneously? doing what?

it matters.

i ask this as someone who runs a $40/mo Linode with a debian/nginx/node/mysql stack that's definitely 20x over-provisioned for an e-commerce site with 10k daily visitors, 15 simultaneous backend users (reporting, order-entry, CRM, analytics) and 0 caching tricks. i could easily run the site on any 5 year old laptop with an SSD and 8GB RAM.

normalize/de-normalize when needed, understand and hand-write efficient SQL queries (ditch ORMs), choose small/fast libs carefully (or write your own), and you can easily serve 100k users per day on a single cheap VPS with no orchestration/replication/hz-scaling bullshit. definitely can't say the same about 200k simultaneous users - that would need proper hardware, but can still be a single server.

Monoliths Are the Future: https://news.ycombinator.com/item?id=22193383

Re:Scaling to 100k Users by sgtfrankieboy on Wednesday February 05, 2020 @06:10PM (#22251860)
>(ditch ORMs)

Depends on which ORM you use I think. Basic ORMs like Dapper in .NET just map your SQL query result to a object model.

Re:Scaling to 100k Users by sk5t on Thursday February 06, 2020 @01:46AM (#22253968)
Dapper's a "micro-ORM" and quite lightweight. Usually folks intend ORM to refer to Hibernate or Entity Framework, maybe ActiveRecord, etc.
Re:Scaling to 100k Users by Scarbutt on Wednesday February 05, 2020 @05:50PM (#22251748)
hand-write efficient SQL queries (ditch ORMs)

Raw SQL strings or a query builder?

Re:Scaling to 100k Users by leeoniya on Wednesday February 05, 2020 @09:24PM (#22252944)
doesnt matter. query builders are negligible overhead.
Re:Scaling to 100k Users by neillyons on Wednesday February 05, 2020 @05:08PM (#22251451)
What is the e-commerce site?
Re:Scaling to 100k Users by leeoniya on Wednesday February 05, 2020 @09:19PM (#22252919)
i'm working on a blog post about the tech choices & stack. maybe it'll make it to the front page in a month or two - that'll be a good stress test!
Re:Scaling to 100k Users by Demiurge on Wednesday February 05, 2020 @05:16PM (#22251513)
Are you offering a free stress test? :)
Re:Scaling to 100k Users by clementmas on Wednesday February 05, 2020 @06:45PM (#22252086)
Same thing here, TravelMap.net has 100k+ users (total), handles photo uploads and runs on a simple droplet with PHP and JS.

I guess it depends on the daily traffic.

Re:Scaling to 100k Users by jiux on Wednesday February 05, 2020 @07:49PM (#22252480)
Do you mind sharing more details on how you store and manage these photo uploads? Are you leveraging NFS or cloudfront + S3 at all?
Re:Scaling to 100k Users by wackget on Wednesday February 05, 2020 @02:38PM (#22250284)
Wow, what's your app?
Re:Scaling to 100k Users by kernoble on Wednesday February 05, 2020 @05:15PM (#22251499)
Hey, really specific question regarding your deployment. A teammate of mine reported difficulties with the .net SqlServer database driver establishing connection from a linux client (container instance based on the public .net core image) to a SqlServer instance (on windows) . Are you familiar with this problem? I think moving our systems over to .NET Core on linux is the future, but this one experience has somewhat soured the idea for some decision makers and the team managing our db.
Re:Scaling to 100k Users by mikem170 on Wednesday February 05, 2020 @07:33PM (#22252371)
I bumped into that at the office - connecting from a linux server to a microsoft sql server. We were able to connect after installing unixodbc and microsoft's odbc driver for sql server on linux. Note that a local mssql account is required, you can't connect with microsoft domain authentication from a linux box. We haven't finished testing and tweaking, but so far it looks like it's working.
Re:Scaling to 100k Users by kernoble on Thursday February 06, 2020 @02:53PM (#22259648)
Right, the auth from non-windows was a separate organizational hurdle we had to overcome, there was real hesitancy about not using a GMSA windows user to auth with SqlServer.

When you say "local mssql account is required" do you mean a user or some other entity on the linux client? Or do you mean an mssql user that the client with authenticate as?

Re:Scaling to 100k Users by mikem170 on Saturday February 08, 2020 @10:19AM (#22276143)
an mssql user account
Re:Scaling to 100k Users by kernoble on Wednesday February 12, 2020 @08:49AM (#22308765)
Two more questions (if you see this, I know it's been a few days), are you running SQL Server on Linux or Windows? And are you using encrypted connections?
Re:Scaling to 100k Users by NicoJuicy on Wednesday February 05, 2020 @01:34PM (#22249723)
Any tests on progress? I'm going on Postgress because of Bson support and no-sql with sql duplicate fields for search.

Using .net core also.

In general: I agree, there's rarely a case for really using the cloud. Page loads of my E-commerce project are 8ms for basket, I'm wondering what should kill it first on a big load, even without caching. Probably the database, not sure yet.

Re:Scaling to 100k Users by arsenico on Friday February 07, 2020 @05:22AM (#22264664)
It also depends on the nature of the fintech app. Imagine running a 100K + users on a stock trading app for day traders, where latency matters, and a lot of streaming happens.
Re:Scaling to 100k Users by grezql on Wednesday February 05, 2020 @01:05PM (#22249423)
SQL Server is a different beast. You get alot of performance enhancement out of box. Yes its costs money you save tremendous amount of tweaking time and headaches.
Re:Scaling to 100k Users by viggity on Wednesday February 05, 2020 @03:17PM (#22250578)
or you can just use AzureSQL and essentially just pay what it costs for the box, because it is platform as a service. Its far cheaper (and easier to maintain) than what it'd cost to buy a SQL Server license and run it on a VM.
Re:Scaling to 100k Users by dillonmckay on Wednesday February 05, 2020 @09:51PM (#22253053)
A 32 core SQL Server license is $700.
Re:Scaling to 100k Users by mtVessel on Wednesday February 05, 2020 @10:20PM (#22253175)
This page[1] seems to indicate licensing SQL Server (Standard Ed) for 32 cores would cost $59,472 ($3,717 per 2 cores). Where do you get $700 from?

[1] https://www.microsoft.com/en-us/sql-server/sql-server-2017-p...

Re:Scaling to 100k Users by dillonmckay on Thursday February 06, 2020 @12:06AM (#22253565)
Re:Scaling to 100k Users by jabart on Thursday February 06, 2020 @09:16AM (#22256068)
That is a 10 cal license, so you can only have 10 users using it, and that is 10 assigned users. Don't quote me on that but you have to use core licenses for web apps if you have internet users.
Re:Scaling to 100k Users by vonseel on Thursday February 06, 2020 @03:49AM (#22254535)
Those prices are too good to be true, reminds me of the sites that sell windows or office licenses to personal users for like 10 bucks.
Re:Scaling to 100k Users by james_s_tayler on Thursday February 06, 2020 @03:12AM (#22254379)
This site isn't Microsoft?
Re:Scaling to 100k Users by rlander on Wednesday February 05, 2020 @12:52PM (#22249288)
I’ll bite. What’s a “fintech system” with “complex Gov app”?
Re:Scaling to 100k Users by jermaustin1 on Wednesday February 05, 2020 @01:01PM (#22249382)
Looking at his CV [1], it looks like he's done a lot of work inside the Serbiam Treasury.

1: https://gist.github.com/majkinetor/877d5174ba322fbb808cc47a8...