News for hackers. Stuff that matters.
Re:Ask HN: How to get practical experience in developing sca... by rahimnathwani on Monday March 11, 2019 @09:59AM (#19358994)
1. Get the book 'Designing Data-Intensive Applications', read the causes of scaling problems, and some techniques to deal with them.

2. Take something you've built, and imagine that it faces a specific scaling problem. For example, let's say the most complex thing you've built is a to-do list app. Maybe you can imagine some things that could make it grind to a halt when running as a monolith on a single server. e.g. what if a single user has 10,000 or 100,000 TODOs? Fake that situation (just insert many many fake TODOs into the database). Now see if you've created a scaling problem. If not, try a larger number, or a different dimension (e.g. what if there are many many simultaneous requests?) until you create a problem.

3. Look in that book again and pick a couple of ways you could solve the problem (database sharding across multiple servers? caching? pagination in your API calls?) and implement one of them.

(If you have difficult at step 2, you could try running your app on a virtual machine with very little RAM and disk, but imagine that's the largest machine type available.)

Re:Ask HN: How to get practical experience in developing sca... by anujmehta on Tuesday March 12, 2019 @02:51AM (#19365935)
Thanks. Will definitely start reading the book and try applying things practically
Re:Ask HN: How to get practical experience in developing sca... by vp8989 on Monday March 11, 2019 @03:23PM (#19361964)
It's pretty much "paint by number" on AWS.

The problem with focusing on "scalable" is that wallets and budgets don't scale infinitely and hosting isn't free.

Being able to build stuff that is "scalable" is not enough. You need to be efficient which generally involves "scaling down", ie how much of this data can we be drop because it's useless and no one cares about it?

Re:Ask HN: How to get practical experience in developing sca... by atmosx on Monday March 11, 2019 @01:53PM (#19361097)
Here: https://12factor.net/ - Some considerations which will not be found in the 12FA:

- If you wanna change configuration on the fly, in parallel, your app should support something like Prometheus reload: SIGHUP or http://<ip>/-/reload - rolling updates might be time consuming (you need to separate between boot config and dynamic config)

- Add feature flags to your app.

- Use proper secrets management from day one. Something like Vault or Credstash could go a long way.

- Perform migrations to the CD system, don't use init containers

... these are the ones I can think off the top of my head now, but there are more.

Re:Ask HN: How to get practical experience in developing sca... by diehunde on Monday March 11, 2019 @10:10AM (#19359069)
Re:Ask HN: How to get practical experience in developing sca... by anujmehta on Tuesday March 12, 2019 @03:08AM (#19365990)
Thanks for sharing, I think this is helpful in clearing system design interviews while I am looking ways of creating highly salable apps to quench my thirst :)
Re:Ask HN: How to get practical experience in developing sca... by usgroup on Monday March 11, 2019 @05:18PM (#19362920)
Make a small cluster and test stuff on it ... like anything else.

Most of the scalability stuff is about failure modes and how your system responds to it; so you don’t need stress tests for that.

Know the properties of your system. Eg what carries load? How does it scale? How much can it bare? Then mess with it on your cluster. Etc.

I suggest you start with Kubernetes and try and put together something like a self scaling PostgreSQL Citus cluster . Put data in it and then smash at it with big queries when ready.

Re:Ask HN: How to get practical experience in developing sca... by Djvacto on Monday March 11, 2019 @09:34AM (#19358815)
I haven't done this yet, but you could always try building an app, and using something like stressgrid to test it.
Re:Ask HN: How to get practical experience in developing sca... by gargarplex on Monday March 11, 2019 @11:48AM (#19359825)
I used beeswithmachineguns at an old job :D too much fun!

https://github.com/newsapps/beeswithmachineguns

Re:Ask HN: How to get practical experience in developing sca... by expopinions on Monday March 11, 2019 @08:56AM (#19358575)
One way to get some experience could be to find an open source project dealing with this kind of system and contribute there.

Examples of such open source applications in use for distributed, large scale systems all over the world are tensorflow, elasticsearch, and hystrix; this is just the tip of the iceberg.

Contributions to OSS will also be beneficial if you need to find a job in the business.