News for hackers. Stuff that matters.
Re:A lot of complex “scalable” systems can be done with a si... by IndrekR on Sunday December 29, 2019 @05:27PM (#21909018)
A site for proof. It keeps amusing me on what hardware/software Stack Overflow/Stack Exchange is running on: https://stackexchange.com/performance

This is way less in HW than most people in the trade (from web devs to devops) seem to think when asked about it.

SO ranks #36 in Alexa right now: https://www.alexa.com/siteinfo/stackoverflow.com

Re:A lot of complex “scalable” systems can be done with a si... by riyadparvez on Sunday December 29, 2019 @08:22PM (#21909966)
One thing to keep in mind that their work-load is very ready-heavy which eases things a lot when scaling the system. The same is true for Wikipedia. Scaling a write-heavy workload is way more complex than scaling a read-heavy workload.
Re:A lot of complex “scalable” systems can be done with a si... by manigandham on Sunday December 29, 2019 @09:01PM (#21910153)
SO content changes all the time. Votes, comments, moderation, edits, tagging, search and recommendations, etc. There are also real-time community features. It's not as simple as it seems.
Re:A lot of complex “scalable” systems can be done with a si... by RhodesianHunter on Monday December 30, 2019 @12:13AM (#21910971)
They said read heavy, not read only.

I guarantee you the bulk of traffic to SO is hitting a page and performing zero writes.

Re:A lot of complex “scalable” systems can be done with a si... by manigandham on Monday December 30, 2019 @12:59AM (#21911147)
They log every single pageview to SQL Server. There are plenty of writes to various other counters, recommendation system, message inboxes, analytics, etc. Also the ads system, although I'm not sure how much of that is still 3rd-party.

Yes it's read-heavy but there's still plenty of work done in assembling a page. It's definitely not as simple as caching at the CDN edge for every hit.

Re:A lot of complex “scalable” systems can be done with a si... by dvirsky on Monday December 30, 2019 @01:43AM (#21911311)
IIRC most of the "real time" stuff is done on Redis, for which this QPS load is usually a joke (depending on requests of course, but counters etc are easy)
Re:A lot of complex “scalable” systems can be done with a si... by fock on Monday December 30, 2019 @10:17AM (#21913818)
so, would you mind showing any non-FANG, but write heavy site? Besides that, even the operation of most social networks should be very easy to be split into town-sized instances with batch-updates in-between instances (see mastodon specs https://github.com/tootsuite/documentation/blob/master/Runni...). What actually costs a lot is constantly surveilling all user interactions, data-warehousing that and playing out ads. None of which is necessary, and a 20ct/per user/per month should cover the actual service nicely...
Re:A lot of complex “scalable” systems can be done with a si... by jjeaff on Monday December 30, 2019 @03:08PM (#21916233)
Most any saas offering is going to be very write heavy in comparison.
Re:A lot of complex “scalable” systems can be done with a si... by rbanffy on Tuesday December 31, 2019 @05:34AM (#21920110)
All that can be implemented as separate services. Doing that also enables graceful degradation, as a failure of, say, voting doesn't prevent me from reading the question.
Re:A lot of complex “scalable” systems can be done with a si... by hliyan on Sunday December 29, 2019 @08:47PM (#21910083)
There is a significant amount of question submission, commenting and voting going on on SO.
Re:A lot of complex “scalable” systems can be done with a si... by yawn on Sunday December 29, 2019 @09:03PM (#21910162)
A slim subset of humans typing things is not what I would label "write heavy". Write heavy is more like 100k+ devices out in the field sending their current position every 10s. That's still very manageable, but requires some thoughtful design.
Re:A lot of complex “scalable” systems can be done with a si... by Retric on Sunday December 29, 2019 @10:27PM (#21910582)
10k simple updates per second is hardly high bandwidth. I knew people handling similar workloads on a 1U server 10 years ago.
Re:A lot of complex “scalable” systems can be done with a si... by pushpop on Monday December 30, 2019 @04:38AM (#21912148)
Indeed and I literally was. And the back end was written in mod_perl (which isn’t exactly known for performance). Even back then 10k/s was our smallest service - both on income as well as traffic - so there wasn’t the incentive to rewrite it in a faster language.
Re:A lot of complex “scalable” systems can be done with a si... by stefan_ on Sunday December 29, 2019 @11:46PM (#21910884)
The Raspberry Pi 3 in my cupboard without a heat sink on can do 80000 Redis write ops/sec.
Re:A lot of complex “scalable” systems can be done with a si... by fnord77 on Sunday December 29, 2019 @08:50PM (#21910094)
> That means we transfer 55 TB data / month

this isn't a lot. Helps that site is mostly text data.

I know of a relatively small cloud security system that transfers petabytes/month to/from a handful of customers.

4 ingest pods, 8 pipeline pods, 7 time-series db servers, 2 sql servers

Re:A lot of complex “scalable” systems can be done with a si... by manigandham on Sunday December 29, 2019 @08:59PM (#21910138)
The amount of data transferred is meaningless to compare. SO is a relatively complex site with dynamic content and real-time features.

Compared to other similar sites like Reddit or Quora which are far slower yet running on more hardware, it shows what proper efficient architecture and code can do.

Re:A lot of complex “scalable” systems can be done with a si... by foota on Monday December 30, 2019 @01:03AM (#21911165)
Reddit threads are also much hotter for writing and have O(thousands) of replies.
Re:A lot of complex “scalable” systems can be done with a si... by manigandham on Monday December 30, 2019 @01:14AM (#21911211)
Sure Reddit has more write activity but I don't see the number of replies being such a big factor. SO questions have answers, comments, tags, related questions, and many other secondary data to load. Reddit is slow due to poor architecture and a terrible frontend.
Re:A lot of complex “scalable” systems can be done with a si... by onlyrealcuzzo on Monday December 30, 2019 @01:55AM (#21911356)
Old Reddit isn't slow. It seems to mostly be the new frontend.
Re:A lot of complex “scalable” systems can be done with a si... by shakna on Monday December 30, 2019 @03:13AM (#21911648)
Old Reddit is still slow, but not too bad. Reddit's new software stack is... Atrocious.
Re:A lot of complex “scalable” systems can be done with a si... by markus_zhang on Monday December 30, 2019 @09:31AM (#21913529)
I recalled that back in the days when the new Reddit has been rolled out, there were quite a few days that it's difficult to open in mobile browser. Now it's much better.
Re:A lot of complex “scalable” systems can be done with a si... by jhgg on Monday December 30, 2019 @02:22AM (#21911458)
SO is really not a great example of a high traffic site. 5500 req/sec of mostly read only traffic is not that crazy at all, and their hardware footprint is incredibly over-provisioned for the workload. I don't really think their example stands well here.

For example, at work, our entire analytics ingest workload (HTTP) for a few hundred million users runs on 8 core VMs on GCP, written in Rust/Go, each node doing ~40k events/second.

Our RTC infra sustains >1m PPS per node on 4 core 3.9ghz 2014-2017 xeons.

Re:A lot of complex “scalable” systems can be done with a si... by z0mbie42 on Monday December 30, 2019 @04:33AM (#21912123)
Hi, I'm curious because I plan to rewrite a Rust service to Go (development velocity is too slow) Which part of your service is in Rust amd which in Go? Do you think that if it would be only in Go it could sustain such a load?
Re:A lot of complex “scalable” systems can be done with a si... by zozbot234 on Monday December 30, 2019 @10:28AM (#21913909)
What specific ergonomics pitfalls did you experience in Rust? "Development velocity is too slow" is a bit vague; beyond the use of GC, which only really matters in specialized domains, there's not much reason to think that rewriting your service in Go would give you better 'development velocity'.
Re:A lot of complex “scalable” systems can be done with a si... by kick on Monday December 30, 2019 @02:24PM (#21915933)
There is, though. Go was designed to be easier to reason with than conventional ALGOL-derivatives, while Rust wasn't.

To quote Rob Pike:

The key point here is our programmers are Googlers, they’re not researchers. They’re typically, fairly young, fresh out of school, probably learned Java, maybe learned C or C++, probably learned Python. They’re not capable of understanding a brilliant language but we want to use them to build good software. So, the language that we give them has to be easy for them to understand and easy to adopt.

Go is aiming for the same niche that Python is. This has the accidental side effect of making it faster to write software in it than most of the ALGOL-derivatives. To quote Eric Raymond (sorry) on Python:

When you're writing working code nearly as fast as you can type and your misstep rate is near zero, it generally means you've achieved mastery of the language. But that didn't make sense, because it was still day one and I was regularly pausing to look up new language and library features!

Go and Python both allow for expression about as fast as you can type, by virtue of being designed for [children in Go's case, shell scripting in Python's case].

It's not a value judgement of Rust or anything, but Rust wasn't designed with the same goals in mind.

Re:A lot of complex “scalable” systems can be done with a si... by zozbot234 on Monday December 30, 2019 @02:57PM (#21916142)
> Go was designed to be easier to reason with than conventional ALGOL-derivatives, while Rust wasn't.

This may be a problem of what's idiomatic in each language, as opposed to a matter of language design per se. After all, Rust development can be made at least as easy as, e.g. Swift, simply by adding enough uses of .clone() and RefCell<>. Is this suboptimal? Of course, but it will still be plenty faster than Python, and perhaps even faster than Go.

Compilation time is a separate issue which apparently OP found problematic. It's being dealt with (for non-release optimized builds) via the cranelift project, which is a Rust-specific backend much like the Go compiler, with no reliance on LLVM.

Re:A lot of complex “scalable” systems can be done with a si... by kick on Monday December 30, 2019 @03:38PM (#21916491)
This may be a problem of what's idiomatic in each language, as opposed to a matter of language design per se.

It's absolutely a matter of language design. Again, I'm not criticizing Rust, but a language explicitly designed to be trivially written by anyone, fast, is going to be easier to write in quickly by anyone. Imagine making that comment about Logo instead of Go. Of course Logo is more painless to write than Rust! It's a child's language. So is Go!

Rust development can't be made as easy as Swift (and it is very unlikely that what you described would be faster than Go). Even Graydon Hoare agrees that Rust is inadequate in comparison to Swift in terms of development ease:

https://www.reddit.com/r/rust/comments/7qels2/i_wonder_why_g...

I'm no stranger to languages with vastly different idioms than normal (I write APL daily), but not all languages are as quick to develop in as every other, and pretending they are is silly. Rust has some innovations, and it's by no means a bad language in itself, but pretending it wins at everything under the sun (even things it's not trying to do) doesn't reflect reality or the perspective of the original author.

Re:A lot of complex “scalable” systems can be done with a si... by erik_seaberg on Monday December 30, 2019 @03:26PM (#21916382)
If you can write a ton of code without thinking very much, you're probably writing boilerplate that should have been generated from a human-level description of the problem. Your job is to only spend time writing what needs to be written.
Re:A lot of complex “scalable” systems can be done with a si... by kick on Monday December 30, 2019 @03:50PM (#21916588)
Languages that allow you to write as fast as you think are a blessing.

"Write as fast as you think" is a far better way to program than "Write much slower than you can think."

Eric Raymond has written some pretty substantial things, and he's not as clueless (on programming, at least, the rest of his views are...no) as you're implying.

The idea that intuitive languages are the only ones you should do development in is absurd. A single line of K can do what a hundred lines of C can, and you can write the line of K substantially faster than you could write the C to match. K only has something like 50 primitives. It's simple enough that you can keep it all in your head at once, and that allows you to develop much quicker than almost any ALGOL-derivative. Taking your comment at face value, everything written must be boilerplate. Looking at reality paints a different picture.

Good languages manage complexity in a way that allow you to express complex things in simple terms. That the languages you seem to be familiar with only allow you to describe simple things in simple terms isn't something that's inherent to every programming language. I'd recommend giving APL, J, or K a try.

Re:A lot of complex “scalable” systems can be done with a si... by erik_seaberg on Monday December 30, 2019 @10:36PM (#21918748)
I'm all in favor of concise and expressive languages (even weird ones). I hate being slowed down by the language itself. But writing the first thing that comes into my head leads me to reinventing the wheel a lot, and (at least at work) we have a responsibility to find reusable abstractions and only create new code that's needed.
Re:A lot of complex “scalable” systems can be done with a si... by kick on Monday December 30, 2019 @11:17PM (#21918912)
I disagree with that. If your code base is small enough, a few redundant lines (idioms) don't matter.

To bring up k again, the language has already done about the maximum amount of abstraction possible. There's really no room for the programmer to make reusable abstractions; any useful ones have already been made. That allows you to do very useful things in very small amounts of code. Picking a random example, here's a complete Sudoku solver in 75 bytes:

http://nsl.com/k/sudoku/aw3.k

Re:A lot of complex “scalable” systems can be done with a si... by kick on Monday December 30, 2019 @05:59PM (#21917463)
Typo: "The idea that unintuitive languages..."
Re:A lot of complex “scalable” systems can be done with a si... by z0mbie42 on Monday December 30, 2019 @01:47PM (#21915668)
By development velocity I mean implementing new features, from idea to deployment.

Rust is slow to compile so it breaks my deep work when programming, also it costs me a lot in CI/CD. Also the Rust type system make implementing some things really hard.

For example I wanted to implement json requests logging. It took me more than 1 day in Rust, less than 2 hour in go.

Re:A lot of complex “scalable” systems can be done with a si... by panpanna on Sunday December 29, 2019 @06:27PM (#21909385)
Note that they use C# and ASP.Net...

(SO and DailyWTF are very much into Microsofts ecosystem)

Re:A lot of complex “scalable” systems can be done with a si... by tyuioyjj on Sunday December 29, 2019 @08:26PM (#21909993)
AFAIK they're using ASP.NET Core

There's significant difference between ASP.NET and ASP.NET Core

https://meta.stackexchange.com/questions/316278/the-road-to-...

Re:A lot of complex “scalable” systems can be done with a si... by Merad on Sunday December 29, 2019 @08:52PM (#21910102)
They use Core today, but SO was using .Net long before Core existed.
Re:A lot of complex “scalable” systems can be done with a si... by FpUser on Monday December 30, 2019 @05:31AM (#21912392)
Anything wrong about that?
Re:A lot of complex “scalable” systems can be done with a si... by panpanna on Monday December 30, 2019 @05:40AM (#21912437)
No, I'm just pointing out that they are different from your average SV company.
Re:A lot of complex “scalable” systems can be done with a si... by pjmlp on Monday December 30, 2019 @03:06PM (#21916217)
However they are quite average regarding typical European corporate stacks, either Java or .NET.
Re:A lot of complex “scalable” systems can be done with a si... by fasteo on Monday December 30, 2019 @07:07AM (#21912760)
Any reason for such a low (<5%) average CPU usage? It seems like a waste of resources to me; that's assuming a "normal" CPU usage read that includes wait i/o time.
Re:A lot of complex “scalable” systems can be done with a si... by perl4ever on Sunday December 29, 2019 @08:25PM (#21909987)
At the risk of exposing my ignorance - look at all those "Peak 5%-20%" labels. Doesn't that mean they have a lot more than they need?
Re:A lot of complex “scalable” systems can be done with a si... by dahfizz on Sunday December 29, 2019 @08:38PM (#21910049)
Stack overflow hosts all (most?) Of their own baremetal servers in their own data center.

Looking at the specs of the machines, they are actually pretty basic as far as servers go. A server isn't barely worth the cost of it's chassis and motherboard if you put less than 64 G ram and 24 CPUs in it.

In other words, these are about the lowest specd proper servers you can get. So yeah, even their modest hardware is still overspecd for running their website.

Re:A lot of complex “scalable” systems can be done with a si... by latch on Monday December 30, 2019 @12:05AM (#21910942)
"Their own data center" implies (to me at least) that they built their own data center. That doesn't sound right, so I looked it up, and it seems like they're colocating. That might be what you meant, but the data centers they use certainly weren't built or owned by SO.
Re:A lot of complex “scalable” systems can be done with a si... by dahfizz on Monday December 30, 2019 @12:27AM (#21911026)
As far as I can tell, you are correct. They manage their own metal servers/racks inside a colocated data center. Sorry if this caused confusion.
Re:A lot of complex “scalable” systems can be done with a si... by 0x445442 on Monday December 30, 2019 @10:27AM (#21913902)
Yeah, I recall an article that said their physical hardware was in jeopardy during hurricane Sandy.
Re:A lot of complex “scalable” systems can be done with a si... by paulsutter on Sunday December 29, 2019 @08:44PM (#21910071)
As CPU utilization goes up, latency will go up. Also, one or two second snapshots of CPU use are averages over time. During any given moment you will invariably get little bursts of requests, and those will have slow responses if running closer to full utilization.

Also, of course, the total cost per hour for 9 servers with 3 year depreciation is in cents.

Re:A lot of complex “scalable” systems can be done with a si... by daxfohl on Sunday December 29, 2019 @08:33PM (#21910028)
Re:A lot of complex “scalable” systems can be done with a si... by vitus on Sunday December 29, 2019 @08:36PM (#21910044)
They have a lot more CPU than they need, but the challenge is often getting servers shaped appropriately (RAM x CPU x disk x network), especially if you're not building your own.

Ultimately, their service is (most likely) not compute-bound.

Re:A lot of complex “scalable” systems can be done with a si... by manigandham on Sunday December 29, 2019 @09:03PM (#21910160)
Yes but the overall hardware is a one-time purchase and cheap considering all the other costs of the business. They're well provisioned to keep latency down and handle any unexpected outages.
Re:A lot of complex “scalable” systems can be done with a si... by hubert1234 on Monday December 30, 2019 @07:31AM (#21912891)
600k open websocket connections means that 600k people opened stackoverflow sites but dont click on anything right? Because they still only have 550req/s. Interesting how much more power you need just to keep track of state.
Re:A lot of complex “scalable” systems can be done with a si... by 29athrowaway on Sunday December 29, 2019 @09:15PM (#21910210)
Plus a content delivery network.
Re:A lot of complex “scalable” systems can be done with a si... by decebalus1 on Sunday December 29, 2019 @09:12PM (#21910198)
> SO ranks #36 in Alexa right now

It would rank way higher if they would be more webscale. It needs more Kubernetes, GraphQL and Golang /s