News for hackers. Stuff that matters.
Re:We 30x'd our Node parallelism by kevstev on Friday December 13, 2019 @11:17AM (#21782877)
I was building scalable node applications a few years ago for a very large e-commerce player- millions of customers. I think node.js is a great platform, but its apparent simplicity means there are hordes, and I mean like 90+% of the community, that can "just get things done" without understanding what is going on under the hood at all. And to be fair, for most startupy types of companies that need to iterate fast, that is what you want to optimize for.

My interview screening question was pretty simple- "Is node.js single threaded or multithreaded?" And to most, they spit back the blogspam headline- "Single threaded!" I think the most correct answer is "its complicated" but would accept that because most people would say that is the "right" answer. So I would follow up with- "what exactly happens in a default installation if we have say... 5 requests come in at exactly the same time to just return some static content from disk?" (Node's default threadpool is 4). And here is where you could see their understanding just fell apart. Some would say they would be handled entirely synchronously, others completely in parallel- but then had no idea what the cause of the parallelism was. Very few actually understood that node is an event loop executing javascript backed by a threadpool for async operations.

Before reading this post, I was like eh this is a waste of time- its typical medium bullshit- they almost certainly found they were doing some blocking call in the event loop and then removed it and voila, 30x speedup. It was interesting because it was a lot worse! They spent all this time and hard work figuring out everything but what was taking so long in the event loop, and it seems that was the last place they actually looked.

Anyway, node can be a highly scalable platform (https://changelog.com/podcast/116) but you need to understand it or else it will bite you in the foot. When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that.

Re:We 30x'd our Node parallelism by NathanKP on Friday December 13, 2019 @12:09PM (#21783337)
This is why I recommend that anyone running Node in production use a tracing tool like New Relic. It's super easy to see what is blocking the event loop. Just choose a duration (say 10ms) and look for any execution spans that are longer than that duration.

Ideally you want to be yielding back to the event loop at least every 1 ms. Anything that takes too long without yielding will show up as a latency delay before your code is able to start handling a new request (technically a background thread in Node.js will pick up the request, but your code won't start executing in response to it until you yield back to the event loop again).

To be honest the more difficult thing to diagnose sometimes is event loop overburdening. If each of your execution spans are taking 1ms, then you can only do a max of 1000 of them per second (assuming there was no delay between executions, but there is). So if you are trying to handle a large number of requests per second the event loop may end up with say 1005 execution spans per second that it needs to execute to handle that request volume. Because you can't do 1005ms of work in 1000ms the extra work will queue up.

So gradually you will end up with 5 backlogged execution spans stacking up per second. Each second you will get 5ms more latency. The overall request latency will just gradually increase and increase as work gets further and further delayed in the queue.

Overall I just think of Node.js as a fancy CPU scheduler. As long as you give it even, decently sized chunks of work to schedule, and you don't give it too many to schedule you will be fine. Anyway I'm a huge fan of Node.js but yeah its easy to fall into some gotcha's if you don't study how it works. The simplicity is a bit misleading

Re:We 30x'd our Node parallelism by inglor on Friday December 13, 2019 @12:18PM (#21783429)
Elastic APM is better than New Relic in how it traces node and it is completely free and open source (you can use a cloud product).

Disclosure and bias: I work on Node core and always hear ranting about incorrect usage in async_hooks in anyone but Elastic APM in core meetings. I used both products and have no affiliation to other companies.

Re:We 30x'd our Node parallelism by NathanKP on Friday December 13, 2019 @12:26PM (#21783489)
Interesting! I've had too many bad experiences over the years with Elasticsearch and ELK stack so I usually avoid Elasticsearch based products like the plague. Maybe if someone else runs it for me though...

Do you know what they are doing differently with async_hooks in Elastic APM?

Re:We 30x'd our Node parallelism by neebz on Saturday December 14, 2019 @08:15AM (#21789355)
How exactly do you use NewRelic to see what is blocking the event loop? I thought we always needed Flame Graphs for it (which NR doesn't provide)
Re:We 30x'd our Node parallelism by NathanKP on Saturday December 14, 2019 @09:53PM (#21794151)
Flame graph is only needed if you are having a lot of trouble pinning down the exact call that is taking a long time. In most cases I've found traces to be all I needed. For example if I see that this span of execution took longer than expected and I read through that code and see a bunch of JSON serialization its pretty obvious.

To be honest that comes with a lot of Node.js experience though. You probably need flame graphs to start out with, but eventually I find New Relic traces to be all I need, as I already have a sense for the relative CPU weight of the various calls inside a trace span.

Re:We 30x'd our Node parallelism by inglor on Friday December 13, 2019 @12:16PM (#21783412)
> Very few actually understood that node is an event loop executing javascript backed by a threadpool for async operations.

This is true, and that JavaScript is mostly a synchronous programming language with host environments that can provide asynchronisity.

A caveat though is that the most important part of I/O is network I/O (tcp/udp sockets) and Node uses real async operations there rather than a threadpool.

FS is just really hard to get right in a cross platform way and that's why it's on the threadpool. Some other stuff like dns is also famously on the threadpool but tcp sockets are not - it's a big part of why Node is fast.

Re:We 30x'd our Node parallelism by specialist on Friday December 13, 2019 @06:27PM (#21786582)
At my last gig, I maintained a handful of existing, and created some new nodejs things. I had previously done a lot of Java and even hacked on Apache. I had no prior nodejs experience.

The first thing that really bothered me about our use of nodejs was no one could say why stuff would fail in production. So many moving parts. One of my team members figured out some edge case interactions between nodejs and nginx (used for HTTPS), which I would have never figured out on my own. It wouldn't have even occurred to me to look there. But other crashes, caused by apparent leaks, were mystifying.

The second, and bigger, thing that really bothered me about nodejs, and expressjs in particular, was the notion of back pressure is completely missing. If it's in there, I couldn't find it. So our endpoints were still accepting new socket connections without processing responses from backend services (eg redis, other nodejs endpoints, auth services), which would either zombie or ABEND those backends. And no one could figure out why.

I only understood what was happening because I'd already been through all that "architecture" madness a decade earlier with Java services.

I guess what I'm saying is while I LOVE nodejs' closeness to the metal, I didn't like going back in time 10-15 years.

Also, npm is crap.

Re:We 30x'd our Node parallelism by pier25 on Friday December 13, 2019 @11:55AM (#21783217)
Here is a SO answer that expands a bit more on kevstev excellent comment:

https://stackoverflow.com/a/22644735/816478

Re:We 30x'd our Node parallelism by golergka on Friday December 13, 2019 @02:38PM (#21784634)
> When I was last doing this stuff, upwards of 80% of our time was being spent essentially just JSON.parse()'ing, and we were looking to move to protobufs to avoid that.

It's only tangentially related to your question, but I can't help but ask this question: why people use JSON instead of protobufs at all?

I'm mostly a client-side developer, and most of my server-side experience is in hobby projects; still, I always used protobufs and loved it. They never damaged my feature velocity, apart from an hour to set up the build system in the beginning, and type safety helped me quite a few times when I forgot to sync changes in protocol on client and server side. Are there some secret advantages of going with json that I don't see because of limited experience?

Re:We 30x'd our Node parallelism by kevstev on Friday December 13, 2019 @02:55PM (#21784791)
JSON existed before protobufs is really it. When I left the node world 3 years ago, protobufs were the new hot thing. Any new project should start with them over JSON imho.

There is some friction to them though, and I think a lot of it is that most tutorials and beginner books like to keep things as simple as possible, and people start their little project, it gets traction, and then they figure out they need protobufs but now its hard to introduce. In most projects, even today, it seems that its the version 2.0 that gets protobufs, v1.0 keeps JSON for simplicity, unless you have a bunch of seasoned devs involved.

Re:We 30x'd our Node parallelism by x86_64Ubuntu on Friday December 13, 2019 @11:56AM (#21783223)
What does happen with the 5th request.
Re:We 30x'd our Node parallelism by NathanKP on Friday December 13, 2019 @12:14PM (#21783395)
It waits in a queue in a background thread in the node.js http library until your code can execute to handle it. So if your code takes a long time before returning back to the event loop the request will just wait in that queue for a long time before the next opportunity for the event loop to execute code in response to the event.
Re:We 30x'd our Node parallelism by kevstev on Friday December 13, 2019 @12:12PM (#21783365)
It will get queued, until one of the 4 requests in front of it has its task, to return the file, complete.
Re:We 30x'd our Node parallelism by jwalton on Friday December 13, 2019 @08:31PM (#21787188)
Well... yes. If you're using `express.static()`, or if all five requests are getting different things from the disk. If you're using something that caches your static content in-memory, then the first request will use the thread pool to read that content, but the other four requests won't touch the thread pool - it's all async IO at that point, so it's all happening "concurrently" in a single thread, being multiplexed right here: https://github.com/libuv/libuv/blob/1ce6393a5780538ad8601cae...
Re:We 30x'd our Node parallelism by rclayton on Friday December 13, 2019 @08:46PM (#21787241)
Is it really only 4? When I look at my VM stats I seem to recall it having something like 15. Of course, this could be a config change on the Node Alpine container.
Re:We 30x'd our Node parallelism by Kiro on Friday December 13, 2019 @01:52PM (#21784255)
I have a Node service where I get tens of thousands requests a second and I still thought Node was single threaded. Where can I read about this?
Re:We 30x'd our Node parallelism by rclayton on Friday December 13, 2019 @08:49PM (#21787256)
The event loop is single threaded. Async tasks are executed on I/O threads which are configurable. So if your app is I/O bound, the event loop will typically dequeue tasks pretty quickly allowing lots of requests.
Re:We 30x'd our Node parallelism by avip on Friday December 13, 2019 @03:37PM (#21785195)
That's a great interview question (especially if you're not so much into hiring :))

Another one is: what happens when a node process completed execution?

  // node ex.js
  function foo() {  // something async here }
  foo()
  console.log('bye...')

This is a fun question to discuss (I think some consider this a bug in node).
Re:We 30x'd our Node parallelism by z3t4 on Friday December 13, 2019 @07:51PM (#21787027)
Node.js is a good abstraction layer. In my experience, everything get leaky once you get hundreds of concurrent users.
Re:We 30x'd our Node parallelism by gameswithgo on Friday December 13, 2019 @12:48PM (#21783675)
>I think node.js is a great platform,

I'm curious as to why. For large scale applications like this, you have other options that offer higher performance ceilings, have more safety and correctness features, and are likely more productive as well. What is the attraction to node?

A guy has to invent a scripting language for browsers in 9 days -> he decides on a lisp -> management says no it has to look like java -> he comes up with something -> its dynamically typed -> lets run a huge banking infrastructure on this

wat

Re:We 30x'd our Node parallelism by NohatCoder on Friday December 13, 2019 @03:12PM (#21784953)
The real killer feature is async. Since a modern web request typically spend most of the time waiting for database calls, file system requests or similar, a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance.

Being used to Node, I was flabbergasted when writing C for Linux* . The file system commands just leave my thread hanging while the result is being generated, if I use it on a network drive it might hang for a minute before timing out, so I have to make a tread for each file system command, solely so that it can stall without bringing down the whole application.

* I have no delusions that Windows is any better, Linux is just what I have first hand experience with.

Re:We 30x'd our Node parallelism by unlinked_dll on Friday December 13, 2019 @03:46PM (#21785284)
>a naively coded server in most languages can handle relatively few requests per thread, so you scale up the number of threads to something like 100 per core, and now the overhead of running and switching between these treads is limiting the performance.

This would be true if you hired someone to write a server in C about 15 years ago. It's not true today. And I hope you're not putting a naively coded server like that in production, or at least doing the hour of research once you notice it's awfully slow to solve the problem.

Like if you wrote your backend in Go, Rust, Java or any number of languages (even C/C++ with common dependencies!) and did a little reading while you designed it, this issue wouldn't exist.

Re:We 30x'd our Node parallelism by cwp on Friday December 13, 2019 @03:36PM (#21785192)
It happens that Windows is better. It has much better kernel support for async IO.
Re:We 30x'd our Node parallelism by kevstev on Friday December 13, 2019 @01:54PM (#21784272)
My background was doing low level C++ in HFT/Algorithmic trading for years, with a bit of Java interspersed, before doing this whole complete right turn of doing webdev in js for ecommerce. In C++, doing web stuff was very difficult, build times were long, JSON support existed but was awkward, iterating was just very difficult, even when you had your whole build/deploy setup going, there was still a lot of work if you wanted to build a CRUD app just marshalling and unmarshalling objects to the DB, etc. It was a drag at best and a real pain at worst. Java... was a bit better, and I don't really have a problem with Java as a language. Java programmers, however, seem to really delight in building architectural monuments and get paid by the abstraction. In every java system I have jumped into, you are always neck deep in xml, and massive object hierarchies and factory factories and its just like where the F is the code that actually does stuff?! I remember working on one project, where I was building the "engine" and another guy was building the web interface for it, and was using Spring when it was relatively new, and he happily declared "all I need to do is wire up my configs now, I am pretty much done." Narrator: A week later, he still wasn't done... and this wasn't unusual in my experience, in most java apps "config" was just as complex and problematic as code, and the mindset of "its just config" made config changes more likely to cause a production outage of some sort.

Then, you take a look at node. You look at a getting started tutorial. Its javascript on the front, and on the back. The JSON in between is "native" and is convenient and easy to use, easy to read, lightweight, and just makes a lot of intuitive sense- especially when I had found myself neck deep in XML in previous jobs for the same tasks. I had a nice looking HTML5 web app running in a few minutes- my mind was blown. Then you take a look at the frameworks- express and hapi, and the vast module ecosystem- and how easy it was to build a simple CRUD website with leveldb, or mysql, or really an endless array of storage options. And people were using those options! It wasn't just the bog standard RDBMS being used every place, with your only real choice being mysql, postgres, or if you had money, Oracle. Building endpoints with routes in these frameworks made your code so easy to divide up along clear lines, and there just wasn't the endless miles of boilerplate/scaffold code, and ugly syntax and type systems to fight with and plan ahead of. Things Just Worked. Turning around a code change was a matter of seconds, not a minutes long build process- I had never felt so productive- and writing code was fun again! Deploys were easy, restarts were fast. Rollbacks, when necessary, were painless. There was a plugin/module for everything (too much in hindsight).

Now, this was 6 years ago. Go was around, but still kind of a blip on the radar, Ruby/Python were probably the closest real contenders. Ruby had lost steam, I honestly took some cursory looks at it, but it didn't seem to have traction. Python, suffered from its single threadedness and GIL, and its popularity with the ML crowd- Flask and such existed, but was pretty rudimentary compared to what Express/Hapi were offering, and no one seemed that interested in those projects. I like Go a lot, and for a pure backend service, it might be my go-to today, as one of the original arguments for Node was "its the same language on the front and the back, no more delineation between FE and BE developers, anyone can jump in and fix the bugs!" Which, along the lines of my original comment, don't really work out in reality, at least not on larger systems. People drawn to FE work usually have never done real systems development and don't understand how things work under the hood- which isn't a problem, until one day it is and then its a huge one.

The dynamic typing argument... is somewhat valid, but I found that enforcing api contracts with hapi/joi gave you the equivalence of type safety at your interface borders, while still giving you the flexibility of dynamic typing within your code. In fact, Joi went even farther than just type checking, it could check that your int was within range for the field, that your dates were formatted properly, etc... In mega large codebases, this will come back to bite you, but I found the plugin architecture of Hapi really discouraged that kind of crap from leaking in and it was easy to build truly modularized code.

The performance ceilings aren't that different, and not that impactful, at least not until you get to FANG scale, and I mean literally only FANG scale. We were running a billion dollar business with on 8 fairly small VMs for the API layer, which handled all of the ecommerce transaction handling. I remember at one point we encountered a memory leak of some sort in node, and the instances were falling over and dying about once an hour, but restarting and recovering- this was causing a few % error rates to our customers. I was insistent that we get all hands on deck to figure this out ASAP, and our head of Ops type person said "kevstev, we can throw hardware at this problem to meet SLOs until you get it under control. Your monthly server costs are less than my studio apartment cost me per month in Jersey City 15 years ago."

You just have to have a basic understanding of whats going on at an architectural level, something a few hours of doing the right reading and experimenting can get you if you have the proper background. The number of gotchas to avoid to get that performance were an order of magnitude, if not more, fewer than in a language like C++ (Which I feel has actually gotten so complicated and difficult to grok its become a parody of itself- and I say that as someone who used it and adored it for 15 years).

Re:We 30x'd our Node parallelism by tracker1 on Friday December 13, 2019 @03:57PM (#21785381)
Thank you so much for putting into words thoughts that resemble my own. I absolutely love how productive I am working with Node in JS vs anything else. I like C# and have enjoyed learning Rust, but nothing really compares.

Yes, there are a number of footguns, but that's true of any language and platform. You can do stupid things in any number of platforms and languages. I don't even see anything particularly egregious in TFA for that matter.

There are fortune 100 companies with systems handling hundreds of thousands of requests per second on a couple dozen servers in Node.js ... is it absolute performance for CPU intensive operations, not really, does it handle more simultaneous requests than everything else, not even close. What it does offer is a really good mix of good enough performance with unmatched developer productivity.

Re:We 30x'd our Node parallelism by DanHulton on Friday December 13, 2019 @03:41PM (#21785235)
You're handily skipping over 15 years of improvement and iteration between the last and second-last points there.
Re:We 30x'd our Node parallelism by a13n on Friday December 13, 2019 @01:18PM (#21783955)
> safety and correctness features

You can achieve safety and correctness features for node via good lint rules and typescript/flow.

Re:We 30x'd our Node parallelism by Nextgrid on Friday December 13, 2019 @01:55PM (#21784283)
You can technically achieve all of that even in an obscure language like Brainfuck but it doesn’t mean it’s a good idea.

Why would you duct tape hacks on top of hacks to achieve the result you want instead of using a language that has already all of the functionality built-in?

Re:We 30x'd our Node parallelism by bfrydl on Friday December 13, 2019 @02:44PM (#21784690)
I find it really annoying how JavaScript is treated on developer forums like this one. Why is it that when the same exact things are done in a typical language it's called “tooling” but in JavaScript it's “duct tape hacks on top of hacks”?

Don't get me wrong, my favorite language is Rust, but pretending the JavaScript ecosystem is unusable doesn't make you cool. I can be extremely productive in TypeScript.

Re:We 30x'd our Node parallelism by Nextgrid on Friday December 13, 2019 @03:20PM (#21785044)
JavaScript itself is not a hack, it works fine for its original purpose - making webpages interactive.

What I consider hacks is the hundreds of different tools and dialects of JS that are used to bend JS into doing something it wasn’t really designed for.

JS is fine in the browser, and TypeScript is also fine there because you don’t really have the choice to run anything different (although Web Assembly might change the game soon). But on the backend you have the privilege to pick between dozens of different languages that are better suited to the task and support the features you want out of the box without layering hacks on top. Why not just go with one of those?

Re:We 30x'd our Node parallelism by kevstev on Friday December 13, 2019 @02:43PM (#21784681)
I can appreciate your suspicion of a dynamically typed language, but type script isn't really a "hack" its a superset of the language. I am not a huge fan of it, to me its kind of the worst of both worlds- if you need type safety, use something else.

However, if you haven't taken a look at Joi- https://hapi.dev/family/joi/?v=16.1.8 you probably should. To me this was a very happy medium- you can enforce your "types" at the point of ingress and egress at your api, and even do validations there, while still being able to enjoy the flexibility of type safety within your code.

Re:We 30x'd our Node parallelism by rclayton on Friday December 13, 2019 @08:42PM (#21787222)
Lol - I don’t think I’ve ever met someone as enamored with Joi as I am. I think I abuse it in all the same ways as you do. I even publish my Joi schemas for request validation with my API specifications.
Re:We 30x'd our Node parallelism by a13n on Friday December 13, 2019 @04:49PM (#21785879)
We chose JavaScript/Node because it's the language we knew best, and because we have to write it in the client as we're building a web-based service. No other language offers us that, unless it compiles to JavaScript (eg TypeScript).

This grants us a magnitude of benefits, like being able to do server-side rendering and share code between the client and server. It also means we only need to hire people who can write JS, instead of JS and something else.

TypeScript, Flow, and ESLint aren't hacks. These are mature tools used by some of the largest, most sophisticated engineering teams in the world.

Re:We 30x'd our Node parallelism by JMTQp8lwXL on Friday December 13, 2019 @02:19PM (#21784475)
Most bugs encountered in production systems aren't type based issues. Types are more useful for developer productivity (e.g., intellisense) than any other purpose.
Re:We 30x'd our Node parallelism by RSZC on Friday December 13, 2019 @03:03PM (#21784872)
Previously worked at a Node company - at one point in an effort to improve code quality we ran statistics on errors we'd seen over the past [period - forget exactly]. Type errors were our most common source of error both by number of total errors and number of distinct errors.

I'd love to see any data or case studies that claim the opposite if you have any.

Re:We 30x'd our Node parallelism by JMTQp8lwXL on Friday December 13, 2019 @03:24PM (#21785069)
https://blog.acolyer.org/2017/09/19/to-type-or-not-to-type-q...

> Is a 15% reduction in bugs making it all the way through your development pipeline worth it to you?...

https://www.reddit.com/r/typescript/comments/aofcik/38_of_bu...

> 38% of bugs at Airbnb could have been prevented by TypeScript according to postmortem analysis

I've never seen a number far outside of the 15-30% range.

In my experience, most bugs are operator error. Developers didn't code for branching paths that should've been accounted for, etc.

Personally, I'm a fan of TypeScript. Just don't expect to remove the majority of your bugs via its usage. The old "no silver bullet" adage.

Re:We 30x'd our Node parallelism by bathtub365 on Friday December 13, 2019 @08:44PM (#21787233)
How is JavaScript worth it if it typically adds 15-30% more bugs? That’s an enormous figure.
Re:We 30x'd our Node parallelism by JMTQp8lwXL on Friday December 13, 2019 @09:18PM (#21787354)
Well, it runs everywhere, for one. That's an attractive quality to a language.
Re:We 30x'd our Node parallelism by z3t4 on Friday December 13, 2019 @06:39PM (#21786665)
One could argue that all errors are type errors. There is not just a type system, static or dynamic, inferred or annotated, which is good enough to catch all bugs. :P
Re:We 30x'd our Node parallelism by kevstev on Friday December 13, 2019 @03:23PM (#21785060)
Just curious- did you use Joi or anything similar to try to at least verify at some point that you had a valid object?
Re:We 30x'd our Node parallelism by JMTQp8lwXL on Friday December 13, 2019 @07:27PM (#21786932)
A follow up, but I also wanted to highlight: plenty of typed systems (like Java Microservices) have bugs. If 100% of bugs were due to typing issues, these systems would -never- have bugs. Ever. Yet, we know that's not the case. That's another rationale for how the 1/4 to 1/3 less bugs ratio makes logical sense.