In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc.
But these are not parallelism problems. These are single threading problems, which the core problem with Node.js, not parallelism in general. Hence I think the question stands: why did you choose node for this?
Hire an architect costs what? Putting the genie back in the bottle was a problem Plaid baked into its early success, which is common with startups hiring engineers with zero architecture knowledge.
> In terms of what issues caused us to move away from parallelism in the first place, it was all the CPU-bound stuff that you might expect: ReDoS-style issues, post-processing arrays in very large edge cases, programmer error, etc.
But it's trivial (a single line) in Node to place breaks in CPU processing to allow the event loop to fire, and as for "programmer error"... many commenters below are also complaining async programming is too hard or finicky.
But that's like complaining about C because pointers are hard, or Java because OOP is hard, or databases because planning indexes is hard.
Once you "get" async, pointers, OOP, or indexes, it's easy. And it's part of your job as a professional programmer to get it. Async is no trickier than anything else.
The setup in the first place makes absolutely no sense to me, using a language exactly opposite of how it's meant to be.
I agree that it would be nice if all developers were infallible – I'm reminded of a friend describing their company, where "we don't write tests because we all write good code". At a certain point, you have to look for processes – linters, monitoring, testing, language choices [1] – where people can't shoot themselves in the foot. (Code reviews being only moderately less fallible than a single engineer.) It's not enough to just say "be better" whenever bad code is written.
I think when the decision was made (years ago) to handle a single request per container, they couldn't find such a process to prevent event loop blockages, other than migrating an already-large codebase away from Node. As others have pointed out, maybe such a migration is necessary – after all, event loop blockages are still an inherent risk because of how Node works. It's just a lower risk than it was a year or two ago, because we've significantly improved our usage of the event loop, and also have tooling in place to catch blockages before they become an issue.
Yeah, external libraries for Node ought to be designed so that any function that ever might take any length of time whatsoever should always be callable as async. But if they're badly designed or not intended for large inputs, they might not be. You'd definitely need to find another library or write your own there, so I get that.
What I don't get is how nobody treats it as an issue when developers coming from Python or Java to C don't get pointers.
The assumption is that you learn.
But for some reason, people think it's "OK" to not get async, that it's the language's fault rather than the programmer's. That's what I don't understand. It's like a different cultural standard gets applied.
I have no patience for persons who don't belong to the discipline.
For example Discord reached out to Rust and built tiny Rust components that are called from Elixir for their server user list. Some servers have 200,000+ people online, and Elixir wasn't cutting it performance wise. Rust, boom now it works.
No you are not. I wonder which CTO would allow this; like everyone here, the exact case is not really clear (or at least why this solution is a great solution for it), but this sounds like a weird solution (and expensive) to some issue. I really don't understand these 'solutions' and I am almost 100% sure I (with a team! but the point that this is not the best solution for the problem) can whip up something far simpler and more efficient for this problem. But ofcourse there are problems that might fit?
> We hypothesized that increasing the Node maximum heap size from the default 1.7GB may help. To solve this problem, we started running Node with the max heap size set to 6GB [..], which was an arbitrary higher value that still fit within our EC2 instances.
Sounds like they were utilizing their EC2 instances very poorly. Why not run more workers per instance, or switch to an instance type with less RAM (or more CPUs)?
But also, though, you have to consider that most places aren't Plaid, and most places developer time is more expensive than throwing an extra machine at the problem.
Still interesting...
We still have an event loop that is trivially blocked by very simple programmer errors, destroying the whole advantage that you describe here.
The fact that Node ships a fully asynchronous standard library doesn't in any way fix the fact that Node is a runtime for a language that itself is a mistake.
So they fixed the issue that some requests blocked... by making all requests blocking.
There is a massive deadlocking design mistake in the centre of the language - literally a huge red button with DO NOT PRESS printed on it. Thousands of programmers pass it by every single day, or hour, or minute, and the creators of the runtime insist that it is impossible to fix that button whatsoever; instead, all users need to work around it by ensuring that their code in no way presses that red button on purpose or even by accident.
These people insist that it is impossible to program normally and in a language that is actually sane and does not advertise obvious and gaping design mistakes as "features of the language". These people advertise the analogue of Python's Global Interpreter Lock as the core foundation of their language.
These people advertise Node and the language it implements as practical for implementing multithreaded applications. Posts such as this show what sort of bullshit it is; it is only practical to use Node for parallelism if each single Node instance is only ever run single-threaded. You don't parallelize by running multiple threads, you parallelize by running multiple Node runtimes.
This is no longer an act against productivity or usability. This is simply insane and shows one of the most basic things that are wrong about Node's language and approach. It is impossible to write a multithreaded program if your language of choice makes it trivial, and practically unavoidable, to globally lock your whole runtime with every single line of code you write and import as your dependencies.
No different than having a dedicated threadpool for asynchronous programming on the JVM.
Yes blocking the event loop is easy. No it's not THAT easy. I've never done it because you think about it while writing code. It's part of the environment. I have had to fix lots of reports etc that try to load up the world and iterate through it in a loop that doesn't yield to the event loop. It's possible to never make that mistake by understanding your environment (just like how managing pointers in C is "hard").
To be pedantic, all JavaScript functions block the event loop. It's just that the vast majority of functions execute so quickly that the amount of time your function blocks is very short.
I once had a loop that processed tons of data and would block the event loop for 1-3 seconds. I ended up solving it by writing an asynchronous loop which used promises and process.nextTick() to make each iteration a separate execution block on the event loop. But I've only had to do something weird like that once in 10 years of node development.
At one company I built API middleware that calculated time spent in the event loop vs waiting on dependencies (other apis, Redis, etc) by managing a couple counters per request. It was very helpful.
However, running multiple containers for parallelism sounds a little bit crazy. In the worst case, each container may be running on its own server, but even assuming multiple containers per host, I'm guessing they were running an insignificant number of instances, which is probably why they were able to save $300k in server costs.
> We were running 4,000 Node containers (or "workers") for our bank integration service.
All I'm saying is that the choice of language is (usually) not the issue. Poor architecture design causes a lot more problems than whether you choose python/java/ruby/node for your webapp.
As someone not well-versed in js, could you describe one such case? Concurrent access to a global from two threads? Mutexes? My background is more with systems languages and I have done very little js for the browser, so I do not see that big red button.
If your function blocks, for example, by performing a wait on something without yielding, then nothing else gets computed because of the wait, but the event queue is occupied since the function has not yielded. This breaks the cooperative part of the cooperative multithreading mechanism of Node.
If the event loop is part of the Node language/runtime, I can see the case for making it preemptive.
All this has happened before. All this will happen again.
In Javascript world I guess you could do the same by replacing time.sleep with some CPU bound code. eg a big calculation or an infinite for loop.
I'm working on a project now that's very CPU bound and using limited workers behind an MQ as a distributed RPC behind a fronting API interface so that I can handle scaling... though it's also a Windows-only library involved. There's definitely an art to scaling certain types of workloads and many different options. Sometimes the simplest solution you can come up with really is the best option.
The first Node.js service I wrote and maintained, processed thousands of requests in parallel and was successfully in production until the company it was developed for ran out of money.
Some features would become unusable, like Promises and async/await, but those would be worthless in such a design anyway.
As much as I'd like to see it, your JS without an event loop is a purely theoretical construct so far.
I wouldn't implement it for the HTTP server/backend use case, because I find the blocking/share nothing architecture of PHP limiting.
The point is the language doesn't prevent this kind of architecture.
I can't help but also feel that is also an issue and in another given language this issue might not happen ... but they'd hit another.
It's so easy to say "don't do X because problem Y won't happen" but hard to predict what happens when you move from (language, platform, or whatever) X to (language, platform, or whatever) Z.... and I suspect people often hit issues and realize that maybe Y wasn't the problem.
I see it all the time and I feel like "Wait guies I'm not sure we're fixing the right thing!?!?!"
This article raises a lot more questions than answers IMO.
I'd bet that the orders of magnitude of speed from Moore's law did in CMT by making PMT doable without a huge speed hit.
Nothing I've seen from async is cleaner, easier to maintain, or better from a cognitive load POV. It's just more efficient for certain types of loads because you're being a consenting adult and not breaking things.
Node's concurrency and parallelism probs don't require deep lang changes but runtime ones and some ergonomics to be closer to Go -- and not far IMO bc of last few years of work for Async, safe (fresh env) eval, and json/buffer msg's.
More exciting would be something like Apache Arrow / Berkeley's 'Plasma', but that stuff is still more exploratory.
https://dev.to/nestedsoftware/is-cooperative-concurrency-her...
This article does a good overview
Creating a system in the 21st century that tries to follow ideals from the 90s gives us the kind of idiotism that we can witness here.
You are otherwise on the right track, though Node does technically have one advantage, which is that it is a cooperatively-scheduled island in a preemptively-scheluded overall OS. In the 1990s, when the cooperatively-scheduled program was not cooperative, you locked the machine, not the process [1]. There is a reason why Apple went very aggressive with the OSX rewrite; the previous Systems had basically written themselves into a corner where they had to use cooperative multitasking because so much code made use of the implicit promises it provides, yet they could no longer afford to compete with Microsoft if they didn't get off it it, because the complexity just kept going up, up, up and the problem was going to continue getting exponentially worse.
For a Node program, you only have to account for the Node program itself, not everything running on the computer. Still, you're in the same exponentially-growing-complexity trap (with a very initially-safe-seeming low exponent, but it still gets you in the end), you just reset yourself back to a point earlier on the curve.
[1] There are various details, caveats, interrupts, etc, the picture is more complicated than one sentence can convey, but the principle still held and it was still possible to wedge the machine fairly badly for varying periods of time with simple bad code.
It doesn't matter if it's in an event loop or thread per request. Architect things correctly.
Can you give an example please?
I think it's much easier to block a thread with C#'s async programming model than node's...
Node can't not be single-threaded. Because Javascript is. Node.js is single threaded. It has a single event loop in a single thread, and all the "concurrency" is simply queued on that loop. It offloads some tasks to libuv for some system-related tasks but that's it. And the thread pool that libuv creates is very limited.
Anything that doesn't end up in libuv (that is, probably vast majority of user code) will only ever run in one thread... because Javascript is single-threaded hence V8 is single-threaded hence Node.js you get the gist.
And of course, node.js even has a separate documentation section titled "Don't Block the Event Loop (or the Worker Pool)" [1] because it's trivial to block the event loop.
[1] https://nodejs.org/en/docs/guides/dont-block-the-event-loop/
https://stackoverflow.com/a/24684037/1026671
"Concurrency" doesn't warrant scare quotes even when describing a single threaded program.
It's true that Node.js allows one task to block many others, but that's an implementation detail of Node.js, not a guaranteed result which necessarily applies to any program using only one OS thread. Other programs suffer from analogous starvation/deadlock/livelock/priority inversion problems, but those would be implementation details too, not guaranteed results from using multiple OS threads.
AFAIK there’s nothing in the JS language that forces a single-threaded implementation. I think the real reasons that Node is single threaded are a) legacy and more importantly b) a lot of existing code would break if events were dispatched in parallel.
Yes, I can. Node.js doesn't do that.
The only thing I can think of where a programmer could "easily" block the event loop would be if they explicitly use sync filesystem or other blocking calls instead of the async API. But in any project with reasonable code review I don't think this would happen.
Parsing JSON. Or executing a regex. Or anything, really, that blocks the thread: https://nodejs.org/en/docs/guides/dont-block-the-event-loop/
There's no magic
In practice I've rarely seen production nodejs applications cause significant CPU blocking issues. Huge JSON parses sometimes, yes, but then it has to be one hell of a payload to cause any significant issue.
Regexes? Have you really seen regexes block CPU for significant time relative to the rest of the application? I'm sure it's possible with a crazy enough runaway pattern, but I've never seen it happen.
I really don't understand what people are getting at here. Nodejs is an async programming model, it's non-blocking by default. Are the people saying it's trivial to break nodejs developers?
> I think it's much easier to block a thread with C#'s async programming model than node's...
Which spawned a discussion in which people seemingly think that you can't block a thread in node, or that since node is async it means it's not single-threaded etc.
Since it is single-threaded, it's quite possible that the original post wouldn't need 4000 node instances otherwise (emphasis mine):
> We were running 4,000 Node containers (or "workers") for our bank integration service. The service was originally designed such that each worker would process only a single request at a time. This design lessened the impact of integrations that accidentally blocked the event loop
Here's how it probably worked: they liked Node, they liked containers, they put Node into containers and it worked, and they stuck with it as the user base grew.
I can't be the only person who reads stories like this and wonders how they arrived at that solution in the first place?
Failing to scale because their previous approach to scaling was a worker per request, a model which was roundly moved away from, because that's how CGI and Apache modules worked and it didn't scale well.
I thought one of the key selling points with Node was an fully async standard library, enabling better scaling in process.
But then you read stories like this, and I find it hard to relate to the original problem.