Re:We 30x'd our Node parallelism
by praseodym on Friday December 13, 2019 @11:32AM
(#21783014)
Related, from the article:
> We hypothesized that increasing the Node maximum heap size from the default 1.7GB may help. To solve this problem, we started running Node with the max heap size set to 6GB [..], which was an arbitrary higher value that still fit within our EC2 instances.
Sounds like they were utilizing their EC2 instances very poorly. Why not run more workers per instance, or switch to an instance type with less RAM (or more CPUs)?
Re:We 30x'd our Node parallelism
by tracker1 on Friday December 13, 2019 @04:14PM
(#21785562)
They were using ECS, it also looks like they had to work through a couple bottlenecks to get multiple requests per node working well... I think they could get further by using the newer workers api, since gRPC doesn't work with the cluster module.
> We hypothesized that increasing the Node maximum heap size from the default 1.7GB may help. To solve this problem, we started running Node with the max heap size set to 6GB [..], which was an arbitrary higher value that still fit within our EC2 instances.
Sounds like they were utilizing their EC2 instances very poorly. Why not run more workers per instance, or switch to an instance type with less RAM (or more CPUs)?