Re:My Node.js is a bit Rusty
by ryanjshaw on Friday September 08, 2023 @07:55AM
(#37432372)
The low throughput is definitely a code smell.
In my experience, a common mistake here is to read the data faster than you process it and run out of memory. For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed, and you'll run out of memory.
Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.
In my experience, a common mistake here is to read the data faster than you process it and run out of memory. For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed, and you'll run out of memory.
Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.