The point the parent is making is that the boring normal code in NodeJS is not slow, at least not as presented in the article. The bottleneck is probably somewhere else. Allocation is usually not slow in a garbage collected language. Deallocation can be, but that is usually easy to check (for example with the --trace-gc flag, or running with profiling enabled and checking the results with chrome devtools). It would have been nice if they shared such analysis.

With the limited information provided in the article, there are other points that I would check before going as far as writing a module in another language. For example

> records.push({

They are collecting all the parsed rows in a single array. This is going to consume memory proportional to the size of the file, potentially in the same order of magnitude (depends on how much information is being collected and how much discarded).

> But is the code shitty? It looks like boring normal code to me. I'm not sure what point you're making here.

1. It should never be that stupidly slow

2. That should be obvious to anyone with a napkin and some math

3. Excessive allocations are highly unlikely to be the root cause of the slowness

4. Instead of debugging, they seemingly spent a lot of time putting it on Kubernetes, scaling it out to 25 instances and then finally rewriting a portion of it in Rust.

The whole thing smells of confusion and multiple issues being conflated together, with the rewrite in rust fixing these by virtue of deletion.

And if the root is was the GC, then some `--trace-gc` output in the post would be nice at the absolute minimum. Showing a for loop with a string split doesn't cut it.

Exactly. I'm not a super-expert in Node.js but worked a lot with it and also know a lot of internals.

And I can definitely say: IO speed is pretty good. To be honest, I've never come across a language with bad IO speed and I've seen quite a lot of languages.

I also did some benchmarks (like orf did) and speed is pretty good. Maybe Rust is a little bit better because it's nearer to syscalls than Node.js is but we're not talking about 760kb/s vs. 250mb/s.

200GB of logs and 25 instances means 8GB per instance. Not sure what they're doing, but processing of 8GB taking 3h is like doing it with my 80286 processor ;-) Even with inserting stuff in MongoDB this seems way to slow. And they only replaced the file IO part with Rust...

I guess the records array is getting pretty huge which might result in swapping to disk. But then: don't let it happen, just flush it to the database or whatever. I believe the problem could have been solved with Node.js without issues.

It's probably all the memory allocations. I know that when I do work like this, creating a huge array up front and assigning to that typically results in massive speedups.

I know basically nothing about Node though, so I could be completely wrong here.

Yeah that's one way to do it. Another is to reuse structures where possible to not force reallocation. There's a bunch of ways.

Node.js has a decent JIT, or so I've heard. I have experience with the HotSpot JVM. For the JVM this code would absolutely stop allocating on the heap past 10k iterations of the loop. "Split a string then grab the parts in a hot loop" is ground stakes for JIT compilers.

If they are seeing any GC activity once the loop gets hot, something else is going very wrong.

Yeah but if you're appending in a loop you'll need to allocate memory every time, right? Can a JIT avoid or rewrite this?

Yep the Hotspot JIT will absolutely allocate on the stack and reuse memory allocations inside loops. It also has escape analysis and it will detect you don't need the array of strings after the iteration. In that case it usually just allocates the strings directly at their final locations on the record object.

Wow, that's super cool. Almost makes me wish i did stuff on the JVM (but until someone rewrites LAPACK for it, that seems unlikely).

The low throughput is definitely a code smell.

In my experience, a common mistake here is to read the data faster than you process it and run out of memory. For example, maybe they have a loop reading lines and doing a bunch of async DB writes: your process will read at disk speed and write at database speed, and you'll run out of memory.

Perhaps this is why it worked to split the file into 8GB units X 25 servers: it didn't blow their memory budget per server.