Yes, of course the unbounded array will cause you to run out of memory. That’s… not news, and it absolutely isn’t a “memory allocation issue” in the sense they are using it in the post.

Try modifying it to clear the array (a DB flush, if you will) every 100k lines.

That's what you get if you use the (simplified) example code that is in the article. It's also what you can derive from their description of the problem.

Now we don't have any Rust code to compare so we can't say how it's different. But everything points towards to a memory allocation issue that is just not their in simple Rust code.

Following you in the abstract, but still a bit confused.

* "unbounded array" - are you referring to `records`?

* If so, is the memory of `records` too large? Or is it that `records[i].pathname` has a reference to `fields[7]`, AND v8 will retain all copies of `fields` until the loop is done?

Do you think this code would be more GC-friendly?

```

let count = 0;

const MAX = 1000;

const flushedRecords = [];

for await (const line of readlineStream) {

  const fields = line.split('\t');

  const httpStatus = Number(fields[5]);

  if (httpStatus < 200 || httpStatus > 299) continue;

  records.push({

    pathname: fields[7],

    referrer: fields[8],

    // ...

  });

  if(count == MAX || isLastLine(line)){

   flushedRecords.push([...records]);

   records = [];
   
    count = 0;

  } else {
    count++;
  }
  

}

```

Or were you thinking something totally different?

We know a few concrete things:

1. It parses 200gb at an average rate of 700kb/s

2. The job splits strings into a format that is loaded into mongodb

3. There where OOM errors that went away with a smaller workload

None of this points to an inherent issue with NodeJS and the way it references memory.

The speed issue alone points to a different problem root cause, and the post is completely lacking in concrete analysis.

If v8 is being dumb and retains references to huge strings, then understanding this is the case and forcing a copy (and thus enabling the gc to reclaim data) is quicker and easier than any of their attempts at fixing the issue.

Splitting a small TSV file and loading it into mongo can be done with even the most memory inefficient language without issue.

"and the post is completely lacking in concrete analysis."

I do agree here, but I think it's telling that the rust solution is, without analysis, just better.

Is this the best developer or best case study? Perhaps not, but it is interesting in that little analysis was needed.

In the case where you have some constraint placed on you (running in the browser, company policy forbids rust e.g.), the analysis is worthwhile. Otherwise, assuming the fix didn't take long (relative to the requisite analysis and dev training), the decision to switch to a less shoot-yourself-in-the-foot language seems reasonable to me.

I think we can all agree that the blog article, while interesting and tells a story about the way the author solved a problem, does not have high enough rigor to serve as a benchmark for "don't do this, do this instead", or "this language is bad at this, and this one is better". Those types of statements would require much more rigor and A vs B comparisons where code A and code B are the same, just in different languages.

BUT, what I am interested in - if I'm using javascript to parse a large file line by line, plucking off specific fields like the example does, what can I try to see if it's more GC-friendly? Obviously the answer is always some flavor of "profile it" and "it depends". If the blog author could have gotten a big boost in performance by adding 10 lines of js code to their loop, that is something all readers of the article could benefit from.

I will play around with it and see what I can learn about the V8 GC. For reference: here is a talk about profiling the ruby gc, which helped me fix a memory leak at work (https://www.youtube.com/watch?v=kZcqyuPeDao).

There's a lot of other considerations when adding a new language. Questions come up such as:

* How long does it take to ramp up on the new code base (author notwithstanding as a user of Rust outside of the company)?

* If the author of the code is sick and there's a bug, or logic needs to be added, how hard would it be for someone else to do this work?

* What is the build story for the Rust code like? How does it integrate with existing CI?

* How do you store the Rust library? Some artifact store that Node can interact with?

Without more details, unless the company was already writing Rust, this doesn't seem like a reasonable choice at all. Doing some profiling and experimenting with allocation, along with comments as to why you're doing something that's not naive (e.g. appending to a large array) seems much cheaper than doing all this Rust work.