News for hackers. Stuff that matters.
Re: by ddltodata on Tuesday January 06, 2026 @09:20AM (#46512572)
Thanks! At 1M rows, I think a few things matter:

Streaming: Can't hold it all in memory. Generate in chunks, write, release, repeat.

Format choice: Parquet with row groups is fast and compresses well. SQL needs batched inserts (~1000/statement). Direct DB writes via COPY skip serialization entirely is usually fastest.

FK relationships: The real bottleneck. Pre-generate parent PKs, hold in memory, reference for children. Gets tricky with complex graphs at scale.

Parallelization: Row generation is embarrassingly parallel, but writes are serial. Chunk-then-merge is on our radar but not shipped yet.

What does your stat product need, realistic distributions or pure volume/stress testing?

Re: by rrr_oh_man on Tuesday January 06, 2026 @10:51AM (#46513843)
Why does this read like AI slop?
Re: by ddltodata on Wednesday January 07, 2026 @12:06PM (#46528998)
because it is, but its still true lol