It is also a good idea to try to organize your way out of large directories (that is, directories with many immediate children). Some filesystems dislike large directories performance-wise, and manual workflows rarely deal well with very long file sequences.

For what it's worth, I was massaging several GB of Newsgroup data into SQL. As I recall, I used grep to decompose the data into components. I crunched one year segments. For each year, each type of header, and message bodies, went into a subdirectory, tagged with the message ID. So in each of those subdirectories, there was one file for each message in the data segment. I did it that way, because each type of component needed custom regex to become SQL fields. In the end, the header types and message bodies became tables, indexed and linked by message IDs.

So anyway, there were lots of temp files.

Why did I do that? You might ask. Well, there was this notorious troll, who was on a vendetta against friends and I. He didn't just post trash. He spoofed our messages, making fun of us, and trying to sow discord. So it was virtually impossible to filter out his shit.

So basically I identified many of his personas, going back a decade or two. And eventually I found posts that included unobscured IP addresses and meatspace email addresses. So I emailed him, and threatened to dox him, if he didn't stop trolling us. So he did. And I never actually doxxed him, because that would have violated his privacy.

It was Ari Silverstein, by the way.[0] None of my personas posted there, but people were aware of my work.

0) https://www.velocityreviews.com/threads/tracking-ip-addresse...