I started running into these issues when I started working with training examples in the context of deep learning. A folder with millions of files is then not unheard of.
Also if you do things like */*/*, then you can quickly get large command lines. Or even if you do long_name/another_long_name/*.
Re:Use Long Options in Scripts
by wodenokoto on Sunday March 23, 2025 @03:31AM
(#43451409)
I've had problems with "cat *.csv" plenty of times processing data that is generated in many small files.
It is really difficult to deal with, because on top of the arg max limit, globs are not guaranteed to be in order.
The solution is not obvious and hard to get to if you don't know the foot guns in advanced and hard to read once implemented.
Since using Linux exclusively, I don't think I've ever encountered an issue due to too many arguments/length. And it's the first time I'm actively searching online for ARG_MAX.
I understand that different shells might be different, but with reasonable lengths is there any chance of it being relevant (aside from xargs, where it's generally intended, or better, to pass along each argument individually).