News for hackers. Stuff that matters.
Re:Debian switches to 64-bit time for everything by wongarsu on Monday July 28, 2025 @10:04AM (#44710991)
tar cf metadata.tar.zstd *.json

in a large directory of image files with json sidecars

Somebody will say use a database, but when working for example with ML training data one label file per image is a common setup and what most tooling expects, and this extends further up the data preparation chain

Re:Debian switches to 64-bit time for everything by NegativeK on Monday July 28, 2025 @10:10AM (#44711058)
I won't say use a database, but I will beg you to compress a parent directory instead of making a tar bomb.
Re:Debian switches to 64-bit time for everything by mort96 on Monday July 28, 2025 @10:16AM (#44711113)
That really just makes the problem worse: tar czf whatever.tgz whatever/*.json; you've added 9 bytes to every file path.

I mean I get that you're suggesting to provide only one directory on the argv. But it sucks that the above solution to add json files to an archive while ignoring non-json files only works below some not-insane number of files.

Re:Debian switches to 64-bit time for everything by stouset on Monday July 28, 2025 @11:20AM (#44711741)
Having an insane number of files in the same directory is already a performance killer. Here you’re talking about having something like 10,000 JSON files in one directory plus some number of non-JSON files, and you’d just be better off in all cases having these things split across separate directories.

Does it suck you can’t use globing for this situation? Sure, yeah, fine, but by the time it’s a problem you’re already starting to push the limits of other parts of the system too.

Also using that glob is definitely going to bite you when you forget some day that some of the files you needed were in subdirectories.

Re:Debian switches to 64-bit time for everything by wongarsu on Monday July 28, 2025 @12:51PM (#44712689)
But in this example the files are conceptually a flat structure. Any hierarchy would be artificial, like the common ./[first two letters]/[second two letters]/filename structure. Which you can do, but it certainly doesn't make creating the above tarball any easier. Now we really need to use some kind of `find` invocation instead of a simple glob

It also just extends the original question. If I have a system with 96GB RAM and terabytes of fast SSD storage, why shouldn't I be able to put tens of thousands of files in a directory and write a glob that matches half of them? I get that this was inconceivable in v6 unix, but in modern times those are entirely reasonable numbers. Heck, Windows Explorer can do that in a GUI, on a network drive. And that's a program that has been treated as essentially feature complete for nearly 30 years now, on an OS with a famously slow file system stack. Why shouldn't I be able to do the same on a linux command line?

Re:Debian switches to 64-bit time for everything by mort96 on Monday July 28, 2025 @02:13PM (#44713618)
> Does it suck you can’t use globing for this situation? Sure, yeah

Then we agree :)

Re:Debian switches to 64-bit time for everything by cesarb on Monday July 28, 2025 @10:17AM (#44711123)
> tar cf metadata.tar.zstd *.json

From a quick look at the tar manual, there is a --files-from option to read more command line parameters from a file; I haven't tried, but you could probably combine it with find through bash's process substitution to create the list of files on the fly.

Re:Debian switches to 64-bit time for everything by syncsynchalt on Monday July 28, 2025 @07:58PM (#44717325)
Yes, workarounds exist, but it would be nicer if the arbitrary limits were removed instead.
Re:Debian switches to 64-bit time for everything by badc0ffee on Monday July 28, 2025 @01:58PM (#44713456)
You don't need to tar everything in one command. You can batch your tar into multiple commands with a reasonable amount of arguments with something like `rm metadata.tar.zstd && find . -maxdepth 1 -name \*.json -exec tar rf metadata.tar.zstd {} +`.