News for hackers. Stuff that matters.
Re:Debian switches to 64-bit time for everything by jeroenhd on Monday July 28, 2025 @07:57AM (#44709967)
You can recompile your kernel to work around the 100k-ish command line length limit: https://stackoverflow.com/questions/33051108/how-to-get-arou...

However, that sounds like solving the wrong end of the problem to me. I don't really know what a 4k JPEG worth of command line arguments is even supposed to be used for.

Re:Debian switches to 64-bit time for everything by tomrod on Monday July 28, 2025 @08:13AM (#44710074)
A huge security manifold to encourage adoption then sell white hat services on top?
Re:Debian switches to 64-bit time for everything by dataflow on Monday July 28, 2025 @08:32AM (#44710203)
> I don't really know what a 4k JPEG worth of command line arguments is even supposed to be used for.

I didn't either, until I learned about compiler command line flags.

But also: the nice thing about conmand line flags is they aren't persisted anywhere (normally). That's good for security.

Re:Debian switches to 64-bit time for everything by eightys3v3n on Monday July 28, 2025 @08:42AM (#44710280)
There were ways for other users on the system to see running process command line flags I thought. This isn't so good for security.
Re:Debian switches to 64-bit time for everything by dataflow on Monday July 28, 2025 @09:20AM (#44710625)
That depends on your threat model.
Re:Debian switches to 64-bit time for everything by Ghoelian on Monday July 28, 2025 @09:16AM (#44710580)
Pretty sure `.bash_history` includes arguments as well.
Re:Debian switches to 64-bit time for everything by dataflow on Monday July 28, 2025 @09:19AM (#44710612)
That's only if you're launching from an interactive Bash? We're talking about subprocess launches in general.
Re:Debian switches to 64-bit time for everything by johnisgood on Monday July 28, 2025 @09:37AM (#44710756)
If you add a whitespace before the command, it will not even get appended to history!
Re:Debian switches to 64-bit time for everything by zamadatix on Monday July 28, 2025 @10:08AM (#44711033)
I had never run across this and it didn't work for me when I tried it. After some reading it looks like the HISTCONTROL variable needs to be set to include "ignorespace" or "ignoreboth" (the other option included in this is "ignoredups").

This would be really killer if it was always enabled and the same across shells but "some shells support something akin and you have to check if it is actually enabled on the ones that do" is just annoying enough that I probably won't bother adopting this on my local machine even though it sounds convenient as a concept.

Re:Debian switches to 64-bit time for everything by wongarsu on Monday July 28, 2025 @10:38AM (#44711322)
I'm mostly using debian and ubuntu flavors (both on desktop and the cloud images provided and customized by various cloud and hosting providers) and they have all had this as the default behavior for bash

YMMV with other shells and base distros

Re:Debian switches to 64-bit time for everything by zamadatix on Monday July 28, 2025 @07:20PM (#44716976)
Proxmox (debian 12.x based) and Arch both didn't for me (both w/ bash). An Ubuntu 24.04 container in Proxmox did though.
Re:Debian switches to 64-bit time for everything by amelius on Monday July 28, 2025 @09:38AM (#44710775)
This is quite annoying behavior, actually.
Re:Debian switches to 64-bit time for everything by nosrepa on Monday July 28, 2025 @10:04AM (#44710989)
Good thing you can turn it off!
Re:Debian switches to 64-bit time for everything by amelius on Monday July 28, 2025 @12:42PM (#44712575)
Yeah, took me a while to figure that out though. Plus I lost my history.

Typing an extra space should not invoke advanced functionality. Bad design, etc.

Re:Debian switches to 64-bit time for everything by wongarsu on Monday July 28, 2025 @10:04AM (#44710991)
tar cf metadata.tar.zstd *.json

in a large directory of image files with json sidecars

Somebody will say use a database, but when working for example with ML training data one label file per image is a common setup and what most tooling expects, and this extends further up the data preparation chain

Re:Debian switches to 64-bit time for everything by NegativeK on Monday July 28, 2025 @10:10AM (#44711058)
I won't say use a database, but I will beg you to compress a parent directory instead of making a tar bomb.
Re:Debian switches to 64-bit time for everything by mort96 on Monday July 28, 2025 @10:16AM (#44711113)
That really just makes the problem worse: tar czf whatever.tgz whatever/*.json; you've added 9 bytes to every file path.

I mean I get that you're suggesting to provide only one directory on the argv. But it sucks that the above solution to add json files to an archive while ignoring non-json files only works below some not-insane number of files.

Re:Debian switches to 64-bit time for everything by stouset on Monday July 28, 2025 @11:20AM (#44711741)
Having an insane number of files in the same directory is already a performance killer. Here you’re talking about having something like 10,000 JSON files in one directory plus some number of non-JSON files, and you’d just be better off in all cases having these things split across separate directories.

Does it suck you can’t use globing for this situation? Sure, yeah, fine, but by the time it’s a problem you’re already starting to push the limits of other parts of the system too.

Also using that glob is definitely going to bite you when you forget some day that some of the files you needed were in subdirectories.

Re:Debian switches to 64-bit time for everything by wongarsu on Monday July 28, 2025 @12:51PM (#44712689)
But in this example the files are conceptually a flat structure. Any hierarchy would be artificial, like the common ./[first two letters]/[second two letters]/filename structure. Which you can do, but it certainly doesn't make creating the above tarball any easier. Now we really need to use some kind of `find` invocation instead of a simple glob

It also just extends the original question. If I have a system with 96GB RAM and terabytes of fast SSD storage, why shouldn't I be able to put tens of thousands of files in a directory and write a glob that matches half of them? I get that this was inconceivable in v6 unix, but in modern times those are entirely reasonable numbers. Heck, Windows Explorer can do that in a GUI, on a network drive. And that's a program that has been treated as essentially feature complete for nearly 30 years now, on an OS with a famously slow file system stack. Why shouldn't I be able to do the same on a linux command line?

Re:Debian switches to 64-bit time for everything by mort96 on Monday July 28, 2025 @02:13PM (#44713618)
> Does it suck you can’t use globing for this situation? Sure, yeah

Then we agree :)

Re:Debian switches to 64-bit time for everything by cesarb on Monday July 28, 2025 @10:17AM (#44711123)
> tar cf metadata.tar.zstd *.json

From a quick look at the tar manual, there is a --files-from option to read more command line parameters from a file; I haven't tried, but you could probably combine it with find through bash's process substitution to create the list of files on the fly.

Re:Debian switches to 64-bit time for everything by syncsynchalt on Monday July 28, 2025 @07:58PM (#44717325)
Yes, workarounds exist, but it would be nicer if the arbitrary limits were removed instead.
Re:Debian switches to 64-bit time for everything by badc0ffee on Monday July 28, 2025 @01:58PM (#44713456)
You don't need to tar everything in one command. You can batch your tar into multiple commands with a reasonable amount of arguments with something like `rm metadata.tar.zstd && find . -maxdepth 1 -name \*.json -exec tar rf metadata.tar.zstd {} +`.
Re:Debian switches to 64-bit time for everything by silverwind on Monday July 28, 2025 @10:12AM (#44711082)
There's numerous examples of useful long commands, here is one:

    perl -p -i -e 's#foo#bar#g' **/*
Re:Debian switches to 64-bit time for everything by PaulDavisThe1st on Monday July 28, 2025 @11:08AM (#44711642)
no limits:

    find . -type f -exec perl -p -i -e 's#foo#bar#g' {} \;
Re:Debian switches to 64-bit time for everything by Someone on Monday July 28, 2025 @01:53PM (#44713412)
That runs perl multiple times, possibly/likely often in calls that effectively are no-ops. To optimize the number of invocations of perl, you can/should use xargs (with -0)
Re:Debian switches to 64-bit time for everything by dr4g0n on Monday July 28, 2025 @03:55PM (#44714852)
No need for `xargs` in this case, `find` has been able to take care of this for quite some time now, using `+` instead of `;`:

    find . -type f -exec perl -p -i -e 's#foo#bar#g' {} +
Re:Debian switches to 64-bit time for everything by tremon on Monday July 28, 2025 @04:58PM (#44715565)
xargs constructs a command line from the find results, so if **/* exceeds the max command line length, so will xargs.
Re:Debian switches to 64-bit time for everything by Someone on Wednesday July 30, 2025 @05:07AM (#44732210)
xargs was written to avoid that problem, so no, it won’t. https://man7.org/linux/man-pages/man1/xargs.1.html:

“The command line for command is built up until it reaches a system-defined limit (unless the -n and -L options are used). The specified command will be invoked as many times as necessary to use up the list of input items. In general, there will be many fewer invocations of command than there were items in the input. This will normally have significant performance benefits.”

Your only risk is that it won’t handle inputs that, on its own, are too long.

Re:Debian switches to 64-bit time for everything by mort96 on Monday July 28, 2025 @10:13AM (#44711085)
Linking together thousands of object files with the object paths to the linker binary as command line arguments is probably the most obvious example of where the command length limit becomes problematic.

Most linkers have workarounds, I think you can write the paths separated by newlines to a file and make the linker read object file paths from that file. But it would be nice if such workarounds were unnecessary.

Re:Debian switches to 64-bit time for everything by AlotOfReading on Monday July 28, 2025 @11:21AM (#44711748)
I tracked down a fun bug early in my career with those kinds of paths. The compiler driver was internally invoking a shell and had an off-by-one error that caused it to drop every 1023rd character if the total length exceeded 4k.
Re:Debian switches to 64-bit time for everything by sdht0 on Monday July 28, 2025 @01:48PM (#44713370)
Re:Debian switches to 64-bit time for everything by ogurechny on Monday July 28, 2025 @04:59PM (#44715585)
This sounds like a job for named pipes. You get the temporary file, but nothing is actually written to disk. Or maybe unnamed pipes, if bash command redirection is suitable for creating the list of options.

Looking back, it's unfortunate that Unix authors offered piping of input and output streams, but did not extend that to arbitrary number of streams, making process arguments just a list of streams (with some shorthand form for constants to type in command line, and universal grammar). We could have been used to programs that react to multiple inputs or produce multiple outputs.

It is obvious that it made sense in the '70s to just copy the call string to some free chunk of memory in the system record for the starting process, and let it parse those bytes in any way it wants, but, as a result, we can't just switch from list of arguments to arbitrary stream without rewriting the program. In that sense, argument strings are themselves a workaround, a quick hack which gave birth to ad-hoc serialisation rules, multi-level escaping chains, lines that are “too long” for this random system or for that random system, etc.

Re:Debian switches to 64-bit time for everything by account42 on Monday August 04, 2025 @04:40AM (#44783307)
> Looking back, it's unfortunate that Unix authors offered piping of input and output streams, but did not extend that to arbitrary number of streams

They did though - file handles are inherited by child processes which allows you to pass one end of a pipe and then feed things into the other end. E.g. make uses this to communicate between recursive invocations for concurrency control.

Re:Debian switches to 64-bit time for everything by qart on Monday July 28, 2025 @11:34AM (#44711876)
I have worked on projects (research, not production) where doing "ls" on some directories would crash the system. Some processes generated that many data files. These files had to be fed to other programs that did further processing on them. That's when I learned to use xargs.
Re:Debian switches to 64-bit time for everything by Brian_K_White on Monday July 28, 2025 @12:13PM (#44712261)
Backups containing other backups containing other backups containing vms/containers containing backups... all with deep paths and long path names. Balloons real fast with even just a couple arguments.