I write statistical programs that often generate temporary intermediate files of multiple GB size, which need to persist at the end of runs for trouble-shooting, partial re-runs etc. i.e. the software shouldn't delete them when it's finished, but they don't need to be kept long term. I kept forgetting to delete them when I finished a session and my disks kept filling up, so I decided to use /tmp and let the OS manage the space. Is there a better way to manage large tmp files than this that I'm missing?
Even before shifting to tmpfs I'm pretty sure Fedora was erasing the contents of /tmp at startup (and I know Gentoo/OpenRC is).
There might be an even more appropriate FHS-compliant directory (/var/run perhaps?)
/var/run "This directory contains system information data describing the system since it was booted" which doesn't seem to fit my use case that well. Not to say there isn't another directory that does fit, but I haven't found it.
In your case, what could work is having the new run use the same file names as the old run, or coming up with a rotation schedule (so you would keep the latest N runs)
But if having them on /tmp as it is working for you, great, no need to complicate things.
There are also speed considerations, if you're using these files as actual intermediate storage (as opposed to a log/debug dump) it may be interesting to have them written to a different disk (like an SSD)
As you say, /tmp is working well for me, my example was to perhaps illustrate that there are (what I think are) legitimate uses for large files in /tmp.
I mount /tmp as a tmpfs for almost exactly this reason. I have an always-on home server/HTPC that runs the OS off of a small (40 GB) SSD, and I don't want to wear down the SSD with writes to /tmp, /var/tmp, /var/run, /var/log, etc. In theory, I don't want any writes at all, unless the thing is actually doing something useful. So I simply threw in 2GB extra RAM and mounted all these temporary/logging directories as tmpfs. Periodically (and on reboot) I zip the logs up and copy them to a backup directory just in case something fails.
Of course this runs the risk of losing temporary data and logs whenever the server goes down, but that's a trade-off I'm willing to make. It's been running just fine for over 2 years now. I'd do the same thing straight away if I ever need to set up some similar system.
Nothing stops you from changing fstab to make any mount tmpfs or disk.
Where else are temporary files supposed to go, exactly? It's no abuse to put in /tmp temporary "scratch" data which have no reason to survive a process restart let alone a machine reboot, that's the mount point's explicit purpose.
Better to add complexity once, in the filesystem, than in every program that writes to it.
If you're going to trash my /tmp with gigantic files please don't. Really
"itβs better to fix the filesystem to make it faster"
Yes, let's spend adding more complexity to the file system to work out every /tmp abuser out there. But his original point was avoiding adding complexity in the first place
So basically, he contradicts himself.
Mounting tmp as a memory backed fs is great (to which there may be better options than tmpfs), avoids (potentially) spinning up disks covered in rust for minor tasks.