My toy has a 3 MB L1 cache, 8 MB L2. You'll notice from the same figures that the resident memory usage of `nnn` was 3616 KB. And in the default start (not du) right now the figure is:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
23034 vaio 20 0 14692 3256 2472 S 0.0 0.0 0:00.00 nnn -c 1 -i
> check if a particular file is cached and if not
I am a userland utility. I don't add deps on fs drivers (and maintain redundant code... I discourage code bloat as much as feature bloat) when standard APIs are available to read, write, stat.
> However regardless of where that cache is, it’s volatile, it’s freeable.
Yes!!! The worst case! And the worst case for you is where all dirs have changed. There you issue rescan, `nnn` issues rescan. `nnn` is accurate because it doesn't omit independent files for _personal_ optimization preferences. You do. You show _stale_, _wrong_ info in the worst case.
Some program executables are multiple megabytes in size these days, your CPU cache is much more valuable storing that than it is storing a random 20MB XLSX file you happened to copy half an hour earlier. ;)
> You do. You show wrong info in the worst case.
Yes, I’d repeatedly said that myself. However I was making a case study rather than giving a lecture on how your tool should be written. Remember when I said “there is no right or wrong answer”? (You have ignored my point that you cannot guarantee the kernels / fs drivers cache isn’t also stale though. That does happen in some rare edge cases).
My point is you shouldn’t be so certain that you’re method is the best because there are plenty of scenarios when accuracy and low memory usage is more costly (eg the fastest file copying tools are those that buffer data - costing more RAM during usage). Plus let’s not forget that syscalls are expensive too. But that’s the trade off you take and that is what is advantageous for the perpose you’re targeting nnn for.
So there really isn’t a right or wrong answer here at all and memory usage is in fact only a very small part of the equation (which is what the other projects author what trying to say).
I have shared the figures from my system right now in my earlier comment. The memory required for the meta info is extremely cache-able.
And yes, let's call it a day now. In fact, I am happy we had a very informative discussion towards the end. This is what I expected from the other dev and you from the very beginning. I am always ready for a productive non-abusive technical discussion. And if I find someday `ncdu` takes comparable/lesser memory than `nnn` I will update that data verbatim as well. But I wouldn't take uncalled-for shitstorm in public forums from strangers lying low.
You’re making so many assumptions about the host platform and they simply don’t stack up for the majority of cases - it’s just that most systems are fast enough to hide your incorrect assumptions.
Also your understanding of how file system caching works is completely off. Granted that’s not something most normal engineers need to worry about, but given the claims you’re making about nnn, I would suggest you spend a little time researching the fundamentals here.
Edit:
> I am happy we had a very informative discussion towards the end. This is what I expected from the other dev
That was exactly what the other dev was doing. You just weren’t ready to listen to him - which is why I started making comments about your “narky” replies. ;)
You cannot just throw numbers up and say “look I’m right” if the test of flawed from the outset.
But yes, it’s probably better we do call it a day.
## Fired `nnn` in my home dir.
- all cached, no major page faults
~$ /usr/bin/time -v nnn
Command being timed: "nnn"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:03.77
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3400
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 274
Voluntary context switches: 6
Involuntary context switches: 1
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
~$
## Fired `nnn` on an external disk root in du mode, went into a subdir, came back to root.
- no major page faults
- File system inputs: 14680
- 220 KB cached extra
~$ /usr/bin/time -v nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/
Command being timed: "nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/"
User time (seconds): 0.02
System time (seconds): 0.27
Percent of CPU this job got: 3%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:08.30
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3620
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 337
Voluntary context switches: 576
Involuntary context switches: 2
Swaps: 0
File system inputs: 14680
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
~$
## Fired `nnn` again (around 2 mins later, I was formatting the above text) on the external disk root in du mode.
- File system inputs: 0
** So even between 2 distinct instances of nnn, _all_ the data fetched in the first instance remained cached.
~$ /usr/bin/time -v nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/
Command being timed: "nnn -S /media/vaio/49462fdf-010c-40cc-89db-ef125e7dae99/tmp/"
User time (seconds): 0.00
System time (seconds): 0.01
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:02.31
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 3616
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 329
Voluntary context switches: 5
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
~$What we were arguing about was the L2 cache - which you argued could hold a 2GB file (insanity I call it) and you’re still not proving that point of yours. In fact you’re not even checking if it’s cached in the system memory (there are syscalls to do that), you’re just checking application performance. While it’s pretty obvious that result would have been due to caching it would have been easy to prove that more conclusively by even just flushing the cache (I assume you actually know how to do that? Hint: you can just write to the /proc directory from your shell to do that. It’s easy) and then repeating the test; but You didn’t even do that!
You’re testing is so rudimentary at covering the cases you’re making claims against it’s laughable. you’ve not even begun to address the “processor memory” point you were arguing.
But I honestly think we should just leave it here. You don’t need to be an expert in kernel design nor practical experience in write file systems to write a decent file manager, so all of this arguing is pointless. I just ask that you aren’t so quick to judge other solutions when it’s painfully obvious there are massive gaps in your own understanding.
However regardless of where that cache is, it’s volatile, it’s freeable. And thus you cannot guarantee it will be there when you need it. Particularly on systems with less system memory (remember that’s one of your target demographics).
If you wanted though, you could easily check if a particular file is cached and if not, perform the rescan. I can’t recall the APIs to do that off hand but it would be platform specific and not all encompassing (eg ZFS cache is separate to the kernels cache on Linux and FreeBSD) so it wouldn’t be particularly reliable nor portable. Plus depending on the syscall used, you might find it’s more overhead than an actual refresh on all but a rare subset of edge cases. As an aside, this is why I would build my own cache. Sure it cost more RAM but it was cheaper in the long run - fewer syscalls, less kernel/user space memory swapping, easier to track what is in cache and what is not, etc. But obviously the cost is I lose precision with regards to when a cache goes stale.
While on the topic of stale caches, I’ve actually ran into issues on SunOS (or was it Linux? I used a lot of platforms back then) where contents on an NFS volume could be updated on one host but various other connected machines wouldn’t see the updated file without doing a ‘touch’ on that file from those machines. So stale caches is something that can affect the host as well.