When setting an environment variable gives you a 40x speedup

Re:When setting an environment variable gives you a 40x speedup by arendtio on Friday April 26, 2019 @07:49PM (#19762739)

Thanks for the additional information but sometimes I wonder that 15k+ files count as 'many' nowadays. I mean, an off-the-shelf Canon DSLR produces 20k files per directory (10k JPG + 10k raw). Nevertheless, I did some test your approach on my PC:

On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance. Without modification, it takes 0.5 seconds to list a directory (on a slow consumer grade HDD) with 14448 files and after setting

  export LS_COLORS='ex=00:su=00:sg=00:ca=00:'

it takes 4.4 seconds.

However, listing a local (SSD) directory with 15k files takes just about 0.12 seconds and gets faster after settings the variable to 0.06 seconds.

For all tests I drop the caches before testing, e.g.:

  echo 3 > /proc/sys/vm/drop_caches; time ls --color=always /mnt/nfs/many | wc -l

[ Parent | on HN ]

Re:When setting an environment variable gives you a 40x speedup by mrb on Saturday April 27, 2019 @02:09AM (#19764089)

"On a NFS mounted on my PC your LS_COLORS tweak actually degrades the performance"

Run "strace -o logfile ls --color=always" and diff the two logfiles.

[ Parent | on HN ]

Re:When setting an environment variable gives you a 40x speedup by arendtio on Saturday April 27, 2019 @04:43AM (#19764563)

Well, this looks rather strange. As expected, settings the variable causes large changes as it removes a lot of lstat calls and also removes the color codes from the writes. So there seems to be no obvious explanation, why it becomes slower when doing less...

  # du -sh logfile_nfs*
  1.7M    logfile_nfs
  668K    logfile_nfs_exported

However, the strace -c output looks kinda different:

Without modifying the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...

With setting the variable: https://bin.disroot.org/?149fa91c08b27312#0w9O6BAWNEEC4SUXEb...

[ Parent | on HN ]

Re:When setting an environment variable gives you a 40x speedup by mrb on Saturday April 27, 2019 @01:27PM (#19766591)

You pasted the same link for both cases. Can't tell what's going on.

[ Parent | on HN ]

Re:When setting an environment variable gives you a 40x speedup by arendtio on Saturday April 27, 2019 @03:39PM (#19767240)

Oh sorry, my fault. The second link should be

https://bin.disroot.org/?56f3dcb618240df0#x3StRGZZgSzM0UJ0BK...

[ Parent | on HN ]

Re:When setting an environment variable gives you a 40x speedup by gnufx on Saturday April 27, 2019 @09:33AM (#19765401)

Plus/minus what? Note that caching on the server can be relevant, and you can't control that from the client. That's one thing that requires care for performance measurements on networked filesystems.

[ Parent | on HN ]

Re:When setting an environment variable gives you a 40x speedup by arendtio on Sunday April 28, 2019 @05:33AM (#19770202)

Sounds as if you don't trust my statement. Sure there is some variance, but when one is measured in seconds and the other in the tenth of a second it should be clear that it is not a coincidence...

But for completeness, I wrote a few loops which should answer your question:

=NFS=

Normal: https://bin.disroot.org/?8967a0f34b26e512#aTYmbyESfeuqAXS802...

With exported Variable: https://bin.disroot.org/?c405d7aa74b50ee0#KverJYJEzNhct7CmdW...

-----

=Local=

Normal: https://bin.disroot.org/?b94e1d1f58e3edb7#tKszD/tjvwBwepJLun...

With exported Variable: https://bin.disroot.org/?d3ac83f1dff9e767#WVODjGnvL1QbOJdWPp...

I also tried dropping the cache on the NFS server but that didn't seem to have a major effect on the performance (probably because reading a 15k file index from a local disk doesn't take that long after all).

[ Parent | on HN ]