We Put Half a Million Files in One Git Repository, Here's What We Learned (2022)

Re:We Put Half a Million Files in One Git Repository, Here's... by eigenvalue on Monday August 28, 2023 @12:50PM (#37297022)

In my experience, the standard linux file system can get very slow even on super powerful machines when you have too many files in a directory. I recently generated ~550,000 files in a directory on a 64-core machine with 256gb of RAM and an SSD, and it took around 10 seconds to do `ls` on it. So that could be a part of it too.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by tp34 on Monday August 28, 2023 @05:32PM (#37300664)

What is the "standard linux file system"?

ext4 on an old system, feeble in comparison to yours, performs much better.

ext4, 8GB memory, 2 core Intel i7-4600U 2.1GHz, Toshiba THNSNJ25 SSD:

$ time ls -U | wc -l 555557

real 0m0.275s user 0m0.022s sys 0m0.258s

stat(2) slows it down, but sill this is not as poor as your results:

$ time ls -lU | wc -l 555557

real 0m2.514s user 0m1.126s sys 0m1.407s

Sorting is not prohibitively expensive:

$ time ls | wc -l 555556

real 0m1.438s user 0m1.249s sys 0m0.193s

Drop caches, sort, and stat:

# echo 3 > /proc/sys/vm/drop_caches

$ time ls -lU | wc -l 555557

real 0m6.431s user 0m1.249s sys 0m4.324s

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by zamadatix on Monday August 28, 2023 @09:23PM (#37302417)

It sounds suspiciously like you measured the time to display 500k lines in the terminal instead of the time to ls.

[ Parent | on HN ]