That's true only for modern filesystems. For decades, the standard directory storage in unix filesystems was a plain unordered list. Finding a single file in a large directory required reading half the directory on average. Double the size of the directory and you double the time it takes to seek through it. It's only recently that most systems are shipping with filesystems that are O(1) or O(logN) in the directory size.