That's true only for modern filesystems. For decades, the standard directory storage in unix filesystems was a plain unordered list. Finding a single file in a large directory required reading half the directory on average. Double the size of the directory and you double the time it takes to seek through it. It's only recently that most systems are shipping with filesystems that are O(1) or O(logN) in the directory size.

This depends on the file system in question, and how it organizes it's data on disk. Some file systems have a max limit on how many files can be in a directory, others have a max limit on files in an entire volume etc. Granted todays system are a bit more robust, but can be still targeted towards a variety of usage scenarios.