I did the same thing to support about 10 million customer websites, but had the opposite experience - it worked pretty well, we used UFS and a 4 level directory structure (i.e. customer "apple" went into '/a/p/p/l/apple' and just wrote a simple apache module to map URL's to this directory structure.(using a hash of the filename would have given more even distribution, but this naive algorithm worked pretty effectively)

It was much easier than coding a database back-end, with no need to run a database server. And, when we needed to scale beyond one machine, it was trivial to use a central NFS server (read-only NFS worked really well for this) and multiple front-end web servers.

Though we did know how many files we'd be hosting, so we sized the number of bytes per inode appropriately.