Did it fail (outside mainframes) for technical reasons, or because the filesystem won the popularity contest? There's plenty examples on tech where one thing wins out for reasons other than technical merit.

With that kind of remark I guess you never used a mainframe.

In what way do mainframe file systems differ? As far as I'm aware, they largely use heirarchical file systems, like the confusingly named HFS or ZFS -- both of which are colliding acronyms from within IBM. HFS is the older Hierarchical File System, and ZFS being the newer z/Architecture File System.

As far as I'm aware, the main difference is support for record oriented files, but the naming and lookup isn't so different.

Well, I think even more than HFS or zFS, they use the MVS classic filesystem (datasets etc.) But that too is semi-hierarchical. (I say "semi" because to some extent the hierarchy is just a naming convention, but to some extent it's real – the initial qualifiers of a dataset name can be really hierarchical in that can select which catalog is used; and then of course PDS/PDSE members are an additional one-level hierarchy on the end.)

But you are right, there is nothing especially database-oriented about file lookup and naming on IBM mainframes. Record-oriented files and key-sequenced VSAM files (and once upon a time ISAM too) are database-oriented features, but they relate to file contents not file naming/lookup/etc.

I think the idea of catalogs is interesting, in that they permit a separation between the naming of datasets and the volumes they are stored upon. A dataset can be moved to another volume without changing the name used to access it. That is arguably more complex on Unix-like systems, since you need to muck around with symlinks or bind mounts to get the same effect.