You are thinking of the Unix compatibility subsystem of z/OS (formerly known as OS/390, and before that as MVS). That came out around 1993. But prior to that there was a long history of running Unix on IBM (and IBM-compatible) mainframes as an independent operating system. This work at AT&T, I don't even think it was the first port, I think Princeton's port in 1977 was the first [1].
And then, as well as Princeton's VM/370-based port and AT&T's TSS/370-based port for 5ESS development, Amdahl UTS came out in 1981 (descended from the Princeton port) and IBM/Interactive Systems VM/IX around 1984; then in 1988, the IBM/Locus AIX/370, followed by AIX/390.
(Also some 1980s-era Japanese supercomputers from Hitachi and Fujitsu had a hardware architecture based on IBM mainframes and some of them also ran Unix, although I'm not sure which version.)
> Anyway, as far as I can remember it was soon superseded by the Linux subsystem.
It was never superseded per se, it still exists. But the use cases are different. If you just want a vanilla Unix-like environment on an IBM mainframe, run Linux – which is not a z/OS subsystem, it is separate independent operating system. If you want a Unix application environment which supports deep integration into legacy mainframe applications and subsystems (VSAM, CICS, IMS, ISPF, etc) then that is the use case for the z/OS Unix subsystem.
In the latest release of z/OS, IBM has brought in a third option: zCX [2]. This runs Linux-based Docker containers on a hypervisor inside z/OS. This is positioned for z/OS system administrators to be able to easily deploy and manage Docker containers on the mainframe without learning extensive Linux skills (which managing z/Linux requires), using mainly their existing z/OS skills along with learning a bit about Docker.
[1] https://www.bell-labs.com/usr/dmr/www/portpap.pdf
[2] https://www.ibm.com/support/z-content-solutions/container-ex...
Despite Posix compatibility it wasn't very smooth. One infelicity I remember was that merely untarring a tar file took an extremely long time because ownership information had to be gotten from the ID system (RACF?). And it made separate, expensive calls for each file. Perhaps multiple.
Anyway, as far as I can remember it was soon superseded by the Linux subsystem.