XENIX for 386 was based on AT&T System V/386, which introduced the AT&T syntax to 32-bit x86. I've found some references to 32-bit XENIX still using an assembler called "masm" but I don't know if it was still based on Microsoft's MASM or just called that for compatibility, or whether it was AT&T or Intel syntax. Also by that point compilers and assemblers weren't included in the base OS anymore, but a "development kit" sold separately.
The Minix compiler and assembler also used Intel syntax.
There was also something called "SCO UNIX" which was apparently more of a straight adaptation of AT&T System V and might have used a pcc compiler and AT&T syntax. It's hard to figure out what happened when because there are a zillion different versions of Xenix/SCO Unix/OpenServer out there.
It was so expensive in Escudos, that for teaching an high school class about UNIX, the teacher would bring a tower (286 or early 386 model) that would be timeshared with the whole class, meaning taking 15 minute slots seating at it, while having prepared the exercises, as much as we could, in Turbo C 2.0 on MS-DOS.
It definitly wasn't SCO UNIX though.
It's "AT&T syntax" because it dates to the 1978 AT&T Labs effort to port UNIX to the 8086. [1] While the 8086 did not have virtual memory or hardware protection, its memory segmentation model was still adequate to support UNIX, without those. It was the first microprocessor practically capable of running UNIX, and this was realized before the chip was even released. The porting effort started immediately. (Though most of the energy would soon switch to the 68000 when that was released a year later.)
The AT&T folks did not even wait for Intel's assembler. (Written in Fortran, to run on mainframes, or on Intel's development systems). Nor did they closely model their assembler after it. They just took the assembler they already had for the PDP-11 and adapted it with minimal changes for the 8086. Quick and dirty. Which was okay. You're not supposed to write assembly on UNIX systems, anyway. Only the poor people who had to write kernel drivers and compilers would ever have to deal with it.
[1] https://www.bell-labs.com/usr/dmr/www/otherports/newp.pdf (see section III)