One can do that, rotate by 45 degrees, do that again, and do some further bit manipulation to combine the two bits of the first 4-way version with those of the second into the three bits this should produce.
That would be four comparisons, a 2×2 matrix multiplication (edit: that’s a matrix-vector multiplication) with a known-ahead rotation matrix (naively, that would be 8 (edit: 4) multiplications and 4 (edit: 2) additions, but I think we can lose those multiplications because all elements of a 45 degree rotation matrix have absolute value 1/√2 and the problem is scale-invariant) and some bit twiddling.
Now, make that branchless…
Edit2: and of course, you can do both splits into quadrants by just looking at the signs of x and y. That rotates the four quadrants by 45 degrees, but that doesn’t matter.
That makes the 4-way version simply (x>0) + 2 × (y>0): two comparisons and a bit shift.
twoBits = (x > 0) + 2 × (y > 0)
x’ = x + y
y’ = x - y
twoMoreBits = (x’ > 0) + 2 × (y’ > 0)
(Some bit twiddling with ‘twoBits’ and ‘twoMoreBits’
Feel free to compute those bit pairs differently if
it makes the bit twiddling easier)