I'm sure there's an even better positional encoding that can get around the ugly feature that both triangle and sine waves have of having distinct points where they "change direction." I suspect there's a way around this by sampling values from a 2 dimensional space while moving in a circle, something like that.