I would simply not define "prime" or "composite" for 1, yes. If you check abstract algebra books (or wikipedia [1]), you'll usually find definitions along the lines of "a non-invertible, non-zero element is prime if and only if ...", and the nice thing about this definition is that it is a useful concept in more general structures than just natural numbers, namely (semi)rings.

[1](https://en.wikipedia.org/wiki/Prime_element)