Visually, non-primes can be represented as a symmetrical shape

e.g. 6 can be shown as

  xxxxxx

but also as

  xxx
  ---
  xxx

or

  x|x
  x|x
  x|x

Whereas a prime like 5 doesn't have this symmetrical layout. Since 1 does have this kind of symmetry it belongs more with the non-primes.