I could be wrong but I'm pretty sure SDL just reads the keydown/keyup events it gets from the X server (that's why you don't need to choose a keyboard device when you use it). So without X running SDL wouldn't be able to grab keyboard events.

It can grab joystick/gamepad events though since it reads those directly. In that case you have to enumerate then specify which gamepad/joystick device to use so it can open the correct /dev/ path.