Surely it's simply because of scenarios like a small rock's ability to insta-stop your board, basically guaranteeing your injury. Pretty much every time I've ever been knocked off a skateboard during my commute would be something you wouldn't even think/care about if you were on a bike. Not even close. One is a bike, the other is a heavy, tall pole (you) on a low, rolling platform. One is far more precarious, like keeping a reverse pendulum balanced.

Hell, the most common injury I've seen on a skateboard is just someone stepping onto one, misplacing their center of mass, and the board just rocketing out from under them, and then bruising their tailbone/wrists. Even an inexperienced bicyclist doesn't have a fail case close to that.