That's the same with anything. How many endeavors have you thought "I'd love to do that but what if I got hurt in the process?" Relationships, startups, new jobs, everything contains a risk.

If it seems interesting, just do it. You'll thank yourself after you get over that fear. (Also buy a helmet and learn how to fall[1], you'll need it. Pads may be overkill but couldn't hurt.)

[1] https://m.youtube.com/watch?v=Hundbrub8iQ