Though even skydiving has a very different time/risk profile than, say, mountain climbing (however you care to define). If skydiving time is measured only by time from leaving the plane to hitting the ground, you're measuring a fatality rate against a pretty short interval of time. (Same with scuba diving.)

Whereas other activities that may have similar risk profiles for typical "active participants" may have much different rates relative to the time spent actively participating in the activity.

There's no single metric that really represents a correct view.