Some non-negligible percentage of horse-riding was also done by the nobility, where the horses were used either for hunting, or for military campaigning—both places where a horse "avoiding danger" can actually be fatal.
Re: hunting, horses would often rear back from predatory-looking animals, dropping their riders and possibly dropping onto their riders, crushing and laming/killing them. "Hunting accident" was the old world's "boating accident."
Re: military cavalry usage, horses could be spooked into stampeding—this was an incredibly large problem when in a closely-packed cavalry formation. Cavalry warfare was phased out upon the introduction of guns not because horses were more likely to die by a bullet than by an arrowhead; but rather because the sound of rifle or cannonfire was usually enough to spook horses un-used to the sound, turning many of the first rifle-era battles into instant routs.
This phrasing makes me curious: how did "horse-related accidents" rank among the causes of death for people who rode horses everywhere? Or people who rode in horse-drawn carriages everywhere?