Really? In the mid-19th century?
https://ourworldindata.org/child-mortality/ says "in 18th century Sweden every third child died, and in 19th century Germany every second child died" (that's before the age of 5; the usual definition of child mortality). The "England and Wales" chart on that site shows that in the mid-1800s child mortality was about 250-300 per 1000.
So out of 10 children you would expect 2-3 to die before reaching age 5 at the time.
If you look at https://en.wikipedia.org/wiki/Charles_Darwin#Children there were 10 children. Their ages at death are:
74, 10, 1 month, 84, 67, 78, 77, 93, 77, 18 months.
So two deaths before age 5, one death at 10, the rest living to perfectly reasonable ages. Doesn't seem at all unreasonable for the time period.
As for today, child mortality in developed countries is well less than 1%. And most of that is infant mortality (death before age 1). For example, http://apps.who.int/gho/data/view.main.CM1320R (sorry, somewhat slow) shows that in the US in 2015 infant mortality was 5.6 per 1000, while child mortality was 6.5 per 1000. Neonatal mortality (death within 4 weeks of birth) was 3.6 per 1000.
And a lot of that is preterm births; for full-term births the US rate of infant mortality is closer to 2.5 according to http://www.webmd.com/baby/news/20091103/preemies-raise-us-in...
(Insert here rant about how different countries report some of these numbers differently, with some counting a 21-week birth followed by death a day later as a miscarriage, some counting it as a stillbirth, and some counting it as a live baby that died. As the mortality numbers drop these edge cases matter more and more. Obviously back when child mortality was in the mid-double-digit percentages these edge cases were not really important.)
What was your naive assumption about child mortality rates, if I might ask?
And as for a century and two ago I'd be waaaaaaay off base. Probably 10% or even 5%. It's hard to say in the hindsight. Now it seems so obvious and makes you feel stupid.
Darwin was aware of the risks of inbreeding, and this marriage in particular, expressing those concerns in various writings.
They had 9 or 10 children, IIRC, with 3 or 4 of them dying in childhood, and only a couple going on to produce a further generation. (Not so uncommon these days, but at the time I believe this was quite noteworthy.)