I think you're right on the culture thing. My kid loves new books and understands that they contain stories. A household without that is far more likely to cause issues with reading because the motivation will be missing.
Because we read with our kids, all of them were reading by late 3/early 4 years old, by their own desire. While none have had a propensity for George Orwell at the third grade, they do read things beyond their age levels.
I was lucky, and my kids are lucky.
I wonder, are there any organizations that folks can volunteer at to help kids at your and my age in elementary school learn to read?
I think this is the huge thing in common here. If kids are read to from a young age some might pick it up almost "on their own" while others may wait and learn in early elementary school "on schedule", but I think will largely see similar outcomes. But the ones coming at it for the first time from _not_ having experienced reading, even if only from being read to, are starting from a perspective lacking both background and motivation.
Your story is a good reminder that there may be more than one way to do it, I often think the way I learned to read (in school, in first grade) is clearly the right way to do it, since it had such good results. (In particular, i still think that earlier than kindergarden is too early for reading).
On the other hand, the thing we had in common was the family culture of reading, everyone in my family were big readers. And my parents read to me a lot before I could read.