Just this prior weekend, we had to be in Columbus, OH for my inlaw grandparents' 70th wedding anniversary and then I had to be back to head out to Re-invent in Vegas. With the hurricane possibility, we cancelled our plans to use our airplane and booked 3 day advance purchase trips on one carrier to Cleveland (to avoid connections on the east coast), then drive to Columbus, then fly out of Columbus on a different carrier.
Was expensive on its face (4 people, late notice direct flights), but cheaper than risking killing my whole family tangling with weather. Of course, the bad weather didn't end up materializing, but you have to be willing to make those calls frequently to use general aviation for travel to take the "get-there-itis" pressure off.
If you're going to sweat the money for commercial tix/hotels/cabs or are not willing to sit in the FBO weather room and pilots' lounge if you're caught in weather in the middle of a trip, your risk is going to be way higher than it needs to be.
So if you remove the "accident waiting to happen" category of pilots, the fatal accident rate will likely be much lower. I've heard it said that GA has a similar accident rate to riding motorbikes, which is another activity where the risk mostly depends on the rider.
The type of pilot who is most dangerous is the type-A business owner who feels they have to get to their meeting no matter what the weather. Unfortunately these are the people who will have $250k in their pocket and want to use a plane for transportation. Perhaps the only solution is to have self-flying planes, similar to google's cars. (And of course, they would need to be hacker-proof!)