Most would not last very long I agree. But I don't think that those that find the hermit existence more to their liking (and can sustain such an existence for an extended period of time) would "come crawling back" or anything like that. Once my kid is out of the house, I don't really care about not kicking the bucket a little sooner if by doing so I could live out my last decade or two in peace and without having to deal with the daily stream of bovine manure that "civilization" entails.
Quick death in old age is one of the better scenarios. Worse things are: chronic pain, slowly losing ability to function (be it neurological or just joints giving up etc.) and also witnessing the same things in your spouse. Traditional cultures deal with it to a degree by children taking care of sick/senile parents, but here it won’t be the case.
A lot of these people would come crawling back to the civilization begging for help if their life depended on it. Which it probably will as they get older and sick. On a large scale, millions of people who contributed nothing to the society, but then demand expensive medical help would not work well.