When you say "we survived without it, we can do it again", you're implying that everyone can do it. But "we" still need to solve several major problems:
- National defense
- Fuel
- Power
- Medicine
- Food
- Construction
- Textiles
The "retreat to a small town" hasn't really ever worked as a large-scale model. Small towns in Europe were still backed up by national militaries, as were small towns in the U.S. There has to be some large-scale, militaristically competitive infrastructure in place before a calm, slow existence could return to a meaningful fraction of the population.
In broader context, we can have everyone say working 20 years in modern, corporate world, so that it does not crumble and we still get to enjoy its benefits, and then retiring to quiet life. It's what people some people with higher salaries are already doing (the FIRE movement), and with more wealth in the economy it would be available to everyone.
If you have a chance to get rid of it and make it work, go for it. We survived before without it, we can do it again. :)