Say you want to try alternative payment processing methods on your site. Your site it tightly integrated with PayPal, and you want to add Stripe. You could (A) spend 1 day to bolt on Stripe with a hack in the HTML, (B) spend 4 days and add enough views to the frontend to make Strip work in a non-disgusting manner, or (C) spend two weeks and refactor the entire backend with a generic interface that will handle an infinite number of payment solutions.
All three lead to success, but you can't optimize for both time and perfection. Set a time budget. Leave breadcrumbs for the next programmer. Instill a culture that says, "Our code isn't perfect, we know this, but we do what we can with what we have." Done is better than perfect.
With forethought, you should have built your system originally with method C in mind. If he had done this at the beginning instead of just hacking it together then all this work after wouldn't have been needed!
1) Apply conscious planning and forethought. Some things are clearly going to be used a lot, some things might not. Use bolt-on hacks where appropriate, invest time for things you expect to be using forever.
2) When the longevity of the tool is not easy to pin down, test it out with a hack. Spend 1 day adding Stripe the hacky way, and keep a slushbucket of time budgeted for revisiting experimental hacks that look like they will last a while.
That's my general approach, anyway. Most of the time, a hack is good enough and anything more is a waste of time. The story is admittedly a little different for me though; for me, software is a tool to do my job- software is not my job.
We have only had to do 2 refactors in that time on 2 controls. That was more about understanding our customers needs more as we progressed rather than not enough planning though.
Previous to that I built a custom ecommerce site for a wholesale distributor. That whole process was more suited to a quick hack over a well thought out decision. That's why I say I agree in principle, so long as the project allows it.
That's how I develop now to be fair and not how I would build a product if I were to build it for myself.
At any rate, I'll tweak the "done-perfect" lever more towards done.
Facebook's motto reminds me of something from a Holman slidedeck: "Nothing great was ever not shipped."
Don't sabotage the larger vision for short-term gratification.
That being said, I guess it depends on what results these random maunderings produce, and it's probably a crapshoot as to whether they are more often than not to result in favorable outcomes. Also, I like your example (obviously :)
Chances are, you'd approach Town Hall with the solution to all their pothole problems and they would ignore it. To make it worthwhile, you'd either have to have earlier signed a contract with Town Hall to figure out their potholes (ie. have a customer) or develop a business based on your solution (ie. develop a market).
By now it's 9:30pm. It's dark and cold. You realize what your original purpose was: Dinner with friends. That was two hours ago. You missed dinner, but hey, you got some satisfaction.
The above was more of an analogy about Yak Shaving than Rabbit Hole Syndrome, but there are parallels. Like you, I used to be obsessive about details and solving subproblems. I used to come home from work and work on my own side projects for similar reasons as your own.
But then an advisor said something like, "You need to focus. If you want this thing of yours to succeed, you have to focus on making it succeed. Nothing else should matter." So I stopped my side projects and I became so effective at building our product that my employees wondered if I ever slept.
Stay on target. Make it to dinner.