As for actual experience, I had a great time in Buenos Aires. You get 90 days visitor visa with a US passport and there is a good size population of ex-pats that take a weekend trip to Uruguay when they get close to 90 days, turn around and get another 90. A bunch of them make some cash teaching English and keep it all under the table, which I don't really recommend (I mean they still get you on VAT, but you don't want to mess with any laws in a foreign country if you can help it), but it isn't a bad place to be.
You will want to be very, very careful with that. I'm neither a lawyer nor an accountant, but the tax rules for US citizens are very different from pretty much everyone else. Tax authorities will generally not be happy if you try to pay no tax locally and send it all home. The rule is typically that you pay tax where you spend the most number of days of that tax year. No idea what happens when tax years for natural persons don't overlap (UK has April-April, many countries have calendar years) - in my case I didn't earn any money in the interim so it wasn't an issue.
Within the EU, you'll probably need to register your new residence as the, or one of the company's offices where its business is conducted. You don't need to register it as a business of that country, though, unless you're a sole trader. Any salary or dividends you pay yourself will usually be subject to local taxes, social security, health insurance and possibly other fees. Your accountant(s) will need to tell you where the business pays tax on its profits.
You may also require a permit for operating a business in that country. I did (UK citizen, located in Austria), and although software development is usually not regulated in the way the medical, legal and other professions are, I did require a certificate of any record on the UK Police National Computer, or rather lack of any record. This took almost 2 months and cost 10 pounds. I managed to sweet-talk the bureaucrat in Austria into not requiring an official translation, but the rule is that you do - this will add extra cost and time.
You may also require a residence permit, even within the EU. That too tends to be a mere formality, but usually they want proof of earning enough to sustain yourself. Not a problem if you're a UK limited company, you'll be used to your income being a matter of public record anyway. I didn't need this in Vienna, but a German friend in another State (Lower Austria) got an unpleasant visit from the immigration police.
UK->EU specific, but: you can forget your E111 for anything but emergencies. You'll probably be forced to get local health insurance (in Austria they write to you and bill you as soon as you register the business) anyway.
Hopefully, you'll encounter less hassle - Austria is exceptionally hostile towards the self employed, and taxes are insane.
You're right. You probably want to find a good tax attorney or accountant that is well-versed in the tax laws and treaties of any and all countries you plan on spending time in, especially if you plan on doing business locally.
Indeed many EU startups are officially headquartered in the UK even if they're based elsewhere in Europe for the favourable legal and tax implications.
[1] exception: if you actually take payment in that location, e.g. selling stuff at a trade show, you need to register the business at the location you're visiting for that time as you are now operating from that country for VAT purposes. Ugh! VAT is a whole new can of worms anyway.
But really, you should consult an accountant or tax advisor for hard information after you have decided on a country to go to and for how long you intend to be there.
As a rule, < 3 months -> no problem, > 6 months = residence, in between, gray area, leaning towards 'extended holiday'.
Currency exchange: rates fluctuate on a regular basis and that makes budgeting a challenge. Sometimes you win and sometimes you lose depending on how the currencies move.
Insurance: health insurance by a provider that is outside your country of residence is going to result in some hurdles. You'll have to pay close attention to what your providers rules are for coverage outside of your country.
Visas: as long as you only stay long enough to stay qualified as a visitor then visas are probably going to be a non-issue. Once you exceed the amount of time allowed as a visitor then you have to deal with that countries rules for visas and they differ from country to country.
Cash, check and credit: acceptance of credit cards and checks in foreign countries will very. Banks tend to charge fees for withdrawing money in other countries so be prepared to eat those fees. Often it is easiest to just use cash, so take enough to get started and have a plan on how to get more while you're there, i.e. figure out where the cash machines are, if they accept your cards and what the fees are.
Tax: IANAL nor am I an accountant or tax specialist and I am not sure how it works in the UK. Having said that, the US is a bit odd in that they tax their citizens worldwide, so no matter where I live I will have to pay US taxes. If I am a resident in another country then I'll likely have to pay taxes there as well and then try to get a credit in the US for taxes paid overseas. It's still early for me on this one so I'm not sure exactly how taxes will pay out.
I hope this helps. Basically my suggestion is retain your UK residence and take extended "vacations" in countries you want to visit - it'll be a lot less of a hassle then trying to immigrate.