The location of the seller is irrelevant for calculating, collecting and paying European VAT. Its applicably to any company in any country in the world.
However, for now having a company outside the EU does make it more difficult to be tracked by EU tax authorities. But its likely that at some point in the future that will change. And tax authorities fine easily.
Some countries also have requirements as to invoice language, or conversion rates that you need to use for foreign currencies. In my case (Poland), I can use the European Central Bank (ECB) rates for VAT, but not for income tax further down the line, so I have to integrated with two central banks to get their rates.
Overall, there is lots of overhead. There is a disconnect between EU officials and small modern companies. Officials want to "promote innovation and entrepreneurship" while introducing heavy-handed legislation (really meant to hurt Amazon, in case of VAT MOSS).
Do not, under any circumstances, even think about running a SaaS business from Europe. EU has made invoicing hell for small businesses, especially those selling SaaS. You will spend time on things like VAT number checking, verifying customer's country using their IP (yes, I know), several kinds of invoices, prices inclusive or exclusive of VAT, calculating VAT for each EU country differently (and in different currencies), reporting VAT, and lots of other silly annoyances.
And if you think this is a "solved problem using a third-party service", you haven't tried it yet.
The only disadvantage of being US-based is that the first serious lawsuit will basicaly crater your business. Otherwise, you are way ahead of your buddies in Europe (like me).