If the American system only works as a result of slavery then it should be discounted. If not, however, it should be understood and replicated where possible.
And we should all study history regardless.
This is the original quote. The issue is with "starting from scratch." You're not starting from scratch if you obtained the land through genocide, assisted by free labor. You are starting with quite a lot. America did not start in 1776. European powers had been investing in the new world for hundreds of years at that point. The whole issue is calling the economy of America at 1776 "starting from scratch." It's ahistorical.
The point Buffett and Munger were making is that the current American system works very well and creates enormous amounts of wealth. Doss the fact that there was slavery for centuries all over the world somehow refute that? I can't see how it does.
If you mean "what if time went on without Europeans in the US", that's a big what if. The indigenous peoples would have carried on in the Bronze Age, and would have been wiped out later by a different collection of foreigners.
The reality is that the US benefited from the culture and know how of Europe without the baggage of supporting useless aristocracy.
Its misleading to say that 99%+ of american wealth occured long after slavery etc ended, because it still started there.