Whenever anyone asks me about ways to improve the performance of their web frontend my first question is always "What do you actually use the data you get from Google Analytics for?" 9 times in 10 the answer is "To see how many people visit my website." My next remark is usually something about vanity metrics and how conversions are what actually count, but that's because I'm a mean person. The point is that adding in Google Tag Manager for the use case of 'counting' is absolutely killing your website's perf. You can, and should, collect that information from the server logs with no frontend JS at all.

If you need more than just a counter then MetricsWave or Plausible or something is better than Google Analytics. The only reason to use Google Analytics is if you need to integrate it with other Google services like ad tracking.