GDPR most certainly applies to institutions like the tax authority. As others have noted, they are required by law to keep some data/records.

But that doesn't mean you don't get to see all that data, can ask for corrections, or ask for deletions in the case where they store data they cannot justify.

How this apply (what difference it makes) will vary from institution to institution.