The question is: are they any more likely to gain a seed that I keep only in my head vs. a list of passwords in an encrypted datastore?
In the case of your idea, once an attacker knows the seed, they have everything. Furthermore, if there's some weakness in the algorithm you use to generate passwords from the seed, then some site which has one of your deterministically-generated passwords could potentially breach the weak algorithm to reverse the process and obtain your seed, along with all your other passwords.
Lastly, with a traditional password manager, you can change the master password at any time. Would you be able to change your seed when needed, or would you need to regenerate your passwords for EVERY site, if you need change the seed because it's been compromised?
I mean, if that were my main concerns, I'd have to rethink most of the bedrock security precautions I take, like passphrases for private SSH keys, etc.
At the end of the day: I understand the risks and view them as absolutely acceptable in order to gain reasonable portability in a password manager.