Spells and classes names cannot be used if they are trademarked, right? But if WotC didn't trademark "fireball", "paladin" or "hobgoblin", you can totally use them.
My understanding is that this is where things get tricky: realistically WotC probably can't go after anyone that creates an ability that is mechanically identical to DnD ability. Things like spell names, might be, but "fireball" is far too general to be copyrighted. But what about a spell called "fireball" that also has the same mechanics? That's where things can get complicated.
So what if they just doubled all HP values and damage values, and changed any names that are not unambiguously generic ("invisibility", "fireball", etc.) ? Damage values no longer overlap, and spell names are different. On what basis does WotC have to claim infringement?
I think you can. I don't think copyright covers gameplay mechanisms, it just covers the actual text of the rules. But IANAL.
The problem is making a whole spellbook with a one-to-one correspondence to WoTC's spellbook, even if each of those entries has been modified. The spellbook is more than just a bunch of separate spells - it's a collection carefully designed for balanced gameplay and has protection in itself. See [1] for one way that "compilations" can be treated as having more rights.
The EU also allows copyright of databases [2], and I think the spell list meets the criteria.
And don't forget you'd also be doing the same thing with races, classes, equipment, monsters, conditions, deities, feats, blah blah blah. Put together I think it's pretty likely to be infringing somewhere! Certainly enough that a lawsuit isn't getting dismissed.
[1] https://www.copyright.gov/circs/circ14.pdf [2] https://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CELEX...
That's what the person you're replying to wrote.
The CEO accidentally CC'ed me (he meant to CC Sean, not Shawn) on a letter from Valve's lawyer threatening us with legal action for copying Beastmaster's mechanics. They had a side-by-side of ours (https://hon.fandom.com/wiki/Tundra) vs theirs (https://dota2.fandom.com/wiki/Beastmaster), and pointed out that the mechanics were identical.
It was very surprising. So the answer is "Maybe not, but that won't stop companies from threatening to sue you." Or "Maybe it is, and companies will threaten to sue you." No one really knows, since it hasn't been tested in court.
It's the specifics of D&D's implementation where you need a license. You'd certainly need to develop your own feats, spells (and not just change the names!). You'd need new classes, or at least major changes (no re-using the level progressions or traits).
You'd need to play test the new variants because changes would be large enough to require new balncing.
All in all not a trivial modification.