The rulebook for chess and go is zillions of published books on pattern matching and book openings and book responses. To play at a high level you have to memorize and perfectly apply uncountable zillions of rules and patterns or you'll just be destroyed by someone who's memorized more rules. If you don't know all the chess book openings and you play against someone who does know them, you may as well just save everyone some time and immediately resign. Much like MtG, how much time and money are you willing to spend to win? This turned me off from the whole genre of "memorize and pattern match" games, I just don't find that to be recreational.

The pinnacle of large rulesets is RPG games, anyway. With the possible exception of some 80s classics like squad leader and starfleet battles there are no rulebooks that compare in size to the foot or so linear shelf space of a full set of Pathfinder rulebooks.

Strategy/tactics != rules. The rules for chess and go are simple. The gameplaying that stems from those simple rules is obviously quite complex--and, as you say, it has been deeply studied so anyone wanting to play even somewhat competently needs to have a passing understanding of quite a bit. But the rules themselves (especially with go) are quite straightforward compared to most modern adult boardgames.

So what is it you're really asking for? A game where even a beginner can beat a long-time fan, yet one that is interesting? I think those goals are conflicting unless we're talking pure gambling.

A game that was basically an IQ test might work as well. Like having to solve those abstract pattern matching or 3D-to-2D geometry questions. Maybe. I'd guess there's some slight advantage to having answered many of such questions, but I doubt it is nearly as big a difference as, say, studying Chess or Go.

Do you find the replayability of smallworld is largely due to the randomisation of which (race, attribute) pairs are available per game?

Arguably Dominion has good replayability for similar reasons: you only play with a subset of the cards every game, and a lot of the strategy revolves around exploiting interactions between different cards. On the other hand Dominion is one of those games where there is very limited interaction between players, which perhaps doesn't make for great replayability.

Randomization definitely helps.

I also play Hearthstone, the Blizzard card game, and that's a pretty awful example, sometimes, of bad randomization with all those effect that can range from harmful to do nothing to win the game on the spot.

The randomization in Smallworld is good because you always have a certain amount of control over it and you can plan for it/around it. On top of that the power swings are not really extreme (some race - attribute combos are stronger than others but the overall power range is reasonably small).

The interaction between players in Dominion is botg subtle and powerful at the high level.

It is all about pacing an the impression of pace. In some regards it is like Cycling sprint discipline.

Watching top level Dominion games can be like watchig a different game. It has crazy, crazy depth and replayability.

I love Smallworld, and yeah, I never get tired of it. Another one whose rules are pretty straightforward is Power Grid, been playing it for years and it hasn't gotten old yet (except for the sometimes pretty sudden and ruthless endgame).

Two-player games are much easier to balance and naturally develop interesting counterplay mechanics.

I found Smallworld unfun.