I belive this is what was attemped in France (they taught set teory in an abstract way in elementary school), but with poor results and abandoned this approach after some time.

> Has it even ever been tried out?

Of course, and it's standard practice in some places.

P.S. I'm talking about extracurricular math clubs/circles. School curriculum is a separate thing that's divorced from actual math.

Math club for set theory before 1st grade and numbers? Do you have a link?