I imagine growing spores with the knowledge that you can grow mushrooms that can be used for psychedelics from them is similarly illegal to buying/growing poppy flowers knowing that they can be used for opium.

It's only legal if you don't know -- a thought crime

Hard drugs and research chemicals are ordered and sent daily via mail, and the only people getting vists are the vendors. I highly doubt anyone cares enough about john doe ordering some spores to put him on a list or show up at his home.

I am in a conservative EU country and you can legally buy spores and growkits since forever here, nobody cares. You have to find specialized shops because there is not even enough demand.

That's what vendors will tell you but customers get visits from customs all the time. And in that unfortunate case you'll get charged with contraband plus everything they'll get on you on top of that. You probably won't have proper opsec and your house will be searched and electronics will be confiscated, plus there will be some surveillance prior to that. I'm not talking specifically about spores here, more about general onion shopping. Its easy to get in the mindset that nobody cares about a user but if you get unlucky your life will take quite a hit.

What country are you living in that has enough money to surveill average joe and kick in his door because he orders some small quantity of drugs? If you want to get dozents of users on the hook for possesion you could just raid a random nightclub, that's a thing that actually happens - but even that is rare.