But above that, the risk is that you have misestimated your level of "knowing what you're doing".
So, the sensible thing to do is make the cutoff at the point where the risk of that error is lowest. And that point is "is it a wild mushroom?"
Why? Because it's not that hard to figure out whether you're misestimating your level of knowledge if you actually do things to attempt that verification.
If you're not going to attempt that, yes -- you should stay away from virtually all things that have ever killed anyone, ever.
I know people who self-admittedly overcorrect and refrain from picking even a few easily-distinguished mushrooms that happen to be vaguely similar colors and shapes to deadly mushrooms, but the idea that you need to not pick any mushrooms at all, ever, in order to eliminate the risk is silly.
Picking wild mushrooms is not done (except by those who haven't done the trivial amount of info-gathering that is actually necessary) by gathering lots of different mushrooms and then identifying them and eating the ones that appear to be edible species. It isn't done by taking a field guide out and IDing mushrooms from scratch and then eating the ones that look like the edible ones in the book. It is done by learning species one or two at a time, along with all of their potential lookalikes. People who do this and start with easy mushrooms will encounter far more risk both from getting to the foraging area and from walking around in it than they will from the mushrooms.
Why is this even a question?