I don't mean to attack you personally in any way, but if you have an awareness that such things as small powerful magnets exist and can imagine what the risks would be, how is it not your job as a parent to be on the lookout for that? You manage to be on the lookout for all sorts of other risks that exist in your household - drowning, fire, chemicals, sharp objects, and so on.
I do not believe that adding an awareness of magnets (which are generally pretty easy to identify because of their magnetic properties causing them to stick to metallic surfaces or each other) causes some unacceptably large burden. You've known what magnets are since you were a small child. You are almost certainly easily able to distinguish between a bunch of magnetic balls and a bunch of non-magnetic ball bearings by simply looking at them. How is it not your responsibility to use that knowledge?
I am glad you are a perfect parent, it is very lucky for you and your children. Congratulations!
I never claimed to be a perfect parent (nice attempt at misdirection there). But I don't try to hold other people responsible for the decisions I make. When people with kids visit me or leave their kids in my care then their safety is my primary responsibility and I'm willing to accept that.
So sure, he and I were both being negligent. But it also seems like a dangerous set up.