Right, I'm saying an outsider doesn't have an experience of the user and so no grounds for any particular insight about them and their use and its effects.

Their family and friends however are not outsiders. They have hard facts about how the use affects them, which is often some of the most significant negative consequences of drug use. I don't think any single person is situated to confidently say that someone else's use isn't a problem. But in cases where it is a problem, family often have all the information necessary to know that it is.