I hear ya. I knew that assertion was going to draw some criticism as its a judgement call about where we draw the line. Who's a novice and what's obvious? However I can't get away from my nagging impression that statistical validity is not inherently clear to the absolute best practitioners. Causality is the goal, and its notoriously difficult, even for world class minds. In my experience the only similar effervescent specter for software development is in security. Such circumstance,seem to me, to require great humility and introspection about ones abilities, but I suppose a little of that would go a long way in general too!
That depends "how" it breaks. As a novice coder myself, I've had things go wrong that I don't notice or can't identify, and it looks like my program is running fine.
I think that's the parent's point: it might be stupid to implement crappy macho learning models into production, but it isn't worrisome. It's expected.