Not only wrong, but dangerous, because nails are often used to fasten structural elements of houses, and incorrect but hard-to-detect flaws like this could result in collapse.

Similarly, if ChatGPT gives you an answer high in truthiness but low in accuracy, it could negatively impact you, whether loss in credibility if you repeat nonsense in front of someone knowledgable, or even worse if you use the incorrect knowledge to try to solve a real world problem.