https://en.wikipedia.org/wiki/Schengen_Area#In-country_check...
It is the obligation of everyone travelling within the area to be able to show a fully valid form of personal identification approved by other Schengen states.[98]
and
Although travellers within the Schengen Area are no longer required to show documents at an internal border (although there have been some controversial instances when they have, and it is fairly common at major land border crossings with Switzerland), the laws of most countries still require them to carry identity documents
Your interpretation of this doesn't seem to be correct. The whole point of Schengen is that you don't have to show ID at internal borders, which were supposed to cease existence. Rather, you have to be able to show your ID at any time whilst travelling ... where "travelling" is such a general concept that it effectively means at all times unless you are e.g. at home.
Additionally, it could just have been chance.
In general, yes, you should have your ID/passport on you, but there is no real punishment for not having it other than being required to later show up at a police station and/or being escorted by the policeman to your apartment/hotel to show your ID.
A bigger problem is usually not having a drivers license on you, that gets some of the people in the police riled up (but again, no real punishment as long as you show up later at the police station)
Why? What about people that just don't have / need a drivers license?
But if you actually have to carry around your ID all the time depends on the laws country you're in. Here in Germany you don't.