Cavemen are a trope, for sure, but there were cave dwellers, known as troglodytes. There were extensive periods of time where caves represented a really good solution to environmental and climate changes over the last 300,000 years.

Troglodytes were relatively rare, and most prehistoric humans were probably of the nomadic or small village types of hunter gatherer cultures.

https://www.smithsonianmag.com/smart-news/ancient-humans-wea... - the toba catastrophe was one such event that likely forced humans to live in caves extensively.

https://en.m.wikipedia.org/wiki/Pinnacle_Point

https://en.m.wikipedia.org/wiki/Blombos_Cave

There are plenty of caves that show evidence of long-term use of fire and burials, a good indication of habitation. However, it's true that caves were never a primary mode of habitation, just one that was convenient when it could be found... and pretty much the only one that we can see now due to differential preservation.