I mean, we use rocks that are millions of years old in our own structures, why wouldn't it be likely that an early human saw some wood and thought that it'd be a strong building material without knowing or considering the age of that wood, or something to that effect?

Just that wood is organic material and suffers faster degradation vs. rocks when exposed to weather, which is implied in "if you can just find it". Mind you, they were digging up material in the old times already to find the best stuff for making their tools, but they were for sure not looking for old wood in that manner.

Plus, well preserved old wood is much harder than its fresh counterparts (better tooling / more time for processing required) and a rare find on top of that, so unlikely to be used for standard utility structures.