It’s not really that humans have infinite context windows, it’s more that the context windows are a very poor substitutes for long term memory.

Memory even in a text heavy field like programming is not only text based so it’s often hard to describe for example an appropriate amount of error checking in prompt.md. Giving a person with anterograde amnesia a book of everything they know - no matter how well indexed or how searchable will not fix the lack of long term memory.