The term is quiet loose.
And I'd agree, that category does generally seem to be an issue. Though, I have to wonder if it's more a problem that that category typically has foods with an absurd amount of calories that are easy to consume.
For instance, pure peanut butter is unprocessed. While it runs through a mechanism to change it from peanuts to peanut butter, there is no oil added, nor are any of the healthful nutrients of peanut butter extracted. Opposite to this is peanut butter like JIF which is processed - the mix is diluted with sugar and vegetable oil to make the same amount of food for cheaper.
Same with tofu - it starts as soybeans and is ran through a mechanism to turn it into the blocks of tofu we see in stores, but we do not add sugar or oils to change the contents of that block, nor do we remove nutrients from the soybean (this may not be 100% accurate, but generally speaking, nutrients are not removed in this). However, many vegan meat products put a lot of unhealthy additions into the mix, thus making it processed.
When people use the term "processed foods" colloquially they are generally referring to NOVA group 4, ultra-processed foods. It seems that that group is implicated in worse health outcomes and it is also that group that has proliferated the most significantly over the past 40 years in tandem with rising rates of obesity and poor health outcomes.