He wasn't making a comment that the names don't matter - he was trying to rationalize why he gained an interest and appreciation for the deep-knowledge of science - not just how to look up reference material.
Names obviously matter - language and communication matter if we're going to collaborate on anything - but when it comes to science, to make new discoveries, the ability to look something up in a book is useless without the ability to understand how things actually work and fit together... and sometimes, books, and even the majority of people, are actually incorrect...
It's disingenuous to take that Feynman quote out of context and argue that it's an argument against naming... it's certainly not.
(Actually, probably London Plane trees, but don't let my ignorance get in the way of my point.)