Maybe not with today's technology, but in the future, they will still have the recording and will have the technology.
It's the same thing with encryption. It's safe today, but in the future it'll be decodeable, and they'll still have the encrypted copies to work on.
They can eventually know who I am by voice recognition, especially they can get corroboration by the fact that we live in the same building.
I imagine we probably said something that would suggest I live nearby (... "Checking database" ...) or that I'm a regular visitor (... "Checking database." ...).
They could be sending an inaudible tone, to be captured by an app on my phone. I don't have an Amazon app, but some other app might capture the tone and sell that fact to Amazon, so that they don't leave money on the table.