On the one hand, that's how I feel about literally everything about myself. There is something wrong with my entire body. I do only use it because I have no better choice. To quote Eliezer Yudkowsky, "I'm sure Einstein was very smart for a human. I'm sure a General Systems Vehicle would think that was very cute of him." Like I said, I'm all aboard the transhumanism train.
On the other hand, you're right that most people don't have that intuition and that "prosthesis" may not be effective communication. This particular issue disadvantages me compared to other humans and solutions to it help me recover basic functionality, but you're right that this technology is good enough that a run-of-the-mill human can benefit from it too. They just don't benefit as much as I do. So, hm, maybe I should switch to augmentative language over assistive. I'll think about it.
I'm sorry, I didn't notice until you clarified that you suffer from actual memory issues; I assumed you were just using hyperbole as a rhetorical device. With the context clear, I see why you would want to refer to those tools as prostheses, given that they lift you to (and hopefully above) the "normal" values. And as you say, they indeed perform both augmentative and assistive functions.
I choose to describe them as "external brain" ("exobrain" is the term of the art now?) - "prosthesis" feels like it implies there's something wrong with it, and that you use it only because you have no other choice.