I recently replaced the battery in my Kindle Keyboard (I love physical buttons). It was easier to replace than I expected, and after half a year of daily use, haven't noticed any issues with the battery.

There are probably at least a few good places to buy replacement batteries; I got mine from batteryship dot com. I have zero association with them except that I bought this one battery, and will probably buy a phone battery from them in the future.