Kindle is the no-brainer solution here, if most of your books are in Epub. I don't have any experience with PDFs on it, though. The simplicity of using it and the possibility to send books via e-mail and get them on the reader in no time is great. Of course you can still use the usb cable to do it, if you don't want to have Amazon account/give them any info.

Plenty of older models should be on eBay. Paperwhite 3 and Paperwhite 4 might be a good start for a used (but still supported and receiving software updates!) cheap option.