and would rather buy from a copyright infringing publisher, that allows you to keep the illegal copy of your book?

I don't understand What do you think Amazon should have done here?

It recieved a book copyright infringing, it removed the book from it's devices, and refunded the customers.

Amazon may have done pretty spooky stuff, kindle books are often overpriced. But you making a opinion that Kindle devices are clearly bad, just from a isolated incident related to their store seems pretty short sighted.

This is equivalent of saying, "Facebook removed my post for copyright infringement, I will never use the internet".

Facebook can do whatever they want with the data they store on their servers, because it's their service and their data.

The fact that Amazon could reach into my device and mess with the contents of my library means it's not actually my device, it's Amazon's device and Amazon's library. I'm not going to pay for a device which gives me less control over my library than I'd have if I bought actual paper books.

Why is it any of Amazon's business which books I have in my library, or what their copyright status happens to be?

Yeah, you can argue it's not your Library, it's Amazon's library. Cause technically, you don't own the book you are using on your kindle, you're just leasing it's content for viewing. If you did put the book there yourself (obtained externally, say a .mobi file), Amazon would not be able to take it away. You own the device hardware, you own whatever you put on it. But it would be unfair to the author if you got to keep something that Amazon "stole" from them, even if you bought it from amazon.

But this would be a issue with any content with the current IP laws. And isn't anything wrong with the Kindle itself. Imagine if you bought a book, photocopied it, and returned it. This is what letting you keep the book will be equivalent to.