Something I was taught awhile back relating to IT, “Data is toxic”. Get rid of it when you can, as often as you can.

You’re right. These companies should absolutely have to weigh the cost of keeping data because they overestimate their ability to keep it safe.

The most valuable data is just necessary to do their business though. Our financial and identity information are the things we hold most dear, but that's also exactly what we need to exchange in order to engage in commerce.

To a degree, yes, which is why it has to be secured properly, with state-of-the-art systems and engineering. I'd venture a guess to say this wasn't. But until we hold executives personally and retroactively responsible, nothing will change.

But must it be kept indefinitely?

Not indefinitely, but typicaly IRS or similar institution will require you to store all the transaction records for, say, 5 years. Which, from the purpose of hackers trying to steal your data is as good as "indefinitely".