someone wrote a script that trawled through the site and
obtained the information for thousands of registrants
Pro Tip for avoiding getting sued when you report a vulnerability: Tell them you made two accounts and managed to access one from the other, so you've proved the vulnerability without seeing any other customers' data.If you download more private data than you need, you're liable to activate someone's oh-shit-is-he-blackmailing-us sense, sending them scrambling to defend themselves.
Financial companies are unlikely to be receptive to unsolicited penetration testing and more likely to come down hard because of a need to demonstrate compliance, the same goes for any healthcare related sites. There's a reason you rarely see them popping up on lists of bug bounty programmes.
It kind of is, kind of isn't. Security approaches I have witnessed were always very cargo-culty. People were aware of the OWASP top 10 and the like and applied best practice recommendations they have read somewhere and were even written into internal project guidelines. But security was not applied holistically.
For example passwords are passed off to a library that used scrypt properly. But hand-rolled session management used regular string comparison for the token, which obviously is unsafe. Similarly parameterized SQL queries were used instead string processing because that's the rule in the project documentation. But there were no rules about lucene queries and boom some devs started to do string processing on the queries again. And every time I have to fight to get sane password policies. In one case they wanted to block copy&paste events in the login forms "for security reasons", but that would have made things more difficult for people who actually use password managers.
Of course not everyone is like that, it just that the 1-2 people who would know to spot such things are often tied up in organizational overhead and then things slip through reviews.
Happens at the top of the chain too, I remember back in 1999/2000 when the Australian government released their website to enable online registration of businesses for in new GST (Goods & Services Tax), it was discovered within a day that anyone could simple trawl through the site and increment the ID at the end of the URL to view the business (and bank) details of every single one of the thousands of business that had registered.
In fact, IIRC, someone wrote a script that trawled through the site and obtained the information for thousands of registrants and put it into a spreadsheet which he emailed to the Australian Tax Office as his evidence. I believe instead of being rewarded, he was reprimanded and warned for doing do.