"When we moved our Github repository from private to public (as the co-founder was leaving and I had to share it with some other prospective co-founders), the AWS keys were all in the open for everyone under the sky to play with. Doing some online search, I figured out we were not the only ones. It’s apparently quite a common mistake developers do."

Yes, Yes and Yes. I have made a similar mistake once on Github for a personal project, I realized my mistake and immediately updated the file and did a new commit. Guess what ? Even after committing a new version of the file, you can always check the "History" of the file which will show you the version with the credentials. My only option was to delete that config file completely.

Having said this, it can be scary if you leave your credentials on sites like Github. I work a lot with Python Flask framework and oen of the commonly used extension is Flask-Mail to send emails. Guess what ? A lot of developers are leaving their email credentials in the open and being a flask dev, I know that most of the time, it is MAIL_PASSWORD variable usually in a config file. Knowing this, I can just do a code search on github with keyword MAIL_PASSWORD. You figure out the rest.