We reduced the AWS costs of our streaming data pipeline

Re:We reduced the AWS costs of our streaming data pipeline by owenmarshall on Thursday May 28, 2020 @08:15AM (#23336351)

I once worked at an IT shop that worked closely with the construction industry. A new sports stadium was being built and we were doing panoramic photos during each stage of construction, and rendering them in a web app where the facilities team could “peel back the layers” and see what was behind the wall or under the floor all the way down to the foundation. This was... 15 years ago? So it was pretty neat technology and a little less ubiquitous than today.

Well, our storage server barfed and the data was gone. Went to restore from backups, all the hourly tar files were there... but were zero bytes.

We looked at the backup script the engineer had put together and it was one of those classic “didn’t give the right parameter to have tar recurse” type bugs. Unfortunately we lost all the photos of the foundation and much of the photos of the electric being run. Oops.

[ Parent | on HN ]

Re:We reduced the AWS costs of our streaming data pipeline by jbn on Thursday May 28, 2020 @11:38AM (#23338564)

One wonders why tar even has an option (and a default!) to not recurse. What would be the use case for that?

The "normal" use case seems to have a recursive archival. Sounds like somebody chose the wrong default... would be an interesting software archaelogy project to figure out where this "feature" originated.

[ Parent | on HN ]