News for hackers. Stuff that matters.
Re:We reduced the AWS costs of our streaming data pipeline by otterley on Thursday May 28, 2020 @02:36AM (#23334161)
First, you need to write the cronjob. But what goes in there? You need to decide exactly how you're going to make a backup, and the process may differ by what's being backed up. Ideally you want a quiescent snapshot, but the way you do that varies by application. What if the application is a distributed application, in which case you need to synchronize the snapshot process among all its nodes? What if it's a master-replica design, where the node that runs the cron job may vary based on the current topology?

And if you need some sort of cluster-aware lock to coordinate backups among different peers, you'll need to decide which system works for you, implement it, and maintain that as a separate system. And if that needs to be upgraded, figure out a bulletproof process for upgrading it while it's still being used as a coordinator.

Then, you need to ensure there's storage for the backup. You need to decide what kind of storage you're going to use, make sure you've got enough space, figure out how to encrypt the storage (very important in secure environments), how to protect the storage using authn/authz. And lots of environments have retention and storage lifecycle policies - you don't want to put the old backups on the expensive fast media; you want it on the cheap slow media. And some environments make you dispose of old data, so you have to figure out how to age it out but without ever losing the backups you want to keep.

Finally, you need to make sure the backups you create are valid and usable. So you'll want to build an automated regression testing procedure to ensure that every time you make a change (regardless of how minor) to the system being backed up or the backup process, that you end up with usable backups.

(Disclaimer: I work for AWS, but opinions expressed here are my own and not necessarily those of my employer.)

Re:We reduced the AWS costs of our streaming data pipeline by ec109685 on Thursday May 28, 2020 @04:13AM (#23334799)
You make it sound like there aren’t cookbooks for many of these scenarios and that the company will have to invent these scripts and procedures by hand.

Yes it is work, but this company’s whole reason for being is to save AWS spend, so I assume they have patterns they employ for their clients regularly that achieve their SLO.

Re:We reduced the AWS costs of our streaming data pipeline by sokoloff on Thursday May 28, 2020 @07:49AM (#23336182)
There are original-definition cookbooks and yet it still costs me time to provision my own lunch vs using the managed service of my corner restaurant.
Re:We reduced the AWS costs of our streaming data pipeline by ec109685 on Thursday May 28, 2020 @10:24AM (#23337617)
Yes, managed services are better in many cases.
Re:We reduced the AWS costs of our streaming data pipeline by otterley on Thursday May 28, 2020 @11:02AM (#23338110)
Sometimes there are cookbooks, but they are of varying quality and often don't have dedicated resources to maintain them, so I would use them with great caution. You also have to implement them and often maintain the underlying infrastructure.

But I was really responding to the brusque naiveté of the "just write a cronjob" response.

Re:We reduced the AWS costs of our streaming data pipeline by harikb on Thursday May 28, 2020 @03:32AM (#23334514)
“managed services” don’t necessarily save you the headache of making sure backups are usable. But other points are valid.