News for hackers. Stuff that matters.
Re:Deployinator: Etsy's deploy tool open sourced by cdavid on Saturday July 30, 2011 @10:39PM (#2827446)
Continuous deployment sounds right to me in principle, but I have yet to see a system to handle rollback in the face of database handling (one slide mentioned this briefly). When you have a bug (say you delete the wrong data), how do you rollback this ?

The only solution I can see is one where databases are handled differently (they generally are anyway at the deployment stage), and IMO, this is the challenging issue. Continuously deploying app servers is not trivial, but I would consider it a mostly solved problem.

Re:Deployinator: Etsy's deploy tool open sourced by rmaccloy on Sunday July 31, 2011 @03:37AM (#2827868)
Soft-delete (+ action logs) for data recovery. Also, code review and lots and lots of tests.

For schema changes, there are a bunch of approaches. Just a few:

* Treat (non-purely additive) schema changes as a special case and take extra care deploying them. (This sounds like a cop-out, but it might be the right effort tradeoff for many startups.)

* Write forwards and reverse migrations for each change, and avoid data-losing schema changes.

* Implement all (or all non-purely additive) schema changes by shadowing data operations onto a new version of the schema (asynchronously migrate all previous data onto the new schema in some back-end process.) Write tests to verify the migration/shadowed data is correct. Switch data reads to be on the new schema. Eventually delete the old schema in a subsequent deploy after an appropriate stabilization/verification period.

I find #3 to be the most general, but there's plenty of overhead (computational, storage and man-hour) that may not be appropriate for your case.

Re:Deployinator: Etsy's deploy tool open sourced by jcrites on Sunday July 31, 2011 @03:18AM (#2827844)
A number of techniques help here. These are useful whether or not you're using continuous deployment.

A simple approach is not to delete data, but rather simply set a flag which marks the data as deleted. The application storage layer acts as if this data were deleted. A separate process (changed separately) is responsible for actually purging data which has remained deleted for a certain period of time.

In any case, do you really wish to delete the data? Keep it around -- it might be useful. The only reason to delete data, outside legal concerns, is to control the cost to retain it. Given that you mostly never really want to delete data, you probably want to orient your schema toward hiding it instead. In that case, if you need to roll back application changes, you can do so by backfilling the data to restore it to a visibile status.

Keeping detailed programmatic logs is also helpful for making the backfill easy to execute. For example, in addition to your main textual log file, you keep something like a log consisting of JSON objects, one per line, each which represent the essential details of some application action.

Re:Deployinator: Etsy's deploy tool open sourced by mcfunley on Sunday July 31, 2011 @10:41AM (#2828415)
You are correct, at Etsy we deploy schema changes once per week. The code always works against both versions of the schema. We never take downtime for schema changes.

We avoid data loss by doing soft deletes as much as we can. Sometimes we do want to do real deletes, especially for data that is heavily denormalized, but in those cases we keep an audit trail so that the data can be reconstituted in the event of a mistake.

Re:Deployinator: Etsy's deploy tool open sourced by Pewpewarrows on Sunday July 31, 2011 @07:49AM (#2828137)
The problem is pretty universally solved:

1. Use some sort of schema migration tool to keep track of and version all changes you've made to the database since your very first commit.

2. Never do a backwards-incompatible schema change. Never. Time and time again these are the ones that will completely bite you in the ass when the day comes that you broke production with a deployment and desperately need the old version back up and running. You're going to end up losing data and shit will hit the fan. It doesn't take much code-wise to support the old scheme, and it will ensure that you have zero downtime while rolling out your updates.

3. Never delete data. The only exception being if some user-submitted content violates a law, like child pornography. In which case, it shouldn't be part of your migration/update process to begin with. Just use deleted flags / boolean fields. Note in your code when certain fields are no longer used anymore because they're only there to support legacy versions.

Re:Deployinator: Etsy's deploy tool open sourced by cdavid on Sunday July 31, 2011 @12:29PM (#2828698)
I am pretty much a beginner in database handling, what would you suggest for 1 ?

Concerning 2: how do handle schema fixes (example: at my previous job, the tables were very badly designed, and the application+schema combination prone to frequent race conditions). What to do in this case ?

As for 3, I already came to this conclusion on my own, so I was not completely on the wrong track :)

Re:Deployinator: Etsy's deploy tool open sourced by quicksilver03 on Monday August 01, 2011 @05:48AM (#2831388)
I've used DbDeploy and Mybatis Migrations with good success, they're written in Java but they've also been handy in PHP projects.
Re:Deployinator: Etsy's deploy tool open sourced by jpluscplusm on Sunday July 31, 2011 @08:54AM (#2828223)
I totally get all these points; is there any literature I can point our devs towards to convince them (a) #2 + #3 don't leave any corner cases and (b) they really should take the time to do it (ultimately so I can put an automation solution around the the DB)?
Re:Deployinator: Etsy's deploy tool open sourced by macros on Sunday July 31, 2011 @07:46PM (#2830064)
I was at the talk and both of these were addressed. For rollback, the answer was they don't support it, only roll forward.

Databases are handled differently from this tool. Iirc he said Thursdays are database day and all changes are done/initiated then.

Re:Deployinator: Etsy's deploy tool open sourced by swanson on Saturday July 30, 2011 @11:52PM (#2827565)
While it doesn't handle every case, using a database migration framework with Up/Down style methods as part of your deploy process can work pretty effectively.

Another option is to use a VM for your database and do a snapshot before deploying.

Re:Deployinator: Etsy's deploy tool open sourced by andrewvc on Sunday July 31, 2011 @01:25AM (#2827691)
Well, yeah, but if you perform financial transactions, you can't just lose 5 minutes of history.

Also, for a DB, LVM's probably just as good as a full VM.

Re:Deployinator: Etsy's deploy tool open sourced by mathgladiator on Sunday July 31, 2011 @02:22AM (#2827778)
Chances are good that important tables do do transactions don't change as often as other things in the system. This is why I support things like CouchDB to handle the non-transaction stuff.
Re:Deployinator: Etsy's deploy tool open sourced by nknight on Sunday July 31, 2011 @02:49AM (#2827805)
Based on a very brief conversation I had with a Paypal engineer a few years back, it's SOP in modern financial transaction handling to store/log details of every transaction in many places and various forms, and there's plenty of (probably excessive) paranoia involved.

The context of the discussion was performance, scalability, and engineering resources, so we didn't get into how the logs are used, but it would be my guess that if the "primary" database had some sort of failure that lead to a rollback, affected accounts would be locked out until the transactions can be properly recovered from other storage/logging mechanisms.