Ask HN: How do big web sites roll out new versions?

Re:Ask HN: How do big web sites roll out new versions? by gaius on Saturday December 27, 2008 @03:22PM (#411305)

They'll take a few webservers and their associated middle tier boxes out of service on their front-end loadbalancers, wait for all the sessions to migrate/fail over to others, upgrade them and put them back into service. The loadbalancers will be smart enough to do affinity (put this customer onto this pool of servers if possible). At any one time after the roll-out begins, x% of the customer base will be on the new code where x->100 if things go well, or if something unexpected happens, (100-x)% of customers will never even see it, and the rest (x << 100) will see only a glitch before the loadbalancers swing them back across onto the old code.

In apps like this the policy is to only add columns to tables and new tables, never to remove anything, so the database can be upgraded hot, and the old code can continue to run on it, only the new code will see any new columns/tables.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by sh1mmer on Sunday December 28, 2008 @02:18AM (#411775)

I've done rolling deployments but also a similar but different way at Yahoo!.

We have access to multiple colos of slightly over-capacity machines. It's not uncommon to migrate by switching all traffic over to one colo. Then upgrade the next colo to the new code and switch all traffic there. Finally upgrade the first colo and then start cross loading the traffic between colos again.

Since we are big enough to have at least a couple of colos per geographic region this means we can avoid rolling migrations which are pretty horrible when you get trapped in a failure mid-roll.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by Timothee on Saturday December 27, 2008 @04:10PM (#411365)

And when all the servers run the new code, are the databases changed to remove old columns or is it an absolute no-no with ever-expanding tables?

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by seldo on Saturday December 27, 2008 @04:36PM (#411390)

At Yahoo we drop columns from the previous version the next time we upgrade the database, e.g. if you stopped using a 1.0 column in version 1.1, then you drop that column as part of the upgrade going to 1.2. The key thing is always being able to roll back if things go seriously wrong; hardware is much, much cheaper than downtime.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by bjclark on Saturday December 27, 2008 @04:23PM (#411377)

It's different for every site, db backend, etc.

If it's just 1 column, and it doesn't store much data, then probably no. If it's a bunch of columns and it's taking up lots of disk, they they probably migrate what they need to a new table and drop the old one. Altering tables is usually not a good idea. In mysql it can stop replication and can mean hours or days of downtime depending on the situation and amount of data.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by russell on Saturday December 27, 2008 @04:27PM (#411384)

Unless you really need the space, I suggest leaving the columns there. Fewer chances for error. If your columns are nullable, the amount of space taken up by an obsolete column tends to one byte, particularly if old data is deleted.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by brianm on Saturday December 27, 2008 @04:18PM (#411374)

Once you are sure you are never going to roll back to the old code you can wipe out the old columns.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by gaius on Saturday December 27, 2008 @04:35PM (#411389)

In practical terms you generally don't tho'. You might need them again, for example, for some as-yet-unplanned new feature. And doing so is of limited value anyway in many databases, you won't get the space back without a re-org and that's a costly operation to do hot, if you even can at all.

[ Parent | on HN ]

Re:Ask HN: How do big web sites roll out new versions? by jjs on Saturday December 27, 2008 @09:16PM (#411614)

If the upgrade is expected to change the user experience, this is also a great time to do some A/B testing across versions.

[ Parent | on HN ]