News for hackers. Stuff that matters.
Re:Recurly billing down -- some customer data lost by sync on Wednesday September 05, 2012 @08:16AM (#4478966)

   On Monday at 3:30am PDT, we experienced
   a hardware failure in our primary encryption hardware device.
   The failure cascaded to the backup slave device as well.
How does a hardware failure cascade to a backup slave device?
Re:Recurly billing down -- some customer data lost by smoyer on Wednesday September 05, 2012 @08:37AM (#4479039)
In the same way a slave database can be corrupted by the master when it has a failure. If you're encrypting data on the master and replicating to the slave, improperly encrypted data may be propagated before a fail-over. This is a case where having the master fail completely and go off-line is preferable to a "working failure".
Re:Recurly billing down -- some customer data lost by awicklander on Wednesday September 05, 2012 @05:25PM (#4481543)
But doesn't a 'hardware failure' imply failure to a specific piece of equipment/hardware? A cascading issue with actual data isn't a hardware failure, it's a software failure.
Re:Recurly billing down -- some customer data lost by mike-cardwell on Wednesday September 05, 2012 @11:18AM (#4479723)
This is why I use incremental backups, and check them regularly.
Re:Recurly billing down -- some customer data lost by phillc on Wednesday September 05, 2012 @08:26AM (#4478989)
One of the racks fell over onto the other racks, like dominios.
Re:Recurly billing down -- some customer data lost by eckyptang on Wednesday September 05, 2012 @09:48AM (#4479309)
Ha - that made me laugh.

Ironically something not far off from that happened to a contract outfit I was working for. They had a telecoms rack rather than a proper one and they couldn't bolt 9x maxed out DL380G2 servers on it as it started to bend. They decided the best approach was to just pile them up on the rack bed.

This was until the one at the bottom needed a RAM upgrade. Cue 6 people carefully trying to lift up 8x DL380's in one go and sliding various OReilly books underneath. The inevitable happened - a hand gave way resulting in a pile of DL380s spread all over the machine room floor and a broken toe from someone in the hosting facility.

Nothing went down though!

Re:Recurly billing down -- some customer data lost by shimonamit on Wednesday September 05, 2012 @08:34AM (#4479024)
"This failure corrupted encryption keys used to access stored credit cards to process recurring transactions"

Looks like they were sharing encryption keys by replication or shared storage.

Re:Recurly billing down -- some customer data lost by alexkus on Wednesday September 05, 2012 @01:28PM (#4480363)
"Flailover"

The primary fails because of a bug and so it fails over to the backup which flails-over because of the exact same bug, c.f. https://en.wikipedia.org/wiki/Ariane_5_Flight_501