News for hackers. Stuff that matters.
Re:Ask HN: What's the worst you've ever screwed up at work? by patio11 on Wednesday January 29, 2014 @03:03PM (#7146384)
I've only cried literal tears once in the last ten years, over business. Due to inattention while coding during an apartment move, I pushed a change to Appointment Reminder which was poorly considered. It didn't cause any immediate problems and passed my test suites, but the upshot is it was a time bomb that would inevitably bring down the site's queue worker processes and keep them down.

Lesson #1: Don't code when you're distracted.

Some hours later, the problem manifested. The queue workers came down, and AR (which is totally dependent on them for its core functionality) immediately stopped doing the thing customers pay me money to do. My monitoring system picked up on this and attempted to call me -- which would have worked great, except my cell phone was in a box that wasn't unpacked yet.

Lesson #2a: If you're running something mission critical, and your only way to recover from failure means you have to wake up when the phone rings, make sure that phone stays on and by you.

Later that evening I felt a feeling of vague unease about my change earlier and checked my email from my iPad. My inbox was full of furious customers who were observing, correctly, that I was 8 hours into an outage. Oh dear. I ssh'ed in from the iPad, reverted my last commit, and restarted the queue workers. Queues quickly went down to zero. Problem solved right?

Lesson #3: If at all possible, avoid having to resolve problems when exhausted/distracted. If you absolutely must do it, spend ten extra minutes to make sure you actually understand what went wrong, what your recovery plan is, and how that recovery plan will interact with what went wrong first.

AR didn't use idempotent queues (Lesson #4: Always use idempotent queues), so during the outage, every 5 minutes on a cron job every person who was supposed to be contacted that day got one reminder added to the queue. Fortuitously, AR didn't have all that many customers at the time, so only 15 or so people were affected. Less than fortuitously, those 15 folks had 10 to 100 messages queued, each. As soon as I pressed queues.restart() AR delivered all of those phone calls, text messages, and emails. At once.

Very few residential phone systems or cell phones respond in a customer-pleasing manner to 40 simultaneous telephone calls. It was a total DDOS on my customers' customers.

I got that news at 3 AM in the morning Japan time, at my new apartment, which didn't have Internet sufficient to run my laptop and development environment to see e.g. whose phones I had just blown up. Ogaki has neither Internet cafes nor taxis available at 3 AM in the morning. As a result, I had to put my laptop in a bag and walk across town, in the freezing rain, to get back to my old apartment, which still had a working Internet connection.

By the time I had completed the walk of shame I was drenched, miserable, and had magnified the likely impact that this had on customers' customers in my own mind. Then I got to my old apartment and checked email. The first one was, as you might expect, rather irate. And I just lost it. Broke down in tears. Cried for a good ten minutes. Called my father to explain what had happened, because I knew that I had to start making apology calls and wasn't sure prior to talking to him that I'd be able to do it without my voice breaking.

The end result? Lost two customers, regained one because he was impressed by my apology. The end users were mostly satisfied with my apologies. (It took me about two hours on the phone, as many of them had turned off their phones when they blew up.)

You'd need a magnifying glass to detect it ever happened, looking on any chart of interest to me. The software got modestly better after I spent a solid two weeks on improved fault tolerance and monitoring.

Lesson the last: It's just a job/business. The bad days are usually a lot less important in hindsight than they seem in the moment.

Re:Ask HN: What's the worst you've ever screwed up at work? by d0m on Wednesday January 29, 2014 @03:52PM (#7146786)
Can't agree more with Patrick, mine is formulated a bit differently:

Lesson #1: Don't push code on Friday afternoon.

Lesson #2: Beer, Code and Commit is totally fine. Just don't push! Wait until next day to review and push/deploy it..

Re:Ask HN: What's the worst you've ever screwed up at work? by jwdunne on Wednesday January 29, 2014 @04:37PM (#7147140)
Lesson 1 rings a mad bell. I distinctively remember a text from a colleague saying "yeah, I'm never launching anything on a Friday".

As a general rule, the best time to launch is first thing on a Tuesday. Why Tuesday morning? We are also well aware of "Monday morning madness" too - pre-planning stress for Monday morning is akin to masochism.

Re:Ask HN: What's the worst you've ever screwed up at work? by alexcroox on Wednesday January 29, 2014 @06:28PM (#7147907)
Agreed, we have a strict no deployments on Fridays or after 3pm on any day at work. Allows us to go home with a clear head and enjoy our weekends.
Re:Ask HN: What's the worst you've ever screwed up at work? by malka on Wednesday January 29, 2014 @06:51PM (#7148048)
Never really understood person that do otherwise. What's the point ? ruining a perfectly fine night / week end ?
Re:Ask HN: What's the worst you've ever screwed up at work? by dustyleary on Wednesday January 29, 2014 @09:33PM (#7148740)
For essentially all companies whose customers are paying consumers:

The technical staff all talk about the dangers of Friday evening deployments.

The company has ignored/overridden these warnings in the past, and sometimes experienced catastrophic failures because of it.

STILL, sometimes the marketing staff will have weekend promotions that are considered more important than these concerns.

In particular, a weekend promotion will be implemented, tested on Wednesday, and ready to go. And only after it's active, is a bug found in it on Friday at 5pm. And the promo is more important than the risk of failure.

I'm not saying this as a "bitter techie"... I'm just explaining that this is how it happens.

I think that the "it's more important" argument is often correct. Not running the promo costs the company $X of opportunity cost, and X is often very well known, from past promotions. Balance "losing $X" against "well, there's only a small chance of something going wrong", and that's why the risk is often taken.

Take the risk enough times, and eventually something goes wrong.

Re:Ask HN: What's the worst you've ever screwed up at work? by steveklabnik on Wednesday January 29, 2014 @05:13PM (#7147455)
I wonder if this is the rationale for Patch Tuesday.
Re:Ask HN: What's the worst you've ever screwed up at work? by mdisraeli on Wednesday January 29, 2014 @08:09PM (#7148415)
Pretty much. It's arbitrary, but is fixed, not the first week or last week of the month (often reporting periods), not Monday, and not the weekend. If you then assume that organisations test the patches, that means they can deploy them for thursday morning
Re:Ask HN: What's the worst you've ever screwed up at work? by alttab on Wednesday January 29, 2014 @06:29PM (#7147914)
No the rationale for patch Tuesday is shoddy development while continually introducing bugs with your fixes.
Re:Ask HN: What's the worst you've ever screwed up at work? by pkinsky on Wednesday January 29, 2014 @05:11PM (#7147438)
I work for a Forex (have USD? Want JPY?) service provider that runs a 24-5 service (At any time of day during a weekday an exchange will be open somewhere, on some continent. They all close on weekends)

We exclusively push code on Fridays.

Re:Ask HN: What's the worst you've ever screwed up at work? by mdisraeli on Wednesday January 29, 2014 @08:07PM (#7148403)
I've worked with a global services industry organisation based in the UK but with MAJOR work happening in the middle east, India and Asia-Pacific regions. To add to that, because so much of the service industry's core activities tend to be based around the weekend and start of the week, we can't simply push at the weekend. Add in regular scheduled downtime and backup periods, having to work with people in the US, and many, many dozens of systems...
Re:Ask HN: What's the worst you've ever screwed up at work? by nextweek2 on Wednesday January 29, 2014 @04:00PM (#7146842)
This is the difference between a software developer and a senior software developer. I never release anything on a Friday OR the day before a holiday. ALWAYS plan it afterwards, give any technical reason you like (solar flares). Project managers will realise and thank you for it in the long run.
Re:Ask HN: What's the worst you've ever screwed up at work? by dllthomas on Wednesday January 29, 2014 @04:33PM (#7147110)
This is definitely something I value about working in finance. Exchanges (the ones we're caring about anyway) are all closed over the weekends.
Re:Ask HN: What's the worst you've ever screwed up at work? by hvass on Wednesday January 29, 2014 @07:54PM (#7148347)
There is a great lesson in one of pg's essays. If you push you better be there for few hours to monitor what is happening. You do not just push and leave. I hope we all learn from this thread.
Re:Ask HN: What's the worst you've ever screwed up at work? by mdisraeli on Wednesday January 29, 2014 @08:02PM (#7148379)
In the IT services industry, we'll often talk about running a "war room" for the hours following a significant change, and then follow with a period of a nominated on-call person who has the authority to wake the project team up, tell them to drop everything and fix the issue.

Funnily enough, change control actually runs a lot more smoothly in terms of getting past senior non-technical managers when you include elements like that

Re:Ask HN: What's the worst you've ever screwed up at work? by aragot on Wednesday January 29, 2014 @10:08PM (#7148835)
It sounds like you don't do continuous deployment. It divides the risk and improves agility. It pushes "change" to a non-management level.
Re:Ask HN: What's the worst you've ever screwed up at work? by mdisraeli on Thursday January 30, 2014 @09:42AM (#7150855)
"continuous deployment" is not a panacea to the issue. There's two primary ways of doing it, from what I understand:

continuous deployment to a dev/test environment is the easiest for most organisations to move to. Due to the live environment being mission-critical, they can't afford the risk of any degradation of service. So you push regularly (after passing test suites) to a test environment, get a small subset of users working, and at some point then push out to live from that. But I suspect that isn't what you are referring to, as this is too similar to typical change management.

the alternative definition of continuous deployment is that of constantly pushing to live, initially for a subset of users then rolling it out from there gradually, but always on the live environment. In many large, often 'cloud-y' solutions, that subset might be all users. Except you can't have a public transport ticketing system fail at peak times. A hospital patient record system must stay available for staff, and give sufficient notice before any possible impact to service to allow manual processes to be used. Payroll, accounts, HR systems... all of these have failure modes that have financial penalties at best.

Hence continuous deployment only makes sense when you can afford to risk service, possibly with significant impact.

Re:Ask HN: What's the worst you've ever screwed up at work? by collyw on Friday January 31, 2014 @10:17AM (#7157043)
It helps a lot though. One of many small incremental changes, or one big monster change, which do you think is likely to break a system? And which is going to be easiest to diagnose and fix?
Re:Ask HN: What's the worst you've ever screwed up at work? by jonalmeida on Wednesday January 29, 2014 @05:00PM (#7147348)
I learned the first lesson the long way. To make myself feel better, my excuse is that it was my first job and I can in on the weekend to fix it.
Re:Ask HN: What's the worst you've ever screwed up at work? by erobbins on Wednesday January 29, 2014 @07:24PM (#7148205)
at my last job we had an unofficial slogan of "don't fuck with it friday"
Re:Ask HN: What's the worst you've ever screwed up at work? by zamalek on Thursday January 30, 2014 @08:09AM (#7150423)
> I've only cried literal tears once in the last ten years, over business.

> Don't code when you're distracted.

Same story here, I can't remember the exact scenario but I was concurrently acting under all three of my titles (Developer, Architect, Escalation Engineer/Critical Debugger). The customer (who was 7 hours different to us) had been battling for 3 months and we were getting nowhere (all thanks to a, since fixed, bug in WinDBG which essentially came down to broken stack traces in certain scenarios), for those 3 months I had been working 20 hour shifts (development by day, support by night).

Under that strain I eventually made a screw up with the dev and it cost QA time. The MD of my region had a sit down with me and I

> cried literal tears

Needless to say they were impressed that I someone cared so much and sent me home to sleep. The next day I came in and decided to go through the 800kloc codebase line-by-line and see what could be causing the issue - I found it in a few hours.

Lesson #4: Get sleep.

Re:Ask HN: What's the worst you've ever screwed up at work? by bedhead on Wednesday January 29, 2014 @05:46PM (#7147670)
Who packs their cell phone in a box when moving???
Re:Ask HN: What's the worst you've ever screwed up at work? by teacup50 on Wednesday January 29, 2014 @07:43PM (#7148282)
Who ships a product release while in no position to monitor the results?

Who ships a product release immediately after a commit?

Re:Ask HN: What's the worst you've ever screwed up at work? by tptacek on Wednesday January 29, 2014 @07:53PM (#7148342)
"People who are shipping actual products instead of talking about them on message boards", perhaps. Running someone else's product company from a message board is a little like playing Jeopardy! from your couch, right?
Re:Ask HN: What's the worst you've ever screwed up at work? by lsc on Wednesday January 29, 2014 @08:20PM (#7148451)
Yeah. This was... my biggest takeaway from running my own company. I struck out on my own because I thought the business people who ran the companies I worked for were unethical idiots, and being not-an-idiot, I could do better.

Turns out? Nope.

I also find that the arguments I had with my bosses come around again, from my employees.

I still don't really understand how business works - but I've learned enough to understand that I don't understand how business works.

Re:Ask HN: What's the worst you've ever screwed up at work? by bedhead on Wednesday January 29, 2014 @09:59PM (#7148807)
All true, but that still doesn't answer what possesses someone to pack their cell phone in a box.
Re:Ask HN: What's the worst you've ever screwed up at work? by cpach on Thursday January 30, 2014 @06:13AM (#7150040)
Seriously guys, take it easy.

My question is: Who never makes mistakes? I certainly do not belong in that set.

And I like Patrick's candid business anecdotes. Snarky comments like this might disencourage people from writing useful advice here. Which would be very unfortunate.

Re:Ask HN: What's the worst you've ever screwed up at work? by bedhead on Thursday January 30, 2014 @09:32AM (#7150803)
Just trying to be light-hearted about it, sorry if that did not come across.
Re:Ask HN: What's the worst you've ever screwed up at work? by lsc on Thursday January 30, 2014 @04:29PM (#7153480)
>All true, but that still doesn't answer what possesses someone to pack their cell phone in a box.

this is the primary difference I see between programmers and sysadmins, development and operations. I know programmers who don't own cellphones at all, while I know some sysadmins who take tertiary backup communication devices on vacation.

It's a difference in focus.

Of course, most programming jobs have /some/ operational responsibilities, and most sysadmin jobs have /some/ development responsibilities, but most people see themselves as primarily one or the other, and act accordingly.

Re:Ask HN: What's the worst you've ever screwed up at work? by lsc on Friday January 31, 2014 @05:26PM (#7159594)
Note, I agree that packing your cellphone in a box was a mistake either way. But if you are primarily an operations/sysadmin type? that would be a really big deal kind of mistake, one that you probably wouldn't make very often. To a Developer type who saw their operations role as secondary, sure, it's still a mistake, but it's a smallish, forgettable kind of mistake.
Re:Ask HN: What's the worst you've ever screwed up at work? by teacup50 on Monday February 03, 2014 @09:48PM (#7175184)
You're right, I just run my own product company from my couch, and we'd never be daft enough to ship something critical right before disappearing into the ether.
Re:Ask HN: What's the worst you've ever screwed up at work? by BU_student on Wednesday January 29, 2014 @04:33PM (#7147109)
> Fortuitously, AR didn't have all that many customers at the time, so only 15 or so people were affected. Less than fortuitously, those 15 folks had 10 to 100 messages queued, each.

Excuse me for caviling at your post, but "fortuitously" is a synonym for "accidentally", not "fortunately".

Re:Ask HN: What's the worst you've ever screwed up at work? by lstamour on Wednesday January 29, 2014 @04:57PM (#7147309)
... and apparently people have been saying that about "fortuitously" since the 1920s.
Re:Ask HN: What's the worst you've ever screwed up at work? by kgermino on Wednesday January 29, 2014 @05:09PM (#7147422)
Well, a quick definition says "happening by accident or chance." Well I suspect he didn't know the proper definition (I certainly didn't) it's not that bad a choice.

Thanks for the comment though - I personally always appreciate a gracious correction.

Re:Ask HN: What's the worst you've ever screwed up at work? by xmrsilentx on Wednesday January 29, 2014 @08:43PM (#7148545)
I think what he meant was "It just so happened,"
Re:Ask HN: What's the worst you've ever screwed up at work? by bradbatt on Wednesday January 29, 2014 @06:42PM (#7147994)
Thanks for sharing—some good lessons learned in there! I love that you posted this as a comment on HN…it is worthy of it's own blog post.
Re:Ask HN: What's the worst you've ever screwed up at work? by zem on Wednesday January 29, 2014 @05:41PM (#7147636)
oh man. the idempotent queue thing reminds me of the time i was working for a very-early-stage startup, and we'd managed to persuade an exec from a pretty big company to sign up for a trial. our ceo got a very angry call the next morning; the guy had woken up to 300 email messages because our queue had hiccuped and, yeah, not idempotent.