This goes double if you're hosting on Heroku as you won't be able to correlate the changes Heroku makes with issues showing up for you. They're lucky that they hadn't pushed a change at the same time as Cowboy changed, or the debugging could have taken even longer.