News for hackers. Stuff that matters.
Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @05:46PM (#11607999)
In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.
Re:Automated Testing for League of Legends by debaserab2 on Sunday May 01, 2016 @06:05PM (#11608071)
When customers discover bugs before you do, it reflects poorly on your reputation and harms the trust you have established with the consumer of your application. That's not a myth.
Re:Automated Testing for League of Legends by im_down_w_otp on Sunday May 01, 2016 @06:17PM (#11608107)
Or you might also work in a hyper-regulated marketplace dealing with huge amounts of trust and money, and so finding a bug in production can be a very, very, very big deal.
Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @07:26PM (#11608311)
Obviously, but not always, or even usually.
Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @07:26PM (#11608307)
Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs.

The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.

Re:Automated Testing for League of Legends by MichaelBurge on Monday May 02, 2016 @09:20AM (#11611059)
I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'.

The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth.

Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick wouldn't fool anyone. But, just be careful about letting fraudsters determine your opinion on a topic, whether for or against, whether they're caught or not.

As for bugs specifically, there are huge benefits to being able to reason about systems as if they are bug-free. It's a PITA when your libraries, kernel, compiler, upstream API, etc. don't behave as expected. As professional software developers, we can work around them; users can sometimes get really confused when things don't work as expected. Sometimes they're scared to tell you.

Smaller companies can probably bear more bugs, since they can give individual users more attention. Larger ones tend to need to work more reliably.

Re:Automated Testing for League of Legends by dimino on Monday May 02, 2016 @11:08AM (#11611944)
I don't think the characterization you're making of my argument is accurate, and I don't get the sense you're interested in understanding what I'm explaining here, so I'm probably done.

I'll leave you with a recommendation to read The Startup Owner's Handbook, and listen to the Stanford lectures given by Sam Altman and friends, who talk a lot more about this.

Re:Automated Testing for League of Legends by meowface on Monday May 02, 2016 @03:14AM (#11609656)
A particular study being debunked does not necessarily make its hypothesis a "myth".
Re:Automated Testing for League of Legends by developer2 on Sunday May 01, 2016 @07:37PM (#11608351)
Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline.

These sort of downtimes hurt customer satisfaction and the bottom line. If serious issues arise often enough, customers lose confidence and patience and may leave your product for a competitor. And while you're offline trying desperately to rush a fix, you're losing revenue.

If the service/product you're offering has competitors, quality matters. Sometimes it's not even the measurable quality, but the perception your customers have. Find me a business owner who thinks major bugs in production are not "a huge problem".

Re:Automated Testing for League of Legends by Tinyyy on Monday May 02, 2016 @02:04AM (#11609442)
On the other hand, Valve tends to treat bugs differently with Dota 2.

They have a test client in which a small group of players can test out new patches and find bugs. Recently when patch 6.87 was released, they practically skipped the test client and deployed it straight to the main client. The result was that there were all sorts of bugs in the game, but they worked really quickly to fix them so that after a few days, they were almost all gone. I think Valve's mentality is that it is more efficient for their large player base to beta test for them instead.

Also, there's been a few times when the game is literally unplayable for one or two hours because Valve pushed a random update.

Re:Automated Testing for League of Legends by Shengbo on Monday May 02, 2016 @08:47AM (#11610872)
Riot does the same, they have a public beta environment(PBE) where they deploy new patches.
Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @08:41PM (#11608567)
> Find me a business owner who thinks major bugs in production are not "a huge problem".

"Move fast and break things" - Mark Zuckerberg.

Re:Automated Testing for League of Legends by Ralfp on Sunday May 01, 2016 @09:01PM (#11608623)
They moved away from that, didn't they?
Re:Automated Testing for League of Legends by developer2 on Sunday May 01, 2016 @09:08PM (#11608644)
Indeed. Quote from Mark:

>> We used to have this famous mantra... and the idea here is that as developers, moving quickly is so important that we were even willing to tolerate a few bugs in order to do it. What we realized over time is that it wasn't helping us to move faster because we had to slow down to fix these bugs and it wasn't improving our speed.

Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @10:43PM (#11608936)
They grew as a company to a place where they could slow down.
Re:Automated Testing for League of Legends by Too on Tuesday May 03, 2016 @11:52PM (#11626045)
Break things does not imply "break things in production".
Re:Automated Testing for League of Legends by jnardiello on Sunday May 01, 2016 @07:27PM (#11608312)
> But the idea that catching a bug in production is a huge problem is a myth

You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions.

And this isn't even considering mission critical software where bugs can (literally) kill.

Re:Automated Testing for League of Legends by JabavuAdams on Monday May 02, 2016 @09:21AM (#11611068)
So, excluding life-critical software, it's still a business decision.

Engineering will naturally tend to geek out on infrastructure projects like this and they need to be kept focused on the business case. There has to be some push-back.

What is the cost of even one engineer working full-time on build verification?

The thing you really want to avoid is getting blind-sided by devastating bugs or systemic process problems. So it's a balance.

I've just seen many cases where projects got bogged down as developers built their super-uber-build-test framework. It's easy for people to push these projects through without rationally investigating the cost / benefit, because ... because ... you're not seriously suggesting that we shouldn't test more, you monster!?

Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @07:31PM (#11608329)
Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
Re:Automated Testing for League of Legends by jerf on Sunday May 01, 2016 @08:10PM (#11608483)
Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not an even more intense focus on getting bugs identified before production because the paper probably understates the impact given real bug distributions.

(I say "identified" because you don't always fix identified bugs. But you really, really want to have as much as possible identified before production. Production is a terrible place to find bugs for the first time. Yeah, every once in a while an identified bug might have a far worse than expected impact, I've had that happen, but at least in my experience the completely unidentified bugs are what kill you the hardest.)

Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @08:39PM (#11608561)
This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies.

If you're building rocket engines that carry people, you think like this. If you're building a social networking website, or a food ordering website, or a home sharing website, the time it takes to go from "oh there's a bug" to "that bug is fixed in production" is a matter of hours, if not minutes.

Re:Automated Testing for League of Legends by jerf on Monday May 02, 2016 @10:12AM (#11611488)
The problem with bugs is not the bugs. It's what the bugs do. Bork up your database and the bug fix is not "minutes". (Especially if you don't notice for a while.) Piss off your customers and the bug fix is not "minutes". Screw up your money handling and the fix is not "minutes". (Well, your part of the fix may be just one minute, though that minute will be "you're fired".)

If you've only ever encountered bugs that can be fixed in "minutes", you're either very lucky, or not working on anything all that important. Or possibly, not very perceptive and you've actually got a mess on your hands and you haven't realized it yet. I've inherited systems run by such people; they think everything's hunky dory but it turns out you can hardly figure out how to connect two records in the database together correctly because they were too smart for academic bullshit like referential integrity, and, lo, their database was low on the integrity. Tends to work, except the amount of elbow grease required increases without bound until it exceeds the capabilities of the developer in question, and suddenly, one day, they wake up and their job is in serious jeopardy because of what's going down and what they can't get back up.

You assert that this is the "older" style of thinking, which shows a lack of understanding of your computer programming history. What you advocate is "cowboy" thinking, and it predates what I'm discussing by quite a bit. The entire 1970s was basically run this way, and it shows in those handful of remaining technologies that are still with us, and their tendency to just pound through problems as if errors are an admission of failure.

Re:Automated Testing for League of Legends by dimino on Monday May 02, 2016 @11:15AM (#11612026)
I think you're having a reaction to the higher risk profile that comes with breaking in production. That's understandable, but wrongheaded. Plenty of companies are very successful with the mindset as I have described.

Netflix runs their Simian Army in production, for example. They sometimes pull infrastructure pieces offline intentionally to test their resilience. A good production infrastructure is incredibly resilient.

Re:Automated Testing for League of Legends by swanson on Sunday May 01, 2016 @06:10PM (#11608088)
Are you referring to research on the 'cost of change curve' or research on the economic impact of production bugs?
Re:Automated Testing for League of Legends by dimino on Sunday May 01, 2016 @07:28PM (#11608315)
I am on mobile, so it's hard to provide links, but I'm referring to the "cost of bug in production" chart that's one of the more famous studies, from a citations perspective. Completely fabricated research.