News for hackers. Stuff that matters.
Re:Everything's broken and nobody's upset by luu on Monday September 17, 2012 @06:20AM (#4531898)
The blurb on the sidebar proclaims Scott's MS experience. It's surprising to see someone with a Microsoft background making this complaint. MS spends more effort than any software company on testing, not only in just plain hiring lots of testers, but also on formal methods. They have some of the top formal methods people in the world doing research for them, and armies of people trying to put that research into practice. Making stuff work is hard, and I'd expect someone who worked for Microsoft to know that.

I work for a hardware company. Bugs are really, really, bad. If we find a hardware bug in real silicon, at best, we catch it the moment we get the first chip back, and it means that we have a multiple month delay as we fix it and tape out a new chip, not to mention the cost of throwing away all of the partially fabbed chips we've got, plus the multiple million dollar cost of a new mask set. At worst, we have a recall [1]. We take testing very seriously, and we do a lot more formal verification than most software companies.

The only things you can be sure about are things that have been formally verified [2], and the list of things that you can formally verify is tiny. We formally verified our adder. It took months. Then we did the multiplier, which was much harder. It took about the same amount of time because of the experience we gained doing the adder, but it wasn't easy. Division took a lot longer, even with the experience of doing addition and multiplication. To think that we can advance the state of the art of "things that work" from something like a multiplier to a complex piece of software with "care" and our "collective will" seems overly optimistic.

Everything's going to be broken for the foreseeable future. Putting more effort into testing and less into features is a difference in degree, not in kind. It won't even prevent articles like this from being written because, if all you want to do is find ten bugs in all the software you use, that's still going to be trivial. Considering how much progress has been made in formal methods since 1970, I expect that finding 10 annoying bugs in all of the software I use will be trivial for my entire lifetime.

[1] Well, you don't have to do a recall. AMD had a hardware bug that could be fixed by a patch that degraded performance by 10%. Sun famously didn't include ECC in their L2 cache, which resulted in transient failures for a number of customers, and they made customers sign an NDA before replacing their parts. Guess how much people trusted AMD and Sun afterwards?

[2] Even then, you're never really sure. How do you know the formal verification process itself isn't buggy? It's turtles all the way down. I know some folks who were trying to build a formally verified OS, and they stopped using ACL2 after discovering a few bugs in it. After all, how can you trust your proof if the proof system itself has bugs? ACL2 is old and crusty, but that's precisely why it's used for more hardware FV than everything else combined, outside of Intel and IBM (both of whom have their own, excellent, internal tools). It's old enough to have great libraries. There are newer systems that have better architectures, but they don't have anything approaching the same level of library support for hardware. Yet another tradeoff of time to market vs. correctness. It can't be avoided.

Say you're an engineer who's worried that ACL2 is too buggy for your company to use. You tell your manager. She points out that maybe five ACL2 bugs are discovered every year, and they get more minor each year, as the system gets cleaned up. Moreover, none of the bugs discovered in the past three years have affected any of your proofs, and you wouldn't expect them to have an effect on any proof techniques you're going to use. So you stick with ACL2. And, because you do, there's a tiny risk of a bug. What does this example have to do with the original post? Bugs come from making little decisions like this. No single decision is sure to cause a problem, or (in a company that's serious about testing) even likely to cause a problem, but multiply that tiny probability by the number of times you have to make a tradeoff and the number of lines of code, and it's a statistical certainty that you'll have bugs.

Re:Everything's broken and nobody's upset by drzaiusapelord on Monday September 17, 2012 @02:32PM (#4534075)
I'm curious as to whether you really believe more testing is the issue? Half the problems on that list are directly related to political/business decisions. Chrome's installer sucks because Google wants to make sure you can install it with non-admin rights. It breaks basic Windows guideilnes because.. fuck you that's why. Ask google. Their war to get Chrome on limited users desktops is a business decision. When you're asked to build a broken thing, more testing isn't going to fix it.

The other issues are easily addressed by stopping to cater to lazy users so much. Do we need yet another bloaty app to do basic things? Look at itunes. Its a monster. I'm not even sure how to get a CSV of contacts off an iphone. In the past this was trivial with any contact manager because the feature was baked in. Now that's a feature for "enterprise admins."

The remaining things on his list are the 'appification of everything.' Everything is now a shitty touch app that's incredibly dumbed down and buggy. Why try hard, especially when you userbase is locked down by 2 year carrier contracts? Or when all their music is trapped in an itunes database that can't be moved anywhere or given to anyone.

Apple won. This is "easy" and "just works." Big buggy apps with dumbed downed features designed to lock user data as deeply as possible. We've entered a new era of vendor lock-in. It the 1980s all over again. We need open protocols to work just enough for web and email. Everything else is up for grabs.

Get used to it, this is what the market has chosen. Meanwhile, I have to somehow train my users on how to use Windows 8's new tablet-based GUI on their standard PCs. This ain't progress. This is fashion.

Re:Everything's broken and nobody's upset by jiggy2011 on Monday September 17, 2012 @07:18PM (#4535405)
Wow, congratulations. I think that one post just about summed up everything I hate about modern software.

It's actually gotten so bad now that I've found myself increasingly hacking together my own small apps and scripts to do some simple things that I want because it's often actually easier to build than figuring out how to use some of the software out there. It's actually an even better investment because I know my own stuff will never magically become shittier because of some update that gets pushed out automatically.

Re:Everything's broken and nobody's upset by parasubvert on Monday September 17, 2012 @07:46PM (#4535490)
I'm not quite sure I agree with you. User data is always accessible - I can get pretty much every aspect of my iTunes library in XML and there are plenty of 3rd party programs that can interpret it. Contacts on your phone usually sync with your PC contacts app (Outlook on Windows, Contacts on Mac OS X) or via iCloud. You could use those apps to export. On Mac, it can be exported as an "archive" which is a bunch of plist XML which is parseable. You don't bulk export directly from your phone because, I don't think too many people have ever asked for such a feature (beyond vCard, which does work easily).

Basically it seems to be sour grapes that consumers are finally getting what they want after 20 years of geek rule... It's not quite the 80s again.

Re:Everything's broken and nobody's upset by ghshephard on Monday September 17, 2012 @11:12PM (#4536097)
" Big buggy apps with dumbed downed features designed to lock user data as deeply as possible."

Apple uses a lot of sqllite and xml for your user data. I'd be curious as to what part of your data you are unable to get access to?

Re:Everything's broken and nobody's upset by JoeAltmaier on Monday September 17, 2012 @11:31PM (#4536162)
If there's no way to export it, its locked down.
Re:Everything's broken and nobody's upset by Flow on Tuesday September 18, 2012 @10:20AM (#4538104)
Just open the database file and do a select...
Re:Everything's broken and nobody's upset by JoeAltmaier on Tuesday September 18, 2012 @10:21AM (#4538110)
...and then? Fit it in an email or something? What about images, binaries etc?
Re:Everything's broken and nobody's upset by eps on Monday September 17, 2012 @08:47AM (#4532268)
Re: Microsoft and testing

Perhaps things have changed now, but previously MS had a setup with 2-4 testers per one developer whereby latter would churn out code as fast as he can (of a respective quality) and testers were expected to identify critical bugs so that the dev can plug them, in the same speedy manner as he'd code the original. I had few close friends work for Microsoft in the 90s and the impression I formed after chatting with them was that the MS products routinely were something that was hacked and hammered into a state of not crashing too frequently. So while Microsoft might have a strong focus on testing, it was used to be completely misplaced and misguided.

Re:Everything's broken and nobody's upset by timtadh on Monday September 17, 2012 @09:49AM (#4532523)
My impression is the Microsoft of the 90s has little to do with the Microsoft of today in terms of QA, testing, and security. This impression is "secondhand" (I have never worked for Microsoft). Reforms were made, so if you information comes from the 90s it may be time to update it.
Re:Everything's broken and nobody's upset by city41 on Monday September 17, 2012 @10:38AM (#4532750)
All MS info should come with a grain of salt from the simple fact it's a company of about 90,000 people with thousands of unique teams. I worked at MS and every team I interacted with handled testing and QA very differently.
Re:Everything's broken and nobody's upset by epo on Monday September 17, 2012 @09:40AM (#4532485)
A focus on testing is not the same as a focus on quality. Also, if your test criterion is "doesn't crash" then a lot of slowness, resource leaks etc, may be, and probably are, acceptable to you.
Re:Everything's broken and nobody's upset by nopassrecover on Monday September 17, 2012 @07:03AM (#4532000)
I don't think it's necessarily about perfection on release (though certainly sometimes it appears that things were never user tested).

Reading through Scott's article, he points out that it's not uncommon for an issue to be raised by many users and ignored by the product owner (my personal pet issue at the moment is Google Calendar no longer allowing me to accept invites sent to non-GMail addresses).

It might not be cost-practical to fix issues that affect a small section of your audience, but I'd like to think there's a certain qualitative aspect that comes into play, and if the issue affects 'n' people it should get a banner regardless of whether 'n' is only a minute percentage of your audience.

Re:Everything's broken and nobody's upset by sirmarksalot on Monday September 17, 2012 @05:12PM (#4534922)
I'm totally stealing this from a friend, but the single biggest cause of terrible user experience at large companies is the term "workaround." Late in the release cycle, a pretty grody bug comes up, causing the application to hang. Or maybe the installer just plain breaks at some stage. However, the dev team has found a "workaround" for the issue. Run the installer twice, and it works. Back out of the hung screen and try again, because the bug only repros 1/10.

The bug gets punted because it's on a non-critical path, there's no data lost, it doesn't crash the entire application, and there's a workaround, even though the workaround sucks. Next time around, the bug is weighed against new feature work, and judged based on how it aligns with the product's core pillars. Not important enough this cycle, maybe next cycle.

Next cycle comes around, and the team no longer knows how to fix the bug, because it's no longer fresh in anyone's mind, or the person who wrote it doesn't work there anymore. The broken module is now essentially a black box, and the cost of fixing the bug has increased drastically. The bug gets punted again until the next release, because hey, we've got a workaround.

The project will be cancelled before the bug is fixed.

Not all big companies work this way, but enough of them do, or have subgroups within them that do, that it creates usability problems like the ones in the article. It's not that the problems don't get found by QA, it's just that nobody can be bothered to address them.

Re:Everything's broken and nobody's upset by dustingetz on Monday September 17, 2012 @05:32PM (#4535004)
> The project will be cancelled before the bug is fixed.

it most certainly won't be cancelled, despite multiple farked edge cases, if the critical path is providing business value. the customer increases revenues, the software company turns a profit, the managers and developers get nice bonuses to give their children a nice education, our 401k's get stable returns, the world gets buggy software but not THAT buggy, and everyone is happy except for a few cranky developers who get paid a lot of money to maintain shitty software from 9-5 inbetween disney world trips and cruises.

Re:Everything's broken and nobody's upset by sirmarksalot on Monday September 17, 2012 @05:41PM (#4535046)
I meant "the project will be cancelled" in the same sense as "the heat death of the universe will occur." Basically, the problem will persist until either the end of time, or until such a time as that particular component is deemed completely obsolete and replaced wholesale.
Re:Everything's broken and nobody's upset by dons on Monday September 17, 2012 @08:21AM (#4532177)
> ACL2 is old and crusty

We have many new shiny FM tools now, though, that are far closer to regular programming languages. In particular, Agda, Coq, Isabelle are fairly easy to pick up for people with an FP background (rather than a formal methods background).

The breakthroughs in SMT solver performance have been astounding to watch, and their use for "brute force" FM is certainly growing.

Of course, verified code is still many times the cost of unverified code, but the gap is coming down.

Re:Everything's broken and nobody's upset by jetti on Tuesday September 18, 2012 @09:58AM (#4537983)
I'm using Coq for a formal method class and the CoqIde is a piece of software that falls into this category. If I have two files open and I run a search, there is a good chance that the CoqIde will crash and I'll lose my work. I've worked around it by saving my work before I search but I shouldn't have to worry about searching in an IDE
Re:Everything's broken and nobody's upset by tluyben2 on Monday September 17, 2012 @10:09AM (#4532610)
MS has fantastic formal methods people indeed. Rise4fun.com is some of the great stuff coming out of there. However; are they using this? Because it seems to be only research and most programmers (inside / outside MS) will not even vaguely understand how to write formal proofs, even with brilliant stuff like Dafny (which no-one uses obviously as it is a research thing).

I studied in Eindhoven under a pupil of Dijkstra and my father studied under Dijkstra; formal methods are kind of a natural thing for me since I was introduced to computers 30 years ago, but no clients are paying for the kind of time needed to write these proofs. Well, rephrasing; no client of us.

I think there needs to be a breakthrough in the way proofs and software are written; now it's not very practical in most cases.

Re:Everything's broken and nobody's upset by rbanffy on Monday September 17, 2012 @02:48PM (#4534164)
> Making stuff work is hard

Making complicated stuff work is hard. There is a reason some people design tools to do one thing and do it well - it's easy to be sure it does that one thing correctly all the time and easier to check it keeps doing it.

Maybe we're betting our simplicity on the stack we build upon (I certainly build a lot of simple stuff on top of very tall stacks of mostly simple stuff), I'm not sure there are no surprises hidden in it. However, we should strive for making things (machines, software) that are simple inside. When we do that, it's easier to make the simplicity transpire and become an elegant form-follows-function kind of design. It will not be possible to avoid all bugs, but by making conscious decisions against complicated designs, we may make some lives less miserable.

Re:Everything's broken and nobody's upset by romaniv on Monday September 17, 2012 @11:00AM (#4532847)
Yours is a typical "everything is uncertain and hard" excuse.

There are plenty of cases when developers do avoidable things that will almost certainly cause issues. And then those things do cause issues. And then developers change nothing in their behavior, and continue doing those same kinds of things - because they don't care. I see results of this almost every day.

Re:Everything's broken and nobody's upset by dpitkin on Monday September 17, 2012 @08:27PM (#4535619)
I think I read this from HN but I use this as the best example of mission critical software, the Space Shuttle at NASA.

The most important things the shuttle group does -- carefully planning the software in advance, writing no code until the design is complete, making no changes without supporting blueprints, keeping a completely accurate record of the code -- are not expensive. The process isn't even rocket science. Its standard practice in almost every engineering discipline except software engineering.

http://www.fastcompany.com/28121/they-write-right-stuff

Re:Everything's broken and nobody's upset by JoeAltmaier on Monday September 17, 2012 @08:29PM (#4535628)
But most engineering projects don't have hundreds of millions of moving parts. At that scale, standard practice becomes intractable.

The Space Shuttle went up the first time with a few K of core memory. The software effort was really pretty trivial compared with systems today.

Re:Everything's broken and nobody's upset by dpitkin on Monday September 17, 2012 @08:33PM (#4535643)
The article is from 1996 but even then they were far beyond a few K.

"Take the upgrade of the software to permit the shuttle to navigate with Global Positioning Satellites, a change that involves just 1.5% of the program, or 6,366 lines of code. The specs for that one change run 2,500 pages, a volume thicker than a phone book. The specs for the current program fill 30 volumes and run 40,000 pages."

Re:Everything's broken and nobody's upset by JoeAltmaier on Tuesday September 18, 2012 @08:02AM (#4537514)
The brain of NASA’s primary vehicle has the computational power of an IBM 5150, that ’80s icon that goes for $20 at yard sales. According to NASA and IBM, the shuttle’s General Purpose Computer (GPC)—which controls, among other things, the entire launch sequence—is an upgrade of the 500-kilobyte computer the shuttle flew with until 1991.

  --  http://www.popsci.com/node/31716
Re:Everything's broken and nobody's upset by JoeAltmaier on Monday September 17, 2012 @11:20PM (#4536126)
My point exactly. That process can't work unless you have millions to spend.
Re:Everything's broken and nobody's upset by roedog on Tuesday September 18, 2012 @08:20PM (#4541339)
This process results in low bug counts. But, even then, NASA still programs the software to accept patches as they fly.

I don't know how applicable the NASA way is to commercial firms. They have different goals. National Mission Success vs. shareholder returns. The commercial firms are still making money and happy shareholders with a lot of bugs. While a single unit conversion bug on the Mars Climate Orbiter resulted in the loss of a multi-billion dollar spacecraft.

Re:Everything's broken and nobody's upset by mzl on Tuesday September 18, 2012 @12:46AM (#4536380)
While a system as complex as ACL2, Coq, or Isabelle most likely will have bugs, the really interesting question is if the proof verifier that is run on the generated proof is too complex or has bugs.

In my view, one should preferably have several independent proof verifiers that have been manually inspected for there to be sufficient trust. It is ok if such verifiers are slow, simplistic, and use lots of memory, given that they should preferably be used as a final check that the verification process worked correctly.