News for hackers. Stuff that matters.
Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @08:50AM (#2257846)
Sounds like they are trying to fluff their reliability reputation.

I develop a web application that is used by schools and just can't entertain the notion of anything other than 100% uptime. I take the reliability of my product very very seriously. If one of my customers had a fire at their school and couldn't access our system for registers - that would be us and them up the proverbial creak without a paddle.

I've built up a company (over 7 years now) with a very good reputation for reliability and uptime. Don't assume that just because something is web based it doesn't require 100% uptime.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by rudiger on Thursday February 24, 2011 @09:34AM (#2258054)
Are you sure your web application "just can't entertain the notion of anything other than 100% uptime"? That sounds like a vacuous promise to me. Even telecoms switches are designed with something like 99.9999999% (9 nines) of availability; that's ~30 milliseconds of downtime a year.

I'm not criticizing you or your product, but if reliability is critical, it's made explicit with a realistic SLA.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @09:41AM (#2258089)
I didn't say we provide 100% uptime. We just don't entertain the idea that we may have to settle for less .. simply, we pro-actively monitor our servers and have enough redundancy in place to keep problems under OUR control.

We don't pretend, promise or have an SLA that offers 100% uptime.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by patio11 on Thursday February 24, 2011 @09:23AM (#2258001)
Keep in mind that the typical school has substantially less than 40% uptime...
Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @09:36AM (#2258067)
Not sure where you are based, but here in the UK schools are rapidly gaining better uptimes. The biggest problem we see is miss-configured proxies playing havoc with https connections.

I couldn't say what the average uptime is, but if a school network goes down we normally get an immediate call asking us if there is a problem our end because they use our system so heavily. So at a guess .. at least 98-99% for the majority of secondary schools, but don't quote me!

Re:Don’t scale: 99.999% uptime is for Wal-Mart by patio11 on Thursday February 24, 2011 @09:47AM (#2258110)
They're only actually occupied for about ~30% of hours in the week during the school year, and the hours are usually fairly predictable, which gives one some advantages. For example, if you had a scheduled maintenance window for 5 hours a week every week starting at 2 AM on Sunday, that would blow 98% uptime but no actual user would ever know. (This has saved my bacon a few times, as 2 of the 3 outages of my software were outside of peak hours, so I could count inconvenienced customers on one hand.)
Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @10:04AM (#2258182)
We use to find the holidays a massive advantage. But schools take some of their holidays on different weeks in different parts of the UK so as we expanded we effectively lost these maintenance windows.

Also teachers/students/parents access the system from home so we have to keep it running 24/7.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by ZoFreX on Thursday February 24, 2011 @03:44PM (#2259748)
Just because students are out doesn't mean the school isn't occupied! I used to work as a network admin at a boarding school, and there would usually be at least one "off-site" staff member working past midnight each day (and of course, on-site staff 24/7, although thankfully most of them sleep at night), as well as staff coming in at the weekend.

Computers have become mission critical in schools now - lessons, paperwork, planning, organisation, telephones, intercoms, security cameras†, locks‡ were all computer-dependent. To date that school network is the most advanced, highest-uptime network I've ever worked on: two server rooms (network continues at almost full capacity with the loss of either) in separate locations and a backup server in a third location. Unplanned outages were basically non-existent, and the loss of any service was very rare and involved lots of people shouting at us :(

† Internal only, external were still on tape

‡ I don't mean all the doors open if the network goes down, but the electronic locks were controlled over the network

Re:Don’t scale: 99.999% uptime is for Wal-Mart by elai on Thursday February 24, 2011 @04:18PM (#2259890)
At one of the colleges I attended, they shutdown the school's course registration/student management system from midnight to about 7 am every day. It was incredibly frustrating at times.
Re:Don’t scale: 99.999% uptime is for Wal-Mart by iwwr on Thursday February 24, 2011 @08:58AM (#2257877)
100% uptime is as fanciful a notion as risk-free investment.
Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @09:02AM (#2257897)
The point I was trying to make is that if I had that kind of public attitude to uptime, I'd be out of business pretty much overnight.

Yeah, 100% uptime is a crazy notion; but something always to be strived for nonetheless.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by iwwr on Thursday February 24, 2011 @09:07AM (#2257917)
To be strived for, but if you sign a SLA, nobody would be offering you 100% and tell the truth.
Re:Don’t scale: 99.999% uptime is for Wal-Mart by timthorn on Thursday February 24, 2011 @10:25AM (#2258269)
An SLA should specify penalties for missing the target. You can offer 100% uptime as the SLA, knowing that you'll only hit x%, but if the cost of compensating for 100-x% is less than the value of offering the SLA at 100% it's worth doing.

An SLA only ever guarantees commercial remedies, not service availability.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @09:16AM (#2257962)
I've been seeing it offered more and more in recent years for colo ip transit. Just do a quick search for "sla 100% uptime network".

Apart from one of the latest colo providers we use I've witnessed downtime with every single one that offered ip transit with a 100% SLA. That's why we use at a minimum 3 geographically distant providers. So far this approach has not let us down.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by cagenut on Thursday February 24, 2011 @09:33AM (#2258051)
thats because they stick in the fine print "once a month scheduled half hour outages don't count, also anything under 5 minutes doesn't count"
Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @09:46AM (#2258102)
There should be a law against that.

Reminds me of the "unlimited mobile phone internet" plans here in the UK which in the fine print resolve to "fair use policy of 1GB a month) .. why can't companies just call a spade a spade.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by Silhouette on Thursday February 24, 2011 @12:52PM (#2258971)
> why can't companies just call a spade a spade.

Slightly off-topic, but in case there's genuine curiosity there: AIUI, most of the advertising standards rulings about "fair use" policies related to Internet and particularly mobile bandwidth were made quite some time ago, when expectations were very different. It is possible that this was not entirely coincidental, and was motivated by the legal teams at major service providers who saw the current situation coming.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by bitwize on Thursday February 24, 2011 @10:26AM (#2258273)
Mah boiiiii, 100% uptime is what all true coders strive for.
Re:Don’t scale: 99.999% uptime is for Wal-Mart by maukdaddy on Thursday February 24, 2011 @12:28PM (#2258851)
I work in IT for a very large school district, and I have this argument all the time. Aiming for 4 9s or more is just a waste of money. Sure having the gradebook or scheduling app down sucks, but it doesn't matter at a school.

Can the teacher still teach if the systems go down? Yes.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @01:39PM (#2259214)
I've worked in education for over 10 years. 7 of which was on site at one of England's largest secondary schools whilst consulting for schools all over the country.

I've had first hand experience dealing with dozens of LA (what you call districts) technical teams that take this kind of attitude and frankly I find it appalling.

I remember when schools where trying to bring in electronic registration, and the LA network teams just didn't give a damn about the schools needs. Instead they made there own conclusions as to what the school needed, and to them, having a reliable network so the school could move into the 21st century wasn't one of them.

Schools are becoming more and more engaged in technology, and downtime and poorly maintained systems set teachers and the work they do, back. Teachers need to be able to rely on the technology, simple as that, no excuses.

You're not expected to reach 100%, but to admit defeat and argue for that defeat is poor form.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by arockwell on Thursday February 24, 2011 @09:35AM (#2258058)
What is your actual uptime?
Re:Don’t scale: 99.999% uptime is for Wal-Mart by xd on Thursday February 24, 2011 @09:59AM (#2258159)
Good question.

We have had 3 incidents in the last 7+ years. The first 2 occurred when we only had servers in one site.

1, http://www.risktec.co.uk/news/business-continuity-%E2%80%93-... (lasted days!!) 2, A thunderstorm knocked out the power to our datacenter, the backup UPS and generators both failed to engage. (lasted a day, we also took our business elsewhere)

After the 2nd outage we decided to take up additional colo space elsewhere in the country. This has worked for years and any failures in our life datacenter have quickly been switched over to a hot backup datacenter.

We have had around a half dozen minor interruptions caused by server component failures ( psu / hard drive). This is because it takes a few minutes to detect the failure and switch over to a backup.

3, The last major failure was with our DNS hosting. They decided to scratch our main zone from there records because of a billing error. Luckily for us it happened on the Friday before a weeks school holiday. Needless to say we took our business elsewhere.

I'm sure we have had other instances of downtime I can't remember, but that's pretty much us in a nutshell. No idea what our uptime on paper would be, funnily.

Re:Don’t scale: 99.999% uptime is for Wal-Mart by arockwell on Thursday February 24, 2011 @01:47PM (#2259267)
That's not too bad. But does that all add up to more than 9 hours a year of downtime? If so, you're already doing worse than 99.9% uptime.

I would also bet your uptime is worse than you think if you're not actively measuring it. However, I'm sure it is "good enough". I doubt it is worth spending an order of magnitude more money to get better than 3 9's for your business, which is more or less the point of the article imo.

I find the percent availability -> hours of downtime chart on wikipedia to be a really helpful reference for these types of discussions: http://en.wikipedia.org/wiki/High_availability

Re:Don’t scale: 99.999% uptime is for Wal-Mart by jcampbell1 on Thursday February 24, 2011 @09:58AM (#2258154)
I also develop a web application used by schools. I have no problems taking the system down in the middle of the night for 30 minutes. Schools are quite possibly the most forgiving customers. Their revenues come from the government, so if your app goes down, they are still going to make payroll.