Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by aresant on Thursday September 23, 2010 @03:50PM
(#1721053)
It's hard to overstate the damage this is causing for merchants that prominently display FB like on their pages - lots of big ugly blank spaces / errors out there.
Realize this is temporary but shows why if you install any external vendor tool as a part of your site make contingency plans that switch on errors.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by justinbach on Thursday September 23, 2010 @04:26PM
(#1721277)
You're telling me--of course this has to happen the day before client review of a major Facebook Connect site whose only authentication is via Facebook Accounts. We had discussed potentially having two types of accounts (Facebook- and email-based), but having two parallel authentication methods just seemed clunky. I guess this is the price we pay for trying to keep things simple...
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by robryan on Thursday September 23, 2010 @06:41PM
(#1721834)
Seeing as Facebook login lets you easily collect email addresses these days you could always have a page that says, facebook is down click here to get an email confirmation link to login or something similar.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by kgermino on Thursday September 23, 2010 @09:23PM
(#1722224)
Unfortunately its difficult to avoid relying on other businesses for crucial systems.
Example: Payment Processing, try building a business, especially an online one, without depending on Paypal, or at least Mastercard/Visa.
Sure you can reduce it by say not having FB Connect as the only log-on option but the simple fact is that you will almost always have to depend on other businesses, the best you can do is diversify and hope for the best.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by byoung2 on Thursday September 23, 2010 @09:59PM
(#1722305)
Exactly...people made this same argument a few years back when Amazon S3 went down for a few hours. Just have a few different options, and the chance that they will all be down at the same time is pretty slim.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by kgermino on Friday September 24, 2010 @05:26PM
(#1725497)
Yes and No. If you have a few options than your entire business won't come grinding to a halt but if half of your customers choose to use FB Connect as their login option than they likely don't care that FB's down, or that people with email logins can still access your site. To 50% of your user-base your site has failed and so in that sense I would say that you still rely on FB for your business.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by athom on Friday September 24, 2010 @05:20AM
(#1723154)
I use Facebook (sometimes), played FarmVille and Farm Town up until this January, and I STILL didn't get it right off. Guess pulling out of those games was a good idea after all. :)
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by moultano on Thursday September 23, 2010 @08:31PM
(#1722121)
The first 10 times I read this comment, I thought it was a pun on facebook's color scheme, until I realized that it's cornflower blue, not cauliflower blue.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by ryanwaggoner on Thursday September 23, 2010 @04:29PM
(#1721300)
I came here to post this same thing, but I just started thinking about it. When was the last time FB went down like this and took a bunch of core functionality down with it? Has it ever happened? And how long are they down...30 mins? Is that 30 mins of downtime every few years really more damaging than that the (potential) value that integration with the graph API or FB Connect can bring? Yes, you look like an idiot to a few users, but maybe you have 100x as many users as you'd have otherwise. I think the real reason not to build your business on the FB API (or Twitter or whoever) is strategic, not because of potential downtime.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by rwhitman on Thursday September 23, 2010 @05:31PM
(#1721555)
You're right, I should have phrased it "And that, my friends, is one of the many reasons its risky to build your entire business on top of FB Graph API."
Just went through a nightmare building a product entirely on top of fb graph. FB fails, a lot. And not just in terms of downtime, in terms of inconsistently implemented technology, poor documentation, lack of transparency, inconsistent policies etc.
Not saying don't use FB ever, just that it's risky.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by ryanwaggoner on Thursday September 23, 2010 @06:55PM
(#1721880)
Oh, don't get me wrong. I won't do FB apps for clients anymore, despite the high demand and great pay. It's just way too stressful to have the clients always angry at me because of the platform's shortcomings.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by jbillingsley on Thursday September 23, 2010 @04:54PM
(#1721403)
I fully agree with you. If your app is heavily social it would be ridiculous not to leverage the worlds largest social network, even with very occasional down-time.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by djcapelis on Thursday September 23, 2010 @05:05PM
(#1721452)
Their SSL site still continues to simply not work:
www.facebook.com uses an invalid security certificate.
The certificate is only valid for the following names:
a248.e.akamai.net , *.akamaihd.net
This is the biggest problem with the SSL PKI. It is partially responsible for the lackluster deployment of SSL. We need to switch to a TOFU model supplemented by a more nimble version of the current PKI for first connect acceptance.
You would not believe how often I run into problems just trying to use SSL on majorsites. If you can't trust a large organizations like facebook and akamai to mess up SSL support during problems, you simply will never get it deployed well on all the rest of the sites.
SSL is simply a poor user experience. Twitter's SSL site often goes down more often than the main site too. SSL users get a second class experience and that is the most damming thing I can think of for a technology where deployment is so critical.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by chrisgoodrich on Thursday September 23, 2010 @04:36PM
(#1721323)
Everyone should calm down and realize that Facebook being down for a few hours is not going to kill anyone's business. It may cause trouble and disruption, but not anything more intense than a daily commute in Los Angeles.
That being said, this makes me question Facebook's ability to deliver on the "Social Graph" promise. If they truly want to build a completely social internet, this kind of downtime can't happen.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by Terretta on Thursday September 23, 2010 @05:33PM
(#1721561)
> "If they truly want to build a completely social internet, this kind of downtime can't happen."
Why not?
Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate that unless you're selling a perfectly commoditized generic product, obtainable instantly from multiple sources, downtime doesn't materially impact revenue.
Users seem to manage to "queue up" their intended actions and do them when a site is back. And it's not like Facebook's users can gravitate en masse to anywhere else.
The founding principle of the internet is not "5 nines" uptime for all components, but resiliency. Switch off your mail server for a few hours; you'll still get all your mail when you turn it back on.
"Kids these days" build apps and APIs (and sites that rely on APIs) that are much more brittle, because today's developers are spoiled into thinking communication conduits are reliable. They're not.
The fabric fails, so your site, your apps, your protocols, should fail gracefully -- and recover gracefully later.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by dansingerman on Thursday September 23, 2010 @04:13PM
(#1721203)
As an aside I notice that 'Facebook' is not a trending topic on Twitter. Seems the term must be blacklisted in some way along with all the swears and other undesirable terms.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by _delirium on Thursday September 23, 2010 @04:58PM
(#1721423)
There are more principled ways to do that than a blacklist, though, like looking at current frequency compared to typical frequency. That way even something commonly mentioned could still genuinely be trending if it gets mentioned much more than usual on a particular day.
(It's possible there are scale-related reasons that make this infeasible, though.)
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by smackfu on Thursday September 23, 2010 @05:19PM
(#1721508)
Just not tracking the frequency of the most common words that will never be in the trending topics is probably a big performance win. I'm sure there is an additional blacklist that includes stuff like "and" and "the".
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by bbuffone on Thursday September 23, 2010 @05:04PM
(#1721444)
In researching web performance DNS configurations issues are common problem. It is usually related to www.foo.com and foo.com being configured differently. This seems to be at a high level the issue with facebook.com. if you go to "facebook.com" the DNS will resolve correctly, but it will redirect you to www.facebook.com. I guess they could stop the redirection and fix the problem immediately, or wait till the sorry.ak.facebook.com.edgesuite.net is fixed.
;; ANSWER SECTION:
www.facebook.com. 0 IN CNAME sorry.ak.facebook.com.edgesuite.net.
sorry.ak.facebook.com.edgesuite.net. 0 IN CNAME a1030.g.akamai.net.
a1030.g.akamai.net. 4 IN A 63.84.59.59
a1030.g.akamai.net. 4 IN A 63.84.59.10
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by joebasirico on Thursday September 23, 2010 @05:53PM
(#1721652)
When I heard FB was down I went over to twitter to see how many tweets this was generating, I was surprised to see the first refresh of "XXXX more tweets since you started searching."
I decided to start keeping track and I made a graph of the difference between refreshes. You can see my graph here. It seems like in the course of just a few minutes (I was probably conducting my little experiment for 10 min?) you can see the frenzy gaining momentum then it seems to die down until only hundreds of people are talking about it per 15 second refresh.
A little bit about the graph, each tick mark on the bottom represents a refresh (15 seconds or so I think). The Y axis is the difference between refreshes.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by yarapavan on Thursday September 23, 2010 @04:16PM
(#1721218)
Current Status: API Latency Issues
We are currently experiencing latency issues with the API, and we are actively investigating. We will provide an update when either the issue is resolved or we have an ETA for resolution.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by stevefink on Thursday September 23, 2010 @06:42PM
(#1721836)
DNS issues are the cause for so many outages by a myriad of prestigious web shops. It's incredible how often overlooked something so pivotal to your infrastructure goes without being properly monitored. I have seen entire businesses suffer hours of downtime because their monitoring systems would query an authoritative name server from RFC1918 IP space - but not the same from an externally visible address. An NS not responding to local queries can be just as detrimental internally as they are externally because of how web services are architected, obviously. In their defense, Facebook rarely incurs outages of this capacity, and I can bet it won't happen again and someone is getting seriously reamed. They can and will happen to the best of them, no matter how many MIT PhDs you have designing your systems.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by natrius on Thursday September 23, 2010 @04:15PM
(#1721212)
I don't think we're uncommon. Most sites that I see plastered with Facebook embeds are publications, and I'm sure they all see results similar to ours. Facebook is RSS for the masses.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by daten on Thursday September 23, 2010 @05:00PM
(#1721428)
The significance of the "like" button missing from other sites is lost on me.
I've been using the Application Boundaries Enforcer feature of no-script in Firefox to block facebook content when I'm not visiting a facebook site directly.
Site .facebook.com .fbcdn.net
Accept from .facebook.com .fbcdn.net
Deny
This is motivated by privacy concerns of Facebook's ability to track which partner sites I visit.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by parbo on Thursday September 23, 2010 @04:34PM
(#1721317)
First I just had an empty feed, then I got the feed back but posting links and comments timed out. Then it worked for a minute, and now I get the DNS error. Seems like this affects several of Facebook's applications.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by zppx on Thursday September 23, 2010 @07:36PM
(#1722004)
I experienced a facebook outage yesterday afternoon (-03:00 UTC) although the DNS resolved the facebook I could not ping their IP addresses. As you can read in facebook's twitter timeline.
Re:Facebook Down, Like Buttons Vanish, Internet Implodes
by doron on Thursday September 23, 2010 @06:03PM
(#1721690)
joking aside, this actually had this effect, as so many pages actually resolve some components to facebook.com, the slower response was felt on a wide variety of sites.
enough is enough!
http://developers.facebook.com/
(YMMV based on local cache etc.)Realize this is temporary but shows why if you install any external vendor tool as a part of your site make contingency plans that switch on errors.
Great, simple services but the macro effect that this is going to have on commerce today is crazy.
Luckily, they added Twitter authentication as well. But still, now a large portion of their site cannot login because of Facebook.
Example: Payment Processing, try building a business, especially an online one, without depending on Paypal, or at least Mastercard/Visa.
Sure you can reduce it by say not having FB Connect as the only log-on option but the simple fact is that you will almost always have to depend on other businesses, the best you can do is diversify and hope for the best.
facebook uptime has been good overall, considering they doubled in size in 12 months to 550m users
http://en.wikipedia.org/wiki/Centaurea_cyanus http://en.wikipedia.org/wiki/Cornflower_blue
In terms of reliability, FB is not twitter, but it certainly isn't google either...
Just went through a nightmare building a product entirely on top of fb graph. FB fails, a lot. And not just in terms of downtime, in terms of inconsistently implemented technology, poor documentation, lack of transparency, inconsistent policies etc.
Not saying don't use FB ever, just that it's risky.
You would not believe how often I run into problems just trying to use SSL on major sites. If you can't trust a large organizations like facebook and akamai to mess up SSL support during problems, you simply will never get it deployed well on all the rest of the sites.
SSL is simply a poor user experience. Twitter's SSL site often goes down more often than the main site too. SSL users get a second class experience and that is the most damming thing I can think of for a technology where deployment is so critical.
That being said, this makes me question Facebook's ability to deliver on the "Social Graph" promise. If they truly want to build a completely social internet, this kind of downtime can't happen.
Why not?
Even commerce sites such as Amazon or Ebay have downtime. Studies demonstrate that unless you're selling a perfectly commoditized generic product, obtainable instantly from multiple sources, downtime doesn't materially impact revenue.
Users seem to manage to "queue up" their intended actions and do them when a site is back. And it's not like Facebook's users can gravitate en masse to anywhere else.
The founding principle of the internet is not "5 nines" uptime for all components, but resiliency. Switch off your mail server for a few hours; you'll still get all your mail when you turn it back on.
"Kids these days" build apps and APIs (and sites that rely on APIs) that are much more brittle, because today's developers are spoiled into thinking communication conduits are reliable. They're not.
The fabric fails, so your site, your apps, your protocols, should fail gracefully -- and recover gracefully later.
http://www.facebook.com/developers/chart.php?type=at_total_t...
Which seems a bit underhand by Twitter.
Edit: I notice that 'Facebook isnt working' has appeared now. I've blogged about this issue here: http://blog.dansingerman.com/post/1174729229/twitter-mostly-... (if anyone cares)
There were hundreds of items on the list
(Edit: here is this link for now: http://techcrunch.com/2009/07/15/another-security-tip-for-tw...
I am sure that I saved the entire page of blacklisted words)
(It's possible there are scale-related reasons that make this infeasible, though.)
http://twitpic.com/1yqrwc
=====www.facebook.com
new-host-3:dpu2 robertbuffone$ dig -trace www.facebook.com ;; Warning, ignoring invalid type race
; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace www.facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19626 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;www.facebook.com. IN A
;; ANSWER SECTION: www.facebook.com. 0 IN CNAME sorry.ak.facebook.com.edgesuite.net. sorry.ak.facebook.com.edgesuite.net. 0 IN CNAME a1030.g.akamai.net. a1030.g.akamai.net. 4 IN A 63.84.59.59 a1030.g.akamai.net. 4 IN A 63.84.59.10
;; Query time: 10 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:29 2010 ;; MSG SIZE rcvd: 144
========facebook.com
new-host-3:dpu2 robertbuffone$ dig -trace facebook.com ;; Warning, ignoring invalid type race
; <<>> DiG 9.6.0-APPLE-P2 <<>> -trace facebook.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 19984 ;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION: ;facebook.com. IN A
;; ANSWER SECTION: facebook.com. 352 IN A 69.63.181.12 facebook.com. 352 IN A 69.63.189.16 facebook.com. 352 IN A 69.63.189.11
;; Query time: 13 msec ;; SERVER: 198.6.1.142#53(198.6.1.142) ;; WHEN: Thu Sep 23 16:57:44 2010 ;; MSG SIZE rcvd: 78
I decided to start keeping track and I made a graph of the difference between refreshes. You can see my graph here. It seems like in the course of just a few minutes (I was probably conducting my little experiment for 10 min?) you can see the frenzy gaining momentum then it seems to die down until only hundreds of people are talking about it per 15 second refresh.
A little bit about the graph, each tick mark on the bottom represents a refresh (15 seconds or so I think). The Y axis is the difference between refreshes.
Graph: http://drp.ly/e3oH
http://en.wikipedia.org/wiki/Evil_eye
> if you don't really need it.
In your case, you obviously do.
I've been using the Application Boundaries Enforcer feature of no-script in Firefox to block facebook content when I'm not visiting a facebook site directly.
This is motivated by privacy concerns of Facebook's ability to track which partner sites I visit.http://developers.facebook.com/live_status
War was Beginning.
Internet: What happen?
MarkZ: Somebody set up us the bomb.
FB: We get signal
MarkZ: What?!
FB: Main screen turn on.
MarkZ: It's you!!
DDH: How are you gentlemen !!
DDH: All your Information are belong to us.
DDH: You are on the way to destruction.