Hi. I work at Facebook. We're working on restoring the set of apps that were accidentally disabled. Many of them are back, some are still in process. (As people have pointed on the thread.)

Sorry about this. We clearly made a mistake.

Hey Chris, appreciate your participation here and on SO.

Are you at liberty to provide more insight? I'm really curious to know why some (and not all) accounts were affected.

(I also work at Facebook) We have systems that block spammy apps that are 99.9% of the time really incredibly sophisticated and get a ~0% false positive rate.

This is a case of the 0.1%. :( Folks here are scrambling to undo this. Very, very sorry. Things should now (17:47 PT) be all set. Please do comment on the bug link if you're still having issues.

Still can't see a few of my apps - App ID: 152351391599356

This one should be fixed - thanks for passing it on!

Can you check again? We think that we've got them all, but I want to verify that it's still missing.

Yep. It's back.

[nod] We'll look into it. Please follow / comment on the bug thread at https://developers.facebook.com/bugs/702273849788774

I don't know the root cause, nor how many apps were affected.

Edit: Sorry, don't mean to sound evasive here. I just don't want to provide incorrect data.

Any chance you can look at restoring 309852345803454. I lost 4 apps but this is the most important one as it's integrated with an iOS app in the app store so I can't just create a new app and carry on as I've done for the other apps.

I'll add it to the list for sure. Can you verify that it's still missing?

Fixed! Thanks!

This one should be fixed. Thanks for passing it on!

Also, what were the other App IDs?

588613334503218 531241493603547

The other one doesn't matter as it was only for development and has been recreated.

These should now be restored. Can you double check for us?

Yes, they are restored. Thanks for fixing this for us.

Could you take a look at 641670749179788 as well? The page tab works fine but the canvas URL redirects to 4oh4.php. Thanks!

Hi! We fixed some more stuff last night. Can you check and see if this one has been fixed or if you're still having issues with it?

Yep thanks, everything looks fine again!

Could you please look at my other comment in this topic? I'd be very grateful if you could help me.

Thanks for being so quick to respond

Now if only they would disable user accounts, the world would be a better place.

All of my apps just vanished, including lots of apps that were built for clients. Anyone at Facebook reading this and have any idea who we can contact to try and get this resolved? I can't subscribe to the bug because I'm suddenly not a developer.

We're definitely working on it right now. Top men (in the non-sarcastic sense) are getting it fixed.

Thanks! Great to hear it!

Your company should consider investing in some top women.

Not "my" company, but why?

Can you check and make sure that the apps have been restored? It should be fixed now.

move fast, something something

Today is definitely a something kind of day.

move fast, brake often

Hi, I work at Facebook along with Chris. All of the accounts and apps should now be restored. If any are not (and were disabled between 3pm and 5pm PST today), please let me know the app ID and I will investigate. My sincere apologies this happened - we are looking into ways of making sure this cannot happen again.

OK. We believe that we have this 100% restored now. Can people here verify that all of their apps are back? (Make sure to reload the apps dashboard.)

If you still have one that's missing can you paste the app id here?

I'm still missing the majority of my apps.

395120080593788 is one, I believe. That one was just a facebook page tab so I only have links to it. From the looks of it, the only apps that returned were ones with multiple administrators.

144346579010587 is another app id that didn't return.

OK, that helps. Other people that have IDs, I would appreciate it.

Both of these are back, just verified.

Yep, all fixed up. Thanks guys!

Apparently I lied here. Now it should be restored.

From their status page: "For updates, please subscribe to https://developers.facebook.com/x/bugs/702273849788774 "

That URL gives me a 404. I'm not surprised...

Edit: they've fixed it now. Correct link is in replies below.

https://developers.facebook.com/bugs/702273849788774 seems to work.

You need a developer account to view. This bug disabled all the dev accounts.

This bug did not disable all dev accounts. My personal one stayed active, the one tied to work did not. I'd love to hear a technical post mortem from Facebook, but that's pretty unlikely.

And just for clarification, this has been marked as resolved.

Yeah, the /x/ was a typo. Remove it and it should work.

And the quotation mark https://developers.facebook.com/bugs/702273849788774/

Please help @chrisblizzard or @dweekly or anyone in charge or at least tell me what happened:

I can't access FB Developers site (can't browse any app, read-only bugs section - can't subscribe or comment). Can't give you app IDs, but can you look at userId: 100000522453817? I'm getting response:

"App creation failed Our automated systems have temporarily blocked you from creating new applications. If you think this is an error, send us your feedback."

I didn't get any mail with information or something, just blocked dev account and nothing else. I'm a developer and such problem directly affects my workflow.

We had an issue with app creation last night. It should be resolved. Can you check now and see if things are working for you? (Sorry.)

No, still can't create new or access my existing apps, especially 314378888602282.

When authenticating:

"error": { "message": "Error validating application. Application has been deleted.", "type": "OAuthException", "code": 101 }

Holy moly!

I admit that is a truly terrifying response.

I like your responses here, though. For somebody going through an oh-shit moment I'm admiring your aplomb.

Smarter people than me are doing the actual work. That's why I have time to reply here.

For a while there I'd forgotten there was such a thing as a Facebook app

They're aware of the issue and are working on a fix.

Status: https://developers.facebook.com/live_status

Why wouldn't they proactively notify the people whose apps were mistakenly disabled, so that questions didn't have to come to HN and SO in order for people to find out what's going on? I mean, from all indications they know which apps those are, and they have contact info for the developers. Best and brightest!

"put on hold as off-topic by kevingessner, Gilles, animuson 6 mins ago This question does not appear to be about programming, within the scope defined in the help center. If this question can be reworded to fit the rules in the help center, please edit the question."

...

Yeah, we were kind of expecting that from stack overflow :(

It's a Facebook end-user support issue, not a programming question. What else would you expect from Stack Overflow? Closing it as off-topic is appropriate.

Oh, I wasn't actually complaining there. It's totally appropriate, we were just wondering when it would happen. (It was actually useful as a channel to find app ids that were having issues.)

Hint: if your app isn't working check if Facebook didn't unset your app namespace. If so, set it again and save changes. Probably it was possible to take over other developers app namespaces due to this bug.

Some of my apps are back, but some are still missing. Anyone in a similar boat?

Yeah, some people are still missing some of their apps. We're working on restoring the rest of them.

Same. I have only have 1 app back. I think i had 6 total.

Can you verify that they are all back? And if some are missing, what their App IDs are?

They are all back now.

Same here. I also had to verify that i am a real person.

Yeah, I got about 6 back out of 30ish.

In terms of response, chrisblizzard has done almost a better job than Twilio did last month. Do we expect a full post-mortem, or is that where facebook will fall short?

The short term plan was to make sure that we got developers up and running again. I know for a fact we're going to do an internal post-mortem for sure. To be 100% clear and honest, I'm not sure if that will turn into an external post-mortem or not. I'll personally push to do one, but I can't make any guarantees. (It's not my call so I can't promise.)

FYI this issue still isn't resolved for a number of people. I'm still waiting on several.

Do you have some app ids so I can go and see what's up?

I put them in the comments as you suggested. The main one was just restored 288731291242658. I don't actually know the full list of IDs, but it looks like the ones I care about were restored.

OK, thanks. As of a couple of minutes ago we finished running the last job to restore the IDs. If you can still find one that's missing please let us know what the ID is.

Can you look at my issue and tell me what happened? https://news.ycombinator.com/item?id=6210288

"This question was removed from Stack Overflow for reasons of moderation."

Got 3 back out of 20. Two had more than 5k users. Both missing.

Can you verify that all of them are restored now? If not, can you post an ID of one that's missing?

They should all be back now. If not, trouble you to post the app IDs?

Still working on getting everything restored.

Had the same issue.