Please don't use uppercase for emphasis. If you want to emphasize a word or phrase, put asterisks around it and it will get italicized.
A couple of days later I caught up with the people allegedly responsible for this stuff in the bigger review meeting. "Shouldn't you open a SEV because you are shipping stuff that can't possibly work and aren't catching it pre-release?" The response was a lot of hand-waving and stupid manager-speak that boiled down to "no, go away".
I basically wrote them off at that point. If they could act that way and get away with it by way of implicit acceptance from upper management, that's just the way it was going to be.
I see the quality is as high as always.
:(
Facebook's own apps did not break since they are on the latest version of FBSDK.
What Facebook needs to do is to test whether their changes don't break other people's apps.
2. Client-side SDKs can be designed in a way that even if the server-side changes (which it shouldn't), it does not cause the whole application to crash. This can of course be hard depending at which level the SDK operates and how integrated it is into the app.
3. Are there no contracts between Google/Spotify/etc and Facebook? If I were FB and there were contracts on the line, I would make sure that there is no way something like this can slip through (automated testing).
I'd argue this is at the very core of why you test. I mean this problem is as simple as (n+1).m vs n.(m+1) in terms of versioning vs revisioning.
If that is not being tested then I'd say Facebook isn't actually testing at all - at least not in a practically relevant sense ...
... despite the shit-load of YouTube videos from FB devs lecturing about CI/CD and what not.
Other than remove the SDK, of course.
Fool me once, and all that.
Bingo! App developer voluntarily pulled in a dependency which they didn't fully understand. When you bring in a dependency, it's not still someone else's code/problem, it becomes your problem.
If hacker news goes down while you are writing this comment, it is your responsibility to ensure you can still post your comment despite the service breakage.
Many of the apps crashing are using Facebook for a portion of their app, such as one login option.
Letting that take down the whole app is insufficiently defensive coding, whether in the app, sdk, or glue between them.
Absolutely, but the problem is that in order for you to use these features, you're supposed to use the official Facebook SDK. The additional problem, is that Facebook has no interest in defensive coding, because if you're not interacting with Facebook in an app, there's no benefit to Facebook for you using that app at all.
(Am not a mobile dev, but it wasn't exactly hard to find that, either).
2. Throw exceptions
Don't tell me "No amount of testing" just because you aren't responsible for your own dependencies.
Removed the SDK myself from a top 100 App Store app.
Recently tested an enterprise marketing app with an SDK ... and we tested if the endpoint wasn't available. Nothing happened -- it worked just fine. Failed quietly and gracefully -- the user would never notice.
"We" know how to do this.
Facebook hasn't done anything to fix this problem since it happened a few weeks ago.
But it's Facebook -- I hope they keep it up. Can't wait for them to AOL themselves.
But if you're Facebook, and you're pushing your SDK as a great solution and it's embedded in high profile apps and you are letting this happen a couple of times, I get to make fun of you. (a dollop of sarcasm, to be sure..)
I can almost guarantee that the programmer who wrote this had a degree and passed a very rigorous testing process to get their job at Facebook. Doesn't mean anything when the company tells them to cut corners to iterate faster.
Stop blaming engineers for the mistakes of management.
Proximate Issues
The app makers cannot easily prevent the crash since it is happening in Facebook provided code. Perhaps the only thing they could do is to code in kill switches. Another possibility would be to modify the Facebook code. That introduces maintainability issues.
The crash happened because of a server side change that triggered roughly the Objective-C equivalent of a null pointer exception.
Objective-C is an old unsafe language.
Ultimate Issues
Client-side libraries have a poor history in the industry. They are often written by server engineers who aren’t expert on the platform.
For instance, if it was really a static load method that called the server, then this was a poor design decision.
Furthermore, weaker server engineers are notorious for breaking backward compatibility. They don’t design nor build in tests to ensure backward compatibility. The backward compatibility must include backward data compatibility.
The above issues are indicative of organizational issues.
Facebook is a little bit of an outlier. It was notorious for badly written client libraries stretching back to the Facebook platform days. The joke was that their libraries may have been written by interns even though so many companies were dependent on them.
The Facebook libraries and the ilk are used for sign on through Facebook and ad attribution. The trade off an app maker needs to make it a whether to run ads on Facebook and whether to support sign in.
Facebook is not the only big company with these issues. Google’s equivalent also crashed albeit at a much lower rate due to a much more convoluted issue.
Apple introduced the Swift programming language to address the deficiencies in Objective-C. That these companies are running into these issues makes me wonder if they never transitioned to Swift. They are certainly way behind the latest version of the libraries.
To summarize, it is a cascade of issues some technical and some organizational rather than a single coding error. These issues span companies.
> Objective-C is an old unsafe language.
This was a safe crash, not an unsafe one. Swift can and would crash the same way.
Looks like the server passed empty value in JSON which got converted to NSNull in Objective-C. It's a common mistake to assume null is returned instead of NSNull.
The Swift equivalent of the JSON library should return nil which you then need to explicitly check for non nilness before using unless you use the unsafe exclamation pint.
$ swift
Welcome to Apple Swift version 5.3 (swiftlang-1200.0.16.15 clang-1200.0.22.41).
Type :help for assistance.
1> import Foundation
2> // IIRC this used to throw an exception.
3> // Now a respondsToSelector: check is silently inserted and we get a surprise nil.
4> (Int?.none as AnyObject).count + "call Swift's bluff".count
Fatal error: Unexpectedly found nil while unwrapping an Optional value: file repl.swift, line 4
2020-07-10 18:03:43.815959-0700 repl_swift[73029:930570] Fatal error: Unexpectedly found nil while unwrapping an Optional value: file repl.swift, line 4
Execution interrupted. Enter code to recover and continue.
Enter LLDB commands to investigate (type :help for assistance.)I think we are in general agreement?
This one was a good one because the app cache was poisoned so the only solution was to ask customers to reinstall the app (after Google deployed their fix, which took a while) or upload an update that nuked the SDK cache on startup.
This would still happen because A) the programmers that made this mistake probably have the knowledge to pass any certification exam you can throw at them, B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff.
The engineering process was the issue here, and sadly the only way to prevent this is not to make personal barriers of entry higher but to slow down the process with more QA, testing or whatever you can throw at it. In other words, make software more expensive. And keep in mind that this would only give better numbers but it would not solve the issue either (see Boeing 737 MAX).
Very unlikely. You probably don't realize how tough some professional exams are in some fields. We are talking dedicated months of study here.
> B) most of the time we are not just building something that has exploded 1000 times before, we are creating new stuff
Most software out there is just CRUD stuff, not really new stuff...
Wow who would have thought no standards, free access to learning, fake-ass meritocracy, and exorbitant profits and market control would have led to such a thing
Pick a standard, whether it's a degree program, a cert, or apprenticeships. Hurry up! Just give us a solid target to aim for.
There's a good chance that the one who wrote this bug had a degree.
Surgeries in medicine still get botched despite all of their degrees and barriers. Degrees don't produce good code, good developers produce good code.
> I will have to go and get a degree
Was not aware that having a degree made you immune to writing bugs.
This will be a fart in the wind and lessons will be learned.
The industry seems to have very low standards. Let's let these mistakes happen over and over and let's just patch them as we go. No long-term thinking.
Someone should collect all these horrible incidents and start a university course on just plainly what NOT to do.
Don't sit in an ivory tower and throw stones in glass houses. Can you guarantee that your own SDLC house is in order? Because it might be, right up until a sneaky edge case nips you in the butt, and then suddenly it isn't.
Why should we just sweep this under the rug and call it a day? There is something very important that can be learned from this. Apparently, FB themselves have not learned, because this is not the first time this has happened. Why was this allowed to happen more than once? Did Apple, Google, Spotify, Facebook etc not take any actions?
"Shit happens" is one way to look at it. Another way is to question whether we as software engineers are really doing our job properly.
> FB themselves have not learned, because this is not the first time this has happened.
They have tens of thousands of engineers likely split into hundreds of different teams/microservices focussing on different parts of their software stack. A ton of them are new to it and nobody knows every part of the stack so shit can happen.
What is the biggest engineering organization size you have worked for and what was your uptime?
The linked issue makes it very clear that the "tiny outage" affects literally every app that has the Facebook SDK merely linked, for a frankly dumb reason (static initialization) that they should have learned from the last time it happened.
It's not a SaaS. "Uptime" for a linked library is stupid.
They're using static initialization and it bit them in the ass again. They should stop using it because it is prone to things like this which result in apps not launching at all. I'd like to see them commit to stop using this method to initialize themselves, and take responsibility for the crashes. Then, IMO, we'd be good.
In a scenario like this, I'd hope they'd want to make this bug's recurrence impossible.
Here is our Android config in case anyone's interested:
<meta-data android:name="com.facebook.sdk.AutoLogAppEventsEnabled" android:value="false"/>
<meta-data android:name="com.facebook.sdk.AdvertiserIDCollectionEnabled"
android:value="false"/>
<meta-data android:name="com.facebook.sdk.AutoInitEnabled"
android:value="false"/>Like they work in many other companies.
In any case, this is not about being bright but about being diligent.
What has this industry become? How are we so goddamn inept at writing software? This is an industry where we can automate testing - not many other industries have this capability - and we still don't have a simple test regime of "check if the app opens" for any of those apps. Somehow, at the most valuable software companies of the world, nobody has set up a system that makes sure that you can't introduce a change that renders these apps unusable?
WHAT THE FUCK.
I have always been afraid of software engineering becoming a trade similar to medicine or law or (regular) engineering where significant barriers will be put up before you can enter the industry, but this sort of shit makes me feel like it could be for the better. I will have to go and get a degree, but that's the price I have to pay I guess.