Insufficient testing at today's Apple is not limited to software. They bragged about their extensive input testing lab [0] when the new line of Magic accessories was released, but the Magic Keyboard with Numeric Keypad launched last summer had all of its inventory pulled from the channel last month because users discovered that the model was so thin that its midsection bowed over time.
[0]: https://medium.com/backchannel/what-i-saw-inside-apple-s-top...
But since I don't work there, I have no good inside info. But just from gut feel, I don't think my anecdata is too far off the mark. Based just on the bugs made public, I just don't get the impression that there are testers at Apple whose sole reason for being there is to tear into a piece of software and break it. There was a bug a few weeks ago posted to HN that I commented on. I don't have a link without digging through my comments, but it was something along the lines of "how could a tester not find this in five minutes of exploratory testing?" This bug is similar. It would take more than five minutes, but were this my area to test I'd pick at it once in a while when I had a few minutes. As I pick at it, I wouldn't expect to find anything, but I've got a minute between builds, so instead of randomly clicking Facebook I'll randomly click this dialog. What did the dev forget? What weird state was not accounted for? Some kind of state overflow if I click the button enough times? Shove some Unicode in there, that didn't find anything; meh, maybe I ought to move o...hey, wait a minute. Did that thing just log me in as root?
But my gut says that Apple doesn't employ a lot of testers like that.
For example, I do not own an iPhone, but at work, I made a bet with my colleague (jokingly) that I could break _something_ on his phone in a few minutes.
I did not have his finger print or pin-code, so I was very limited, I even joked "I don't need that, give it here!"
Finding out I only had a hand full of options, I focused on the emergency dialer. As any good tester would be curious about, I wanted to check the max field length, so I entered digits, copy/paste it a few times, copy/paste that string, ("wait, no limit? Not even at 1000? why?") and so on, until I noticed the interface became laggy, so of course, I kept going.
Boom, suddenly back at the login screen, tried to open the emergency dialer, but got a full blank white screen, in the meantime the phone started heating up substantially. Since it was a new Phone (iPhone 7 with iOS 10.x I believe) and the dev getting nervous, we decided to reboot it. That fixed the issue. (Curious if this is still an issue in iOS 11.x)
TL;DR: As a tester this simple curiosity should be in your blood, and especially covered in behavioral tests when your software has been around for 5+ years.
All in all, it took me about a minute to break it, and around 5 minutes to get it working again. I was getting a bit nervous.
- Performance
- Usability
- Brute-force capabilities
- Error handling
- And in this case, Security, a bug where trying to log in a couple of times on the Login screen with an empty, or set, password.
A test scenario closer to this would be:
---
When I am on the login screen
And I enter 'root' in the 'Username' field
And I enter 'thispasswordisfalse' in the 'Password' field
And I press the 'Login' button '10' times
Then I should see the text 'Your password is invalid'
---
Please note that this issue is not just in the Settings page, it takes place on the login screen as well, that's why I'm shocked, it's such a core functionality, touching so many system components.
Actually, I've been wondering why I hear less about people working at Apple than at other big tech companies. It seems everyone and their mother work at Google or Facebook, but no so much at Apple. Do they have less software engineers, or their employees are required to be more discrete?
I know but a few that work at Apple, and of those few they strike me as less forthcoming than the multitudes I've worked with and know at Microsoft. I've wondered if part of that is because Microsoft previews/pre-announces just about everything, whereas Apple (mostly, and not so much anymore) announces it when the shipping trucks show up at the local Apple store.
So the outcome from the Microsoftie is, "it'll do this that and the other, but that's all I can say right now." From a recent conversation with an Apple employee: "they make me go in a special room to use the hardware, and I can't work from home. That's all I can say."
Probably more so, last I looked, Apple has considerably fewer software employees than the other big companies.
I don't think this is true. Apple, Google, and Microsoft all have on the order of 100K employees.
Yes, I believe so. I've heard there are strict requirements on even internal discussion. (Who you can talk to; about what; where.)
The only people I know locally that work for Apple are remote customer support folks.
They’re now retreating from that strategy: https://factordaily.com/apple-to-pull-back-development-work-...
[1] https://www.computerworld.com/article/2528936/mac-os-x/snow-...
It also marks the decline of the desktop UI to introduce increasing amounts of iOS-like behaviour and appearance to the detriment of a usable desktop. Like proper scrollbars etc.
While some nostalgia might account for holdouts, it was the peak of MacOS in the minds of many, including myself. As a developer, I've been quite disappointed by its direction and declining quality. For the amount we pay for this hardware, it's not much to ask for some basic maintenance work and testing to be done.
Also I think when most people think of Snow Leopard they're thinking of 10.6.8, at least that's the version number you always see get thrown around on the internet.
Unless you’re arguing Lion wasn’t major because you didn’t like it, but that’s an argument that proves to much, methinks.
Are there any "(tech) household name" engineers doing system-level work on iOS/macOS these days? It seems like Google and Facebook have a slew of them.
Of course, until recently they had Chris Lattner as well.
[1] For some iOS releases, they converted HFS to APFS in-place, report the results back to Apple, but did not write the APFS 'superblock' to keep the filesystem HFS+. It's quite a smart idea, because they got reports from millions of devices without actually switching them to APFS.
I have a feeling that anyone who does would get fired for commenting here about it.
Open software enables people to take a look inside to what is going on. It isn't a cure for bug free development.
Some security bugs exist in the Linux/BSDs kernels for a loooong time before someone notice and fix it (e.g., https://media.defcon.org/DEF%20CON%2025/DEF%20CON%2025%20pre...)
At the minimum, i'd say i feel apple release less innovating os versions while producing at least the same number of bugs.
I know testing is hard, but a company with Apple’s resources shouldn’t be making slip ups like this. It suggests some real issues such as lack of unit/automated tests and/or sufficient release testing, which pretty urgently need addressing.
Anyone got any inside scoop?