Our firm subcontracts to other firms. I am among other thing resident Sysadmin which means I get to do everything related to AWS.
Out of 6 contracts we had, where firm had their project on aws I got root credentials in 4 cases. out of 4 i only managed to convince 2 to take their account security more seriously and lock things down.
What I am trying to say is that what happened at Ubiquiti is far more comon than people realize.
Because when all is said and done you still need someone who thinks like sysadmin.
There were a lot of smart and experienced people at Ubiquiti when I worked there.
Nick Sharp manipulated his way into total control over everything and wouldn't let anyone outside of his isolated team touch it. Nick was hired out of his job at Amazon because he was supposed to be the AWS expert. He used that to lock out anyone but himself and a trusted team.
It didn't matter that others knew better. Nick controlled it.
This always cracked me up. From what I can tell, he was a mid level dev on the Alexa web api team. He knew AWS sure, but he did not have the cred at all to justify the position and responsibility he was given at Ubiquiti.
Management abdicated its responsibility.
Ubiquiti now has public evidence their security posture is inadequate and there will be pressure for them to demonstrate they have changed this situation.
It’s very difficult to completely prevent malicious actors engaging in deliberate efforts to infiltrate and plan actions like this once they have any measure of access and trust. What matters now is how they respond to it.
I had never heard of GuardDuty so I got curious. It doesn't seem like there's a free tier available for it: https://aws.amazon.com/guardduty/pricing/ In fact at larger scales it seems pretty expensive even.
Thank you for the correction.
All the same the point I was trying to drive is when the service is incredibly effective, both in practice and cost, it should in nearly all instances be implemented.
Edit: I wanted to amend my statement but the window to make those changes has passed.
That would also point to the CEO being too powerful when it comes to security, which is also a knock against the organization, but in a different way.
> * Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.
The "Cloud Lead" that Nick took over from gave zero fucks. He ran all the AWS stuff for Ubiquiti under his personal AWS account. Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle (my own personal opinion of Robert is... not the greatest).
One thing to understand about Ubiquiti (at least during those times) is that the company had zero C-level execs. There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.
So when Nick came in, a very... let's just say "forceful" personality, he immediately won over Robert and ended up with carte blanche over pretty much all of Ubiquiti's cloud accounts. Which were basically... everything. All the UniFi Network services, UniFi Protect services, you name it. If it was connected to the cloud in any way, Nick had access to it.
So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.
Also, for some perspective, at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to. And they were in plain-text. So... yeah.
> * Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.
See above for the quality of security processes and practices this company had in place.
> * Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion
See above. (re: "god") Nick answered only to Robert. And he'd already successfully hoodwinked him. He could do whatever he wanted. Eventually he fell from Robert's good graces, but seeing as Ubiquiti as a company didn't really have a ton of checks and balances, he kept his god-level access far longer than he should've.
> * Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.
Simple. Fear of share price falling. I was constantly given this as a reason we couldn't be transparent. Not by Robert, nor where he could hear. But it was pretty much well known that the company kept shit quiet for fear of the share price dipping.
> All the AWS configuration I'm speaking of above, I would describe as Security 101.
To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.
Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.
And that doesn't really even cover most of it. That year took a toll on my physical, mental, and emotional health, not to mention put a crazy strain on my marriage. I'd rather honestly forget it, but the schadenfreude of what's going on is too delicious to ignore.
This is frankly worse than any of this other news. So there's essentially zero trust associated with the code signatures since any employee, past or present, can sign a payload. Wonderful.
Also, even though they may have had read access, not many knew it existed. But it wasn't super hard to find (I stumbled across it basically).
Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.
But either way it definitely wasn't clear that opting out meant data was still collected. Super sketchy.
We didn't have read access until Nick Sharp and his team took over GitHub permissions and gave everyone access. Wonderful security work.
> Oh and then there the whole metrics collection debacle, where the controller basically phoned home about the topology of every network that it managed. Even if you opted out. Opting out just meant they fuzzed your ID so any given record couldn't be linked back to PII. Which may or may not be legal, IANAL.
Nick Sharp was at the core of this too! He built the 'trace' system to collect all of these metrics and had all of these ideas about how to secretly collect the data in ways that would be hard for people to detect.
He pretended to be a principled person who stood for security and privacy, but whenever he saw an opportunity for political gain he abandoned all principles. He was the only person I knew at the company who was enthusiastic about collecting all of that data.
He basically dictated that you couldn't use any kind or repo+deployment pipeline except for what his team was building. Which wasn't actually functional for like 8 months. So we never even got a dev or staging tier to test against for months.
And then when I ended up with access to push things along, the actual apps for the trace system we're... not well implemented.
Ugh... I could bitch about this stuff for literal days but I gotta drop my kids off.
The quality/feature set is there and the software is well designed, even if not quite as networking beginner-friendly as Unifi has become. Mikrotik's RouterOS can do much the same tasks as Unifi's management console, and can configure for auto-adoption of APs/other hardware in the Mikrotik range just like Unifi does for their own hardware.
Most competitors (I see Aruba suggested) are priced much more into the enterprise/business buyer realm. Unifi has generally been keenly priced in this market, their latest Wifi 6 APs are just 99 dollars each (when in stock of course...). Microtik's pricing is generally comparable or cheaper than Unifi in my experience.
edit: Secondhand Ruckus/Brocade switches are solid, at least on the 7000 series the evaluation key has no time limit so you're not license-limited in what you can do with them. Switches are mostly <$250 on eBay if you're buying an ICX7150, ICX7250, etc. Yes, that includes PoE models.
Is it just software that's UniFi's weakness? Anything wrong with the hardware itself? I've had quite good luck with UniFi in my home myself but perhaps I'm not using all the features...
This right here is why I'll never use Ubiquiti gear. These devices are so obviously backdoored and like swiss cheese, they offer the complete opposite of security. Thanks for sharing the true facts.
Nick's whole strategy was to find a problem, exaggerate it as much as he could get away with, and then offer himself as the hero who would fix it all.
He exaggerated or lied about everything he wanted to use for political advantage, right up to the end where he fabricated a hack and used Krebs to exaggerate it as much as possible for his own personal profit.
You have to realize he did the same thing during his time at Ubiquiti: Found problems he could use for political advantage, exaggerated them as much as he could get away with, and then amplified his lies until they were gospel. A lot of what you're saying has some roots in truth, but I can tell you have the exaggerated Nick Sharp version of events.
> There was Robert.... and then nobody knows. I asked repeatedly why we didn't have a CTO, or a COO, or a CFO, or CMO or ANYTHING and I got nothing but shrugs and "idunno" as a response for the whole year I was there.
This wasn't some big mystery. Everyone knew that Robert ran everything as CEO and the legal, marketing, and other teams operated out of the New York office.
> Nick came in and started putting "proper" AWS structure and security in place, primarily by scaring Robert (the CEO) into giving him the keys to the castle
Nick was hired specifically to run AWS. That was his job from the beginning. The old cloud team quit and Nick was recruited from his job at Amazon because supposedly he was an AWS expert.
The incident where he scared the CEO was the first of his political games to exaggerate or fabricate security incidents for political gain.
> So why wasn't anybody else notified? Simple. Because he was basically "god". If anybody was gonna be notified, it would've been Nick. He was the top of the totem pole company-wide when it came to AWS.
Yes, this. All of these news stories are missing the point that Nick was the cloud lead. You don't have to believe anonymous commenters. His LinkedIn profile will confirm it. He was recruited out of Amazon to lead the cloud efforts, but he was in over his head and had severe personal issues.
> at that time Ubiquiti kept all the hardware signing keys in a private GitHub repo that every employee had read access to.
This is another Nick exaggeration. It's true that older devices had hardware signing keys stored in a Git repo before the system was updated and keys rotated. However, those old keys were only accessible by a few people until Nick and his team took over GitHub and restructured permissions with the web portal they built themselves. In the process they made too many repos accessible to too many people.
> To keep with the metaphor, Ubiquiti couldn't even get Pre-school level security in place, much less 101. I have no idea how something even more massive hasn't happened yet. Must be dumb luck.
Ubiquiti's overall structure is far from perfect, but you were only there during the Nick Sharp era. Ubiquiti had a lot of people who took security and proper practices very seriously before Nick Sharp took over everything, but it was also a distributed company with a lot of isolated divisions. Nick Sharp got into power by taking the worst and oldest parts of the company and convincing people that everything was equally bad and that only he could fix it. If you got your security information from Nick Sharp, you'd think that Nick is the only person who can do anything properly at the company.
> Speaking of, by the time I left the company, the team that was handling the door entry-way systems (UniFi "Access" I guess) had been caught with numerous security issues, not the least of which was logging user credentials in plain text (not just storing, but logging, in response to authentication events). They were also based in China and subject to Chinese laws around government access, so take that how you will.
I also heard that, but I think it was just incompetence on their part. Nick was pushing the conspiracy that they were doing something with the Chinese government, but it doesn't follow that they'd do it by sending the data to AWS servers under his control. I think they just made a sloppy prototype to impress the CEO and got caught doing dumb stuff. I do blame the company for not cutting that team off, though. They had no idea what they were doing other than their ability to put together quick prototypes to impress the CEO.
If you're telling me I worked there at literally the worst possible time frame, I'd believe it. I may have my experience skewed through the perspective of Nick's influence, but tbh many of my issues were unrelated to him or his sphere of influence.
The C level thing may not have been a "big" mystery, but it was to me, and as somebody who was running the dev of a flagship software product (UniFi) it set off alarm bells that nobody I talked to could explain who was handling the roles of those execs. I'm not exaggerating when I say I effectively got "I dunno" as a response when I inquired, and I dug.
It is good to know, though, that what I experienced wasn't chronic for the entire company's existence.
To clarify on the China thing, I wasn't trying to imply that anything nefarious was actually happening. Just that it warranted some scrutiny when a security focused product was being developed on the Chinese mainland and by a team of Chinese citizens that are subject to CCP laws. Given some of the things that have happened around that country's involvement in tech in recent years, I don't think such scrutiny is unwarranted, especially when the team has a track record of security "goofs".
I’d never use Ubiquiti’s switches, routers, etc. There are great alternatives there. But when I went to replace my APs earlier this year I still could not find anything less shady that still did what I needed.
Is there an alternative that does PoE w/ multiple APs that hand-off well? And decent hardware…
TP-Link has a similar offering but with similar problems.
OpenWRT on [consumer gear] is not an answer here. More effort goes into the plastic than the hardware. Never dealing with cheap NICs, bad SoCs, inadequate memory, garbage drivers, etc. again.
With that said I'm moving away from Ubiquity after years of broken pointless updates, years long outstanding bugs, and after this thread obvious massive lacking security.
I just only turn my controller on when I’m working now and occasionally have to re-adopt.
And alternatives likely have similar issues. It’s what naturally occurs when small businesses get large, and especially when companies go from embedded to SaaS development.
Makes me wonder, why are they even settings? Why aren't they just always on?
"Why didn't they just ..."
Where the problem is that the "..." is the subset of non-default security configurations that would have stopped this specific insider(!) attack. They never mentioned that:
- You can't predict which attack you'll be hit with, so you have to implement every non-default / optional security setting or feature in order to be "protected". This is a metric ass-ton of work, typically on the order of multiple man years of up front effort, and then with ongoing multiple FTEs required (e.g.: for separation of roles).
- While you're not actually under insider attack, this provides zero business benefit that can be measured in dollars.
- The people implementing the protection against insider attack are the same people that the protection would typically be designed to stop. No one person can be trusted to do this. Not even any one team. Just organising the Byzantine security required for this is a project manager's nightmare.
- Insider attacks by senior technical staff are stupidly difficult to protect against. I've never seen any org that could survive an intelligent, motivated attacker that already has significant admin rights. Even the FAANGS would take a significant "hit" in this scenario. Ubiquiti is nowhere near their scale or capabilities in this space.
- All that stuff described like "QR codes in a safe" are hilarious to me. Every time I've tried to implement even 1/10th of that, everyone who ought to be responsible for the secure handling of this kind of key material ran away screaming from any personal responsibility. Literally nobody ever wanted to deal with anything even vaguely similar.
- Many of complex delegation systems and IAM technologies suffer from permission-role inversion. Often the CEO/CIO/CTO has virtually no access to the technology stack, but the junior intern from the subcontractor has the keys to the castle and could delete the whole org for shits and giggles. But, you see, he's not "allowed to". But he has the physical permissions. That he's not supposed to use. Because he was told to. You see?
- Now you're going to say that the senior staff should have the admin rights, and delegate the appropriate rights to the junior staff. Oh, you sweet summer child. That requires responsibility! (see above). It also requires that they learn, manage, and monitor something as technically complex as AWS IAM. But you see, senior people are busy people. They're busy with meetings. With memos. And more importantly, they're busy playing politics and jostling for the next promotion. The fiddly security rule stuff they just delegate to the juniors. That's the ticket to the next pay grade!
You really cannot have good security if the goal of your “cloud lead” is to ransomware you.
So yes - they did not have good security so they hired him to fix it :)
Obviously we have to take the linked comment at face value.
Oh my sweet summer child. You haven't worked in large organizations with thousands of employees before, have you? Surely not if you think this is "Security 101".
At around ~1,000 staff though you quickly lose "insight" into platform security as it's delegated out to sub-orgs within a larger organisation.
Though I've not worked at any FAANG companies where I'd hope security is... erm, of a certain 'maturity'. E.g. Google and BeyondCorp.
But I've no desire to rehash leet code exams or get back in that space so likely will never go there :).
* Why was it so easy for a lead engineer to get access to a root AWS user without anyone else being notified? I.e. AWS GuardDuty provides FREE alerting for when an AWS root IAM account is logged in or used, this account should be under lock and key and when used, confirmed and audited by relevant persons or teams.
Start edit/
* Furthermore on the root account being easily accessed, the root account in the companies I've worked at had MFA enabled, and the QR code is locked in a safe only accessible by two people agreeing it needs to be accessed in a break glass situation, where warranted.
/End edit
* Why was he also able to delete critical CloudTrail logs and reduce their retention to 1 day? I.e. These logs should be in a S3 bucket or other environment where such changes cannot be made. Alternatively, they should be shipped to a redundant service that manages this risk to prevent data deletion
* Why did Ubiquti not announce they were compromised sooner? The hack started in early December, Ubiquiti noticed the compromise on Dec. 28. Ubiquiti told the market on January 11th. Is that a satisfactory turn around? Giving them some credit for the XMas break I'll say this partially understandable.
All the AWS configuration I'm speaking of above, I would describe as Security 101.
Most of these settings can be set and managed from AWS Organisations for free, and backed up with alarming and alerts for Guard Duty trivially. That a company of Ubiquiti's size and maturity had such basic risks not managed is still a concern.
I understand AWS Organisations can be difficult to set up for legacy AWS accounts, but even with that said, setting the alarms and monitoring up that would help manage the risk associated to the questions above is not difficult and should have been in place.
That Ubiqitui would only find relief ultimately from the developers poor OpSec rather than Ubiquiti's own security policies and procedures provides a commending perspective of their internal security posture.