I go through security reviews all the time with them, they have so many security processes that you get dizzy and on paper everything looks fine. They create security zones with massive risk reviews, but for some reason those security zones then share subnets with the entire LAN.
They also have a default configuration which makes everything access the standard intranet directory once its deemed secure. Enterprise security tools like Cyberark are deemed more secure than say yubikey HSMs, which may result in root ssh being enabled in a lot of settings. They have system configurations that are done with massive Excel sheets. Their cloud VPCs basically only have one risk profile and once its deemed secure it gets access to things in the intranet. They also vehemently refuse to do threat modelling when designing anything.
These people can tell you so much about the theory of security by heart that it will make you dizzy but then won't actually understand the underlying problems.
And the offenders are always the same, advised by Accenture, Infosys etc.
I've thought greatest failure of many professionals in this field is in the "protect the network" perspective rather than "protect the data". While many of them fess up to "we can make it difficult but not impossible" to breach the network, that is not evinced by the protections instituted.
If companies actually understood that they WILL be hacked, the focus would turn to protecting the data. Actual resting data protection would allow a "I don't care if I'm hacked," posture. Either behind encryption, VM's, segregation, or architectures, preferably all, if data is actually protected, then a hack can be weathered. It's still a pain if the computer-touchers have to rebuild and reload, but that's what you pay them for. If the data is protected, a hack is just a painful exercise rather than a newsworthy event.
I do understand that segregating (through protection and architecture) data is difficult, but I do not understand why it is not the focus.
Imagine if the solution to Covid was "the virus can spread to anyone, we need to replace all humans"
It's only prohibitively expensive because most of these enterprise tools and servers are actually very much focused on manual setup.
That's quite interesting. Where can I read more about that ?
https://www.csoonline.com/article/3247848/what-is-zero-trust...
Protecting the data means encrypting it (which Google also does) such that the client needs a key to decode it.
But if the client is hacked, the hacker can get their keys! so encrypting the data isn't a magical cure-all.
Moreover what if a hostile party constantly hammers-disrupts your IT, letting you teams "rebuild and reload" 24 hours/day (in other words you don't have any information system anymore)?
None of this is rocket science and these people probably aren't stupid, so somehow, somewhere, something is going horribly systemically wrong (incentives? Training? Organisation? I don't know).
It's not like Colin Percival or Theo de Raadt , perfect as they are, could just audit and secure all of the Fortune 500.
It doesn’t tell me whether they understand how it all works together, or if they understand the organization’s environment, or if they are a good worker.
I don’t hold it against people who fail (I’ve seen good people fail the test) or who don’t have it - I just have to ask a few more probing questions to ensure they know the tech I’m discussing. But I don’t outright ask if someone is a CISSP, so typically I ask the clarifying questions anyway so our understanding of the problem is accurate and aligned.
And cert or not, I’m still more interested in whether you know what you’re doing than what you put on your resume.
It's just a cram-and-forget vocab test, it doesn't mean anything other than that they could afford the training and the test.
it follows that
"Centralized hosting and management (SaaS, PaaS model) has the advantage of insecurity at scale."
It's as if someone sells you a laser that shoots intruders and tells you, you can leave the front door open from now on, but that laser only works 1 in 3 times.
But not every person who has executive oversight of operations thinks like I do, and all of them are represented in the company's finances as a 'cost center' that is second only to Payroll in terms of how juicy a 'cost reduction' target it presents.
So when the going gets tough, the company cuts back its IT budget.
There is a common misconception that CFO's fiduciary duty to their shareholders determines that they should protect the long term stability of the company, but now most shareholders are in the company for 6mths tops. The duration of a CFO's fiduciary duty is arguably about 6mths out. The devastation of large companies in the economies of the west since 1980 is a testament to this.
Fake Numbers: sales "generates" 100,000,000 in revenue and "costs" 1,000,000 however IT "generates" 0 in revenue and "costs" 2,000,000
So to an bean counter, clearly the "investment" in sales is better because they make money for the company, where IT just costs money...
Thus IT is often treated like a utility service, say something like custodial services, where they want to find the cheapest way to keep the bathrooms not disgusting... not clean mind you, just not disgusting
If you are looking at US curriculum, my experience is that you will see the discussion in terms of dollars and their "flow" through the firm from the customer and perhaps ultimately to a bank account (in the case of having positive cash flow) or how much 'short' the company is when it comes to a negative cash flow situation.
Understanding the cash flow dynamic for a company is critical to the company's success. If a company does not understand how they make money and how they spend money, they will not be able to manage themselves to a sustainable level.
As with engineering, it is a simplification to group "like" costs, and "like" revenues together. So for example all the money made by extended warranties and charging for repairs might be grouped as "service revenue." Similarly, all the money spent on leasing office space might be grouped of "real estate costs."
Every accounting program I have seen (and it isn't exhaustive of course, just consistent in my view), facilitates this grouping of costs into larger and larger groups. Depending on the size of the enterprise, the manager at a particular layer who had "profit and loss" responsibility could see a small number of these groups (which I have only ever heard referred to as either "revenue sources" or "cost centers") and they could get an idea of the health of their part of the business by seeing if their margin target (total_revenue - cost) / (total_revenue) was being met.
And at the managerial level, they typically would split their activities into ones that "improve revenue" or "cut costs." Doing either increases the gross margin which is what they are measured on by their manager, whether it is another person at the company or the board of directors. Because these are fundamentally an accounting thing, increasing money coming in by say raising the price of the product or restructuring pricing plans is called "growing top line revenue" because that usually the top line of a financial report. And when they cut costs or improve efficiencies so that they can make more product for less money, that is called "growing bottom line revenue" because the amount that gets subtracted from the top line is reduced and so the number at the bottom of the page gets bigger.
Finally, nobody is an expert on everything. And the larger the enterprise the wider the expertise needed to understand the costs and expenses of that enterprise. What is worse, is that sometimes the people in that role were experts at one time but the area where they developed their expertise has moved on and so they believe they know what is the right answer and don't bother to check. And sometimes they don't know the right answer but don't want to "look stupid" and they buy all the reasons the sales guy gives them for using their product as pass that along as justification without knowing the risks.
It adds up to a bad choice. And when that choice is to move to open offices (for example) the impact of losing productivity in people who cannot deal with that environment isn't readily apparent. And when it leads to outsourcing something which wouldn't be outsourced, the error might only become apparent when you're suffering a ransomware attack.
Meanwhile, best practices are slow to reach the curriculum and so there is a lag between people doing things poorly and it being taught as a bad thing in business school.
It sucks, because I know my company is quite small but we take security extremely seriously (we have 9 people, 4 are security engineers, and the other 5 have varying degrees of experience in security). I think people might worry that, because of our size, we won't be as secure as a larger company. But the irony is that larger companies are often far less secure than us, because we've done shit right from day 1.
There's just not a lot of ways to prove it. Compliance is meaningless. You could get a pentest report, but it really comes down to who's doing the pentest, and so if your pentest becomes a public doc the incentive is to have them go easy on you - not to mention that lots of reports contain "findings" that are nonsense but a casual reader might misunderstand.
We plan to give talks and blog about how companies at our stage can do things that would make companies 100x our size jealous, because that's kinda the only thing we can do to really explain that it's possible.
I think it's totally criminal that companies ask for RCE on all of your devices and then push out some closed source C++ app that's probably parsing all sorts of random shit, reading poorly authorized commands from some C2, etc.
You can be compliant and buy insurance or you can be secure. Pick one.
Sounds interesting and rather extraordinary, would be great to read more on your thoughts on that. Do you refer to the Grapl blog?
To do the same is trivial at a small company. What would take years and lots of effort becomes something you can do in spare time.
Of course, we put considerably time into security, it's not just something that one does once in a while with spare time. The point is that we can go much much faster.
Embedding this kind of software deep in your internal networks/systems, with access to basically everything, is a recipe for disaster. I expect these sorts of supply chain attacks to get more and more common, they’re excellent back doors into basically every government agency and megacorp.
Let's say you are designing a system - any kind of system - with the philosophy that everything should be connected to everything else. Your first part goes in quick with no connections. Your second part goes in quick and has one connection. Your third part has to be connected in two places for it to work right, but that's not a problem. Your hundredth part has to be connected in a ninety nine places for it to work right, and now you're spending more time wiring than you are on making parts.
Then we ask, "what can we do when that happens?"
You have to put effort into the design of the system, reassigning duties and studying the nature of the problem it's solving, so that you lay down the connections along the true contours of the map, and not between every single component. Afterwards the next component you add has to be connected only to the three other things it's actually related to and you're back in business. This results in a period of time with no new features or even bugfixes, but afterwards you move faster.
Then we ask, "why do people call it debt?"
Because you pay interest on it when you have it, you run it up when you're short, and you better have a plan to pay it down or else you will go out of business.
Should I attribute you or someone else? :-)
Edit: added as a private bookmark to pinboard with tags:
technical_debt quotes by:whatshisfaceTechnical debt is like not cleaning your house to save a bit of time everyday. When you actually have to clean it, it's going to take longer than the time you saved. And until it's not clean, everything you do will be a bit worse because the house isn't clean.
"Remember when you were a student and didn't do the dishes, and then when you finally did them everything was dry and sticky and stinky, and it took you a lot of time to wash everything and you felt terrible? That's dishes debt. Technical debt is the same. When you make a change, you produce dirt in the codebase, and if you don't or can't take the time to clean every time, dirt accumulates."
Like when you need to fix all the support columns in your building, but instead of spending millions to take them down one at a time and replace the corroding rebar inside, you just patch over the exterior cracks. They will look fine from the outside and get the job done on a day to day basis, but they hide structural problems and one day that debt will come due. Most of the time it's in the form of a giant project to finally fix everything, but sometimes it's catastrophic failure.
Often the most maintainable solution is simple and elegant, but it takes a lot of refactoring to implement, so a hacky, complex solution is implemented instead, because it’s faster/easier to implement. Such solutions tend to either contain bugs, or lead to bugs when built upon, and a lot of security vulnerabilities are basically bugs in hairy parts of systems that are hard to understand.
I should just edit the wikipedia page, but they won't accept edits from my current IP address.
There is significant cognitive load in understanding the code and what it does and why it does it that way etc. Keeping all the important bits in one mind is challenging and a lot of the little fixes can lose sight of the big picture in a way that comes at a cost and, over time, this can really add up.
Over time, different people may work on the code and have different reasons why they made different choices and at some point it may all stop playing well together. Then there comes a point where someone needs to try to reconcile all the different bits and understand what needs to happen and why and rebuilding the entire catalog of goals, features, etc. in one mind at one time so someone actually understands it all and gets it right is a substantial future cost that only grows as you keep delaying that step.
(From one lay person to another -- I do write code, mostly html, and run some web projects, mostly blogs and Reddits, and spend too much time on HN. So technical debt isn't alien to my experience though I'm not really a programmer.)
(I jest. A bit cynical but that's often how it comes out in practice).
1. Knows the wider requirements but isn't involved in the implementation. They can't fully specify what's needed without doing the actual implementation; the map is not the territory.
2. Is told the broad requirements but probably can't grasp the things they aren't told in the imperfect spec. So under time pressure, in good faith, they do the simplest workaround possible.
3. Is given the next set of requirements. Instead of re-engineering the original design, under time pressure and in good faith, they add a workaround for the workaround.
Each new workaround is "tech debt".
When you add the next feature you now have to deal with multiple levels of complexity not in the original spec.
Understanding the actual implementation now takes more time than expected. The chances are that no one fully does, which leads to further mistakes and workarounds. So more tech debt.
Either you pay the debt down and re-engineer or you pay the compounding interest forever.
...and so on. Each new level of complexity gets harder and harder to understand and debug because no one really knows how the real design, held in the actual works.
You can cook a bunch of times ignoring these secondary results but over time cooking will be slower and of worse quality due to the mess and at some point it will be impossible (too dirty, no usable pots and pans, etc).
"Softhwair", if you will :D.
If you did it when building a plane, and it killed people, you would go to prison, but in the technology industry, this is acceptable as "the cost of being first"
Anyway, here's a good introduction: https://en.m.wikipedia.org/wiki/Technical_debt
Redteams / hackers now target the infastructure, because it's way easier and they're more outdated in regards of code, stability and used libraries.
Most enterprise-grade VPN solutions still use OpenSSL from decades ago, and most of their fixes (even if they react to CVEs) are always too late.
As SOCs need VPN access because they are usually not on-site, especially at larger corporations...the result is when you exploit the VPN gateway, you are the new administrator because you have a large time window until the SOC team arrives on-site. These couple hours are usually everything you need as a time window to raid the place, install and run ransomware, and clean up afterwards.
From a cybersec perspective I cannot even begin to write how stupid it is to put literally all your company's value in the hands of a single security company - which is legally not responsible for anything by contract. Security through obscurity never worked, why should it do in this case?
Last year showed that we desperately need an open source OpenVPN based graphical and scalable alternative that uses a standard TOTP based token generation mechanism and not some proprietary crap for authentication.
Using a token generator with embedded analytics was just wrong in the first place, but...yeah.
Personally I'd love to see better Wireguard support and adoption outside the Linux world.
So when I became CFO I fired them (outside company), not just for this, but it didn’t help.
It’s bad software. 24/7 full low level access is exactly what it is. We had an add on that stored admin credentials in a JSON… so looking back on that, it seems this should have happened sooner.
https://doublepulsar.com/kaseya-supply-chain-attack-delivers...
I discussed this with their security team leads, and they answered with a straight face that it's okay because they had to spend their budget before the end of the financial year.
And everyone knows this is how it works too – so the Powers That Be keep setting the wrong incentives too.
This is why I never worked for a large Enterprise company or government agency. I'd go crazy.
Security by obscurity is laughable nonsense. We should all be demanding transparency in hardware and software from our vendors. I'd pay handsomely for it.
Any code executing in privileged mode can bypass security, and is therefore inherently part of a system's trusted computing base (TCB). (Linux is a monolithic kernel running in ring 0)
Most companies are not Linux contributors, they are trusting the kernel developers to write bug free, secure code.
Minimizing the TCB and opting for an auditable open source TCB are really useful concepts in security.
But the cause of these breaches is much more trivial than what you are worrying about: these companies are basically installing whichever piece of software can decrease their costs without thinking about what they’re doing.
>Any code executing in privileged mode can bypass security, and is therefore inherently part of a system's trusted computing base (TCB). (Linux is a monolithic kernel running in ring 0)
It's way nicer to be able to look at the code running in Ring 0 =)
There was an AV integration, and then Kaseya changed to Kaspersky. I don’t remember what the prior AV software was.
I always thought it bizarre we were actively installing AV software from Russia on banking and medical office PCs.
I viewed it as safer to buy products from anywhere other than someone that has ANY potential at all to go to war with the government of the country I live and work in. I really hope it never happens, but 'cold war' tensions might be waged with little cyber attacks and that software came to mind as a risk.
I wouldn’t run K, but I know from experience it’s actually effective.
- Can you articulate specific reasons to buy anything beyond the default windows defender?
- If anyone went to an actual war with the US, would the source of your antivirus software get even close to top 5000 things you care about at that point...
And yeah there's a decent chance if the US went to war with another country it might not impact the majority of US businesses very directly especially in the short term IRT their IT plans. McDonald's kept selling burgers when we invaded Iraq (multiple times). Ford was still producing vehicles during WWII. There have been lots of military engagements the US has been involved in where things in the mainland US weren't massively affected in day to day operations. Who knows what some potential future war with Russia would look like. Would it be a true head to head war with tanks rolling, fighter jets scrambling, cities bombed? Would it be more skirmishes testing how far the other would really go? Would it just be escalation of supply chain attacks and attacks on infrastructure to weaken the other? Of course this greatly varies based on the specifics on what that potential future war looks like, it would be naïve to think wars will always look like WWII, Korea, Vietnam, Iraq, etc from a US mainland perspective.
That being said, RMM tools have plenty of examples that they need to beef up their security practices or get replaced.
It's been wallpapered over as just cutting unnecessary expense for too long.
I get it at a pretty deep level individually but even knowing this I make enormous mistakes.
It sounds to me like if it means anything, it's denying that any probability depends on the exact dimensions of your ignorance.
There is no statistic that applies to everyone, unless that person is a completely generic person with no known qualities.
1) How secure is the software? Where are the audits?
2) If your software compromises my business, how much of the losses I incur as a result will you cover?
Is your opinion of their products the same?
On the other hand, all of the other networking HW sucks just as much. E.g. here are Netgear vulnerabilities published just this week: https://www.microsoft.com/security/blog/2021/06/30/microsoft...
IMHO, the worrying things about Ubiquiti at the moment are:
1. Their handling of the security breach/downplaying/whistle blowing fiasco which came to light some months ago. Check our Troy Hunts podcast from around that time.
2. Requiring a cloud account to manage your local device. Everyone seems to do that these days. It's not impossible to remove the cloud account management but it is an extra post install PITA step to work-around. And has some consequences if you do.
I'd like to see if they've learnt their lesson from at least the first point and become less opaque security-wise going forwards. Not sure their security is passing the smell test at the moment.
Well then I guess we will have a lot threats from hackers for days to come.
Yeah, sure. We should have a person on each of hundreds of sites whose only job is to check manually every router, switch, and vending machine. Maybe in the best HN traditions you will train the necessary workforce in a weekend?
> tools that have extreme low-level access to networks and systems
The emphasis being on the low level access. The solution is not having hundreds of people checking things by hand (though I'm sure that could contribute to security). The solution is more privilege separation; so that when the "remote monitoring tool" is compromised, not every part of your infrastructure is also compromised by default.
Having hundreds of people in each location whose only task is to do a boring monitoring an very occasional management tasks is a waste of your resources and their intelligence. We do automation to escape doing stuff that we can but which are to mind numbing. The illusion that every one of those hundreds of people will do their job to the necessary level of quality and without lapses of diligence is optimistic to say the least. Doing automation badly is not a reason not to do automation at all.
Management is always going to have access, so maybe you should not enable remote management access of everything to a centralized system? Make it lean and secure, possibly segmented, dual-factor, use HSM etc.
Monitoring - there is no good reason why it should have access to anything. Make it ingest only (use firewalls and reasonable protocols), and you've cut out most of the "monitoring and management" vulnerabilities.
At the same time, you don't have a way to solve a problem that your monitor has alerted you for. Every solution proposed includes either a person at the location who does the job manually or a way to connect to the network from the outside which is vulnerable to similar attacks as before with added costs and possibilities to mismanage keys and passwords.
Security vs convenience is a well-known dilemma that people very often love to solve in the most absolutist way.
As for remote management. I'm saying you should wisely choose what needs to be remotely managed, what doesn't, what are the foundations for your security and then balance it with reasonable methods to secure access. Which would probably not be "Kaseya VSA Remote Monitoring and Management" for all your systems and devices.
Yes, make sure you don't need on-site personnel to restart your web server, but maybe also don't expose management of your switch that you never reconfigure to your monitoring SW, and maybe use separate HSMs¹ or at least HSMs instead of the enterprise management system for the most important parts.
¹ e.g. FIDO2 ed25519 for ssh
I have the bad feeling that this achieves security through unavailability.
I mean, I'd be willing to trust security to Microsoft or Apple (I mean, at some level, you've got to trust the OS). But giving the keys to the castle to some mid-tier company is just a recipe for disaster, and the bad guys know how extremely lucrative these targets are.