There should be full transparency on this and all the open source eyeballs should be able to study and scrutinize this. If not, anyone will be able to get away with data theft tomorrow saying, "my server got hacked". What will cause them to not do it except some sense of personal ethics which is rapidly degrading these days?
Private bridges, have owners who give an accounting to no one when they collapse. At best, if you have standing you can sue them and they will defend themselves by giving an accounting for why it’s not their fault or they did their best.
GDPR is a good idea but there must be supervision for it being applied as defined.
Are other companies any better? It seems like now it’s assumed that all of these companies will eventually be hacked, so if you use them you need to have systems in place to mitigate damage. And if you don’t use them them then you need to have mitigation strats anyway. Basically either way you’re screwed.
I'm bringing this up because the circleCI blogpost says that the attacker did memory-dump encryption keys from a running process. See https://circleci.com/blog/jan-4-2023-incident-report/
So even if they were using hashicorp/vault, the attacker could probably still have been able to mem-dump vault's process.
Set up within seconds using a few lines of cloud-init: https://gitlab.com/21analytics/gitlab-runner-cloud-init
Most of the time, it's also cheaper and maintenance is close to zero.
It was easy to take an existing open source terraform module [0], modify it a little for our purposes, and deploy runners that provision job executors in our tightly controlled VPC. It is very simple and everything is open source so you can really understand what it is doing. In our setup, all secrets are issued from our private vault instances. Most of them are short lived or one time use when possible. We are looking at moving our stuff to EKS now as well.
Overall, it was pretty easy to get going but we have the resources to do this. I could see why a small startup would outsource this to someone like Circle CI.
[0] https://registry.terraform.io/modules/npalm/gitlab-runner/aw...
The beauty about CodeBuild is that there is no “lock-in”. All it is fundamentally is a Linux or Windows Docker container with popular language runtimes and a shell script that processes a yaml file or you can supply your own Docker container.
You just put a bunch of bash commands or PowerShell commands in the yaml file and it runs anything.
The Docker container and the shell scripts are all open source and you can quite easily run them locally.
I could see outside of AWS keeping your Docker containers for your specific build environments in a local repository and doing all of your builds inside them using Jenkins using a self hosted CodeBuild like environment.
https://github.com/aws/aws-codebuild-docker-images
https://docs.aws.amazon.com/codebuild/latest/userguide/use-c...
For a “batteries included” approach though, I really like Microsoft Azure DevOps Pipelines.
I’ve even done a couple of integrations between Azure DevOps and AWS when we had clients that are Microsoft shops.
For AWS, if you use CodeCommit (AWS git service), all access is via IAM and granular permissions. If you integrate with Azure DevOps, the AWS credentials do have to be stored in a separate MS hosted credential storage.
CodeBuild also supports at least Github natively.
I’m not shilling for AWS. I have an MS development background (.Net) and only have “DevOps” experience using AWS and Microsoft tooling.
The more Actions matures, the harder it is for them to continue - their market will be legacy projects already on the platform, and perhaps the new projects from companies already using it for older projects (the one reason I can think of to prefer Circle over Actions for a new GH-hosted project).
Defending against internal threat seems like a losing battle that can only be mitigated, slowing down more than preventing an attack.
Circleci's business model is hosting CI runners for you, so of course they need to be able to decrypt the data, and if you have access to dept new infrastructure, you likely have permission to read encryption keys (or deploy new infrastructure that can read said keys and then use those machines to get the keys).
What CircleCI _have_ done is set up enough logging and auditing that they were able to figure out who was compromised, how they were compromised, the time frame and the resources they accessed, which IMO is about as much as you can ask for.
They also point their finger at anti-virus for not detecting the malware. That is a lame excuses. Professional malware developers will check if their product goes undetected by the major anti-virus providers. Anti-virus should not be relied upon to protect against sophisticated attacks
> This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software
That's not blaming that's telling people that their antivirus didn't detect it. If that wasn't there, people would be talking about why they didn't use X antivirus which would probably detect it.
> CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access.
This is a significant move of the goalposts. Of course CircleCI messed up, but to go back to the OP's point of "nobody should have that much access to production", well that's just not true.
However, if they need this, you need a detection mechanism. It doesn't have to be advanced, a threshold on the number of production keys accessed per day is sufficient. It should raise an alarm to the manager in question, who can then confirm this is expected behaviour given the task assigned to the employee in question.
Giving access to customer data to your staff by default is a design decision and can be avoided. However, in the end, it is a cost-benefit analysis where you have to decide how much you care about your customer's security. I have worked for enough start-ups/growth companies to know the value put on customers' security is sometimes shockingly low.
This is why Antiviruses are totally useless, they're incredibly easy to bypass.
Not all attacks are targeted and AV can detect abnormal behaviour. Regardless, having the operating system do proper sandboxing is much more valuable than trying to fix it after the fact with AV.
The name is obviously misleading though.
There are lots of tools for key management, but lots of companies don't care about it.
They could run everything in Nitro Enclaves or similar, that require multiple people to deterministically compile and sign new software for, and release secrets into.
I design quorum controlled infrastructure for a living, mostly in fintech where no single human can ever be trusted. You 100% can run infrastructure that, barring a platform 0day, can prevent any single human from having access to the memory of customer workloads and secrets. Customers likewise would encrypt any secrets or code directly to keys that only exist in the enclaves.
CircleCI had negligent security design, but all its competitors are just as bad, to be fair.
Building with security in mind makes you last to market, which is unforgivable in our industry. Getting hacked however is just considered a cost of doing business.
Ominous. What about GitHub? The amount of secrets I trust them with is practically all my life force at this point.
GitHub/NPM have historically failed to support supply chain integrity practices in their public offerings such as hardware anchored code signing, signed code reviews, reproducible builds, multi-party approvals, etc. It is reasonable to expect they are not doing any of that internally either.
Assume any secret you give GitHub will become public knowledge and act accordingly.
The good news is there is never a reason to trust a VCS or CI system with high value secrets. They should never ever need any power beyond running tests, accessing a test environment, or sending notifications.
> Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data.
...what? why did this engineer have access to everything? Does CircleCI know what minimum access policies are for?