News for hackers. Stuff that matters.
Re:CircleCI says hackers stole encryption keys and customers... by ferminaut on Saturday January 14, 2023 @11:28PM (#34386728)
There shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them.

> Zuber said that while customer data was encrypted, the cybercriminals also obtained the encryption keys able to decrypt customer data.

...what? why did this engineer have access to everything? Does CircleCI know what minimum access policies are for?

Re:CircleCI says hackers stole encryption keys and customers... by bdcravens on Sunday January 15, 2023 @12:36AM (#34387000)
From the outside, it looks like many key infrastructure operations are running things like they are still startups, moving fast and breaking things. No access controls, engineers with machines they can take anywhere and install anything on, with full access to everything. Can't get in the way of moving fast, after all.
Re:CircleCI says hackers stole encryption keys and customers... by kodah on Sunday January 15, 2023 @02:36AM (#34387506)
Developer laptops are the next logical vector from zero trust networks backed by SSO. Reading between the lines it seems like the session was an SSO token. I think you're right about giving too broad of access. A lot of companies I've been at assign you to LDAP groups over time and if you've been there a while you'll belong to a lot of LDAP groups. There is no ritual to minifying access that I've ever been made aware of. We definitely need some developer oriented antivirus software, because moving to a life where every bit of software needs to be approved would be a literal nightmare.
Re:CircleCI says hackers stole encryption keys and customers... by lemon_bottle on Sunday January 15, 2023 @12:46AM (#34387052)
Unpopular opinion but I think whenever these 'hack' incidents happen, there should be a full disclosure and the company should be made to tell us what exactly was hacked right down to the nuts and blots, sql queries and server processes level.

There should be full transparency on this and all the open source eyeballs should be able to study and scrutinize this. If not, anyone will be able to get away with data theft tomorrow saying, "my server got hacked". What will cause them to not do it except some sense of personal ethics which is rapidly degrading these days?

Re:CircleCI says hackers stole encryption keys and customers... by stingraycharles on Sunday January 15, 2023 @01:24AM (#34387200)
That is not an unpopular opinion, people have been arguing on this forum for proper legislation around these types of security incidents and breaches for years.
Re:CircleCI says hackers stole encryption keys and customers... by foepys on Sunday January 15, 2023 @03:45AM (#34387881)
GDPR requires a full disclosure of which data and users are affected and when it happened, which I think is fair for the average user.
Re:CircleCI says hackers stole encryption keys and customers... by AYBABTME on Sunday January 15, 2023 @04:30AM (#34388077)
But that's like a bridge collapsing and only having to produce a list of who got injured and what their injuries were.
Re:CircleCI says hackers stole encryption keys and customers... by true_religion on Sunday January 15, 2023 @07:04AM (#34388793)
Bridges tend to be owned by the government which in democratic counties gives an accounting to the public.

Private bridges, have owners who give an accounting to no one when they collapse. At best, if you have standing you can sue them and they will defend themselves by giving an accounting for why it’s not their fault or they did their best.

Re:CircleCI says hackers stole encryption keys and customers... by pedro2 on Sunday January 15, 2023 @05:13AM (#34388283)
Those things need to be supervisioned. FlyTAP has gotten their entire rewards DB (if not all their DB) hacked and if it were not for IHBP I would never know because the company never told me. As far as I know FlyTAP never got a fine for ignoring its obligations.

GDPR is a good idea but there must be supervision for it being applied as defined.

Re:CircleCI says hackers stole encryption keys and customers... by BaculumMeumEst on Sunday January 15, 2023 @07:08AM (#34388806)
That opinion seems like common sense. Why do you think it would be unpopular?
Re:CircleCI says hackers stole encryption keys and customers... by bryan0 on Sunday January 15, 2023 @01:47AM (#34387300)
> There shouldn't be any coming back from this. There are failures on multiple levels here & CircleCI demonstrated no one should keep any sensitive data with them.

Are other companies any better? It seems like now it’s assumed that all of these companies will eventually be hacked, so if you use them you need to have systems in place to mitigate damage. And if you don’t use them them then you need to have mitigation strats anyway. Basically either way you’re screwed.

Re:CircleCI says hackers stole encryption keys and customers... by jay-barronville on Sunday January 15, 2023 @02:10AM (#34387386)
Not necessarily. Yes, it can be hard to fight off dedicated hackers, but it shouldn’t be that easy. I know it’s easy to judge them when we’re not in their shoes, but one compromised employee shouldn’t be able to produce these results. There wasn’t even proper key management protocols in place here. I can set up a production-ready Vault (HashiCorp) cluster (with decentralized key shares) in a weekend and exercise better OPSEC than them. That’s actually disappointing.
Re:CircleCI says hackers stole encryption keys and customers... by AYBABTME on Sunday January 15, 2023 @04:42AM (#34388141)
Using Vault isn't going to make this impossible. The secrets still end up somewhere on a host in an env var. People still get permission at some point to read from Vault. Combine enough parts of the puzzle and you can find cracks in the system.
Re:CircleCI says hackers stole encryption keys and customers... by komuW on Sunday January 15, 2023 @06:54AM (#34388746)
Note that dumping the Vault's process memory is beyond hashicorp/Vault's threat model. See: https://github.com/hashicorp/vault/issues/1446#issuecomment-...

I'm bringing this up because the circleCI blogpost says that the attacker did memory-dump encryption keys from a running process. See https://circleci.com/blog/jan-4-2023-incident-report/

So even if they were using hashicorp/vault, the attacker could probably still have been able to mem-dump vault's process.

Re:CircleCI says hackers stole encryption keys and customers... by robszumski on Sunday January 15, 2023 @08:05AM (#34389105)
You can run Vault inside of an enclave to protect it's memory: https://edgebit.io/enclaver/docs/0.x/guide-vault/
Re:CircleCI says hackers stole encryption keys and customers... by dehrmann on Sunday January 15, 2023 @03:16AM (#34387737)
Who has a better CI offering these days? I moved to CircleCI after Travis CI abandoned its open-source support.
Re:CircleCI says hackers stole encryption keys and customers... by m00x on Sunday January 15, 2023 @03:34AM (#34387820)
Github actions are pretty amazing. It can run most CI workflows.
Re:CircleCI says hackers stole encryption keys and customers... by riddlemethat on Sunday January 15, 2023 @03:44AM (#34387874)
Azure Pipelines + GitHub Actions
Re:CircleCI says hackers stole encryption keys and customers... by alex_suzuki on Sunday January 15, 2023 @04:07AM (#34387974)
Gitlab pipelines work well for me. I’m sure the Github and Azure offerings are on-par.
Re:CircleCI says hackers stole encryption keys and customers... by jeltz on Sunday January 15, 2023 @05:05AM (#34388242)
From my experience: no, Gitlab has a much better offering than GitHub.
Re:CircleCI says hackers stole encryption keys and customers... by lclc on Sunday January 15, 2023 @04:29AM (#34388069)
It's very easy to run your own GitLab-Runner (their open-source CI: https://docs.gitlab.com/runner/) on your server (or any cloud).

Set up within seconds using a few lines of cloud-init: https://gitlab.com/21analytics/gitlab-runner-cloud-init

Most of the time, it's also cheaper and maintenance is close to zero.

Re:CircleCI says hackers stole encryption keys and customers... by stingraycharles on Sunday January 15, 2023 @05:55AM (#34388483)
I second GitLab runner, used it a lot on bare metal CI servers, because we have some really exotic hardware requirements. Happy customer for 3 years now, and the CI part costs us nothing.
Re:CircleCI says hackers stole encryption keys and customers... by time0ut on Sunday January 15, 2023 @07:34AM (#34388939)
Gitlab is fantastic.

It was easy to take an existing open source terraform module [0], modify it a little for our purposes, and deploy runners that provision job executors in our tightly controlled VPC. It is very simple and everything is open source so you can really understand what it is doing. In our setup, all secrets are issued from our private vault instances. Most of them are short lived or one time use when possible. We are looking at moving our stuff to EKS now as well.

Overall, it was pretty easy to get going but we have the resources to do this. I could see why a small startup would outsource this to someone like Circle CI.

[0] https://registry.terraform.io/modules/npalm/gitlab-runner/aw...

Re:CircleCI says hackers stole encryption keys and customers... by damagednoob on Sunday January 15, 2023 @04:41AM (#34388131)
* * *
Re:CircleCI says hackers stole encryption keys and customers... by FBISurveillance on Sunday January 15, 2023 @05:23AM (#34388335)
We're on Google Cloud and pretty okay with Google Cloud Build - not as good and powerful as Buildkite but does not require us to run/maintain our own agents and one less vendor to manage.
Re:CircleCI says hackers stole encryption keys and customers... by mschuster91 on Sunday January 15, 2023 @05:35AM (#34388399)
If you're into AWS, CodeBuild.
Re:CircleCI says hackers stole encryption keys and customers... by scarface74 on Sunday January 15, 2023 @10:49AM (#34390231)
Disclaimer: I work for AWS in Professional Services. All opinions are my own.

The beauty about CodeBuild is that there is no “lock-in”. All it is fundamentally is a Linux or Windows Docker container with popular language runtimes and a shell script that processes a yaml file or you can supply your own Docker container.

You just put a bunch of bash commands or PowerShell commands in the yaml file and it runs anything.

The Docker container and the shell scripts are all open source and you can quite easily run them locally.

I could see outside of AWS keeping your Docker containers for your specific build environments in a local repository and doing all of your builds inside them using Jenkins using a self hosted CodeBuild like environment.

https://github.com/aws/aws-codebuild-docker-images

https://docs.aws.amazon.com/codebuild/latest/userguide/use-c...

For a “batteries included” approach though, I really like Microsoft Azure DevOps Pipelines.

I’ve even done a couple of integrations between Azure DevOps and AWS when we had clients that are Microsoft shops.

https://aws.amazon.com/vsts/

For AWS, if you use CodeCommit (AWS git service), all access is via IAM and granular permissions. If you integrate with Azure DevOps, the AWS credentials do have to be stored in a separate MS hosted credential storage.

CodeBuild also supports at least Github natively.

I’m not shilling for AWS. I have an MS development background (.Net) and only have “DevOps” experience using AWS and Microsoft tooling.

Re:CircleCI says hackers stole encryption keys and customers... by OJFord on Sunday January 15, 2023 @07:59AM (#34389062)
IMO, CI as the product companies are done for, to some extent they built a business on GitHub not having CI built in, and now it does. (GitLab already did, and when have you seen a GitLab-hosted project using CircleCI or similar?)

The more Actions matures, the harder it is for them to continue - their market will be legacy projects already on the platform, and perhaps the new projects from companies already using it for older projects (the one reason I can think of to prefer Circle over Actions for a new GH-hosted project).

Re:CircleCI says hackers stole encryption keys and customers... by tacker2000 on Sunday January 15, 2023 @08:37AM (#34389282)
Good old self hosted jenkins still does the job…
Re:CircleCI says hackers stole encryption keys and customers... by bingo-bongo on Friday January 20, 2023 @04:04PM (#34459917)
Yeah, but you do need to know how to set it up securely - this isn’t any better than the CircleCI incident: https://news.ycombinator.com/item?id=34446673
Re:CircleCI says hackers stole encryption keys and customers... by dragonwriter on Sunday January 15, 2023 @11:15AM (#34390467)
Honestly, while there are some difference, the reduced integration overhead of using the offering from either your main cloud platform or your source code host probably wins.
Re:CircleCI says hackers stole encryption keys and customers... by AYBABTME on Sunday January 15, 2023 @04:39AM (#34388119)
Stuff is so complex nowadays that there's a ton of vectors an employee can use to eventually access almost anything on production systems. Unless you kneecap their ability to reach anything at all. Even if say you use Vault for secrets, extensive ACLs, on-demand access, there's an almost infinite amount of ways to squeeze out data in ways it hasn't been anticipated. Kuberneters, containers, etc doesn't always help, often sometimes it makes things worse.

Defending against internal threat seems like a losing battle that can only be mitigated, slowing down more than preventing an attack.

Re:CircleCI says hackers stole encryption keys and customers... by debarshri on Sunday January 15, 2023 @04:47AM (#34388162)
Stopping employee from accessing anything and everything in production impacts productivity and engineer often do not want to work in that kind of environment.
Re:CircleCI says hackers stole encryption keys and customers... by AYBABTME on Sunday January 15, 2023 @04:48AM (#34388171)
It's also not possible, by definition someone needs some access. Compromise enough of those people and there's no way around it.
Re:CircleCI says hackers stole encryption keys and customers... by debarshri on Sunday January 15, 2023 @05:00AM (#34388215)
Exactly. Only thing you can do is make a drill where you assume everything is hacked and audit access daily, weekly and monthly.
Re:CircleCI says hackers stole encryption keys and customers... by wepzen on Sunday January 22, 2023 @04:05AM (#34475635)
Implementing a zero-trust architecture with a trust-score system for users and a dynamic policy for accessing resources can help to limit potential damage in the event of a security incident. But I agree that the balance between protecting against attacks and maintaining productivity can be delicate.
Re:CircleCI says hackers stole encryption keys and customers... by maccard on Sunday January 15, 2023 @05:37AM (#34388416)
At some point, someone somewhere has to have access to infrastructure, and be able to deploy it (if not, then generate a token of set of credentials that _can_).

Circleci's business model is hosting CI runners for you, so of course they need to be able to decrypt the data, and if you have access to dept new infrastructure, you likely have permission to read encryption keys (or deploy new infrastructure that can read said keys and then use those machines to get the keys).

What CircleCI _have_ done is set up enough logging and auditing that they were able to figure out who was compromised, how they were compromised, the time frame and the resources they accessed, which IMO is about as much as you can ask for.

Re:CircleCI says hackers stole encryption keys and customers... by oxfordmale on Sunday January 15, 2023 @05:46AM (#34388455)
CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access. Fraud analysis can be basic and just have a threshold on the number of encryption keys accessed each day. For this particular employee, there will have been a spike.

They also point their finger at anti-virus for not detecting the malware. That is a lame excuses. Professional malware developers will check if their product goes undetected by the major anti-virus providers. Anti-virus should not be relied upon to protect against sophisticated attacks

Re:CircleCI says hackers stole encryption keys and customers... by maccard on Sunday January 15, 2023 @05:57AM (#34388492)
They didn't blame the anti malware. In the blog post[0] all they say is

> This machine was compromised on December 16, 2022. The malware was not detected by our antivirus software

That's not blaming that's telling people that their antivirus didn't detect it. If that wasn't there, people would be talking about why they didn't use X antivirus which would probably detect it.

> CircleCI have sufficient logging, however, failed to do fraud analysis to detect irregular access.

This is a significant move of the goalposts. Of course CircleCI messed up, but to go back to the OP's point of "nobody should have that much access to production", well that's just not true.

[0] https://circleci.com/blog/jan-4-2023-incident-report/

Re:CircleCI says hackers stole encryption keys and customers... by oxfordmale on Sunday January 15, 2023 @06:29AM (#34388636)
I can't judge if their employees need that much access to production.

However, if they need this, you need a detection mechanism. It doesn't have to be advanced, a threshold on the number of production keys accessed per day is sufficient. It should raise an alarm to the manager in question, who can then confirm this is expected behaviour given the task assigned to the employee in question.

Re:CircleCI says hackers stole encryption keys and customers... by maccard on Sunday January 15, 2023 @07:23AM (#34388880)
Again I'm not saying circle CI didn't mess up, but that's orthogonal to whether or not someone has access to customer data.
Re:CircleCI says hackers stole encryption keys and customers... by oxfordmale on Sunday January 15, 2023 @08:14AM (#34389155)
Not necessarily. AWS engineers cannot access your encryption keys unless you give them explicit permissions. They even offer Nitro enclaves where AWS Engineers can never access your keys.

Giving access to customer data to your staff by default is a design decision and can be avoided. However, in the end, it is a cost-benefit analysis where you have to decide how much you care about your customer's security. I have worked for enough start-ups/growth companies to know the value put on customers' security is sometimes shockingly low.

Re:CircleCI says hackers stole encryption keys and customers... by iLoveOncall on Sunday January 15, 2023 @06:01AM (#34388509)
Not just professional malware developers. Any script kiddie can pay 10 bucks to make their virus undetectable.

This is why Antiviruses are totally useless, they're incredibly easy to bypass.

Re:CircleCI says hackers stole encryption keys and customers... by charcircuit on Sunday January 15, 2023 @09:41AM (#34389719)
>This is why Antiviruses are totally useless

Not all attacks are targeted and AV can detect abnormal behaviour. Regardless, having the operating system do proper sandboxing is much more valuable than trying to fix it after the fact with AV.

Re:CircleCI says hackers stole encryption keys and customers... by vsareto on Sunday January 15, 2023 @10:40AM (#34390156)
AV stops low-brow shit, which is still something worth stopping and having around.

The name is obviously misleading though.

Re:CircleCI says hackers stole encryption keys and customers... by xiphias2 on Sunday January 15, 2023 @06:09AM (#34388543)
They can use hardware wallets for digital signing the operations, and keep encryption keys offline, split the keys to make sure that there's no 1 person that can access it.

There are lots of tools for key management, but lots of companies don't care about it.

Re:CircleCI says hackers stole encryption keys and customers... by maccard on Sunday January 15, 2023 @07:25AM (#34388889)
That's completely unrelated to whether someone needs access or not though.
Re:CircleCI says hackers stole encryption keys and customers... by lrvick on Sunday January 15, 2023 @11:08AM (#34390406)
> At some point, someone somewhere has to have access to infrastructure

They could run everything in Nitro Enclaves or similar, that require multiple people to deterministically compile and sign new software for, and release secrets into.

I design quorum controlled infrastructure for a living, mostly in fintech where no single human can ever be trusted. You 100% can run infrastructure that, barring a platform 0day, can prevent any single human from having access to the memory of customer workloads and secrets. Customers likewise would encrypt any secrets or code directly to keys that only exist in the enclaves.

CircleCI had negligent security design, but all its competitors are just as bad, to be fair.

Building with security in mind makes you last to market, which is unforgivable in our industry. Getting hacked however is just considered a cost of doing business.

Re:CircleCI says hackers stole encryption keys and customers... by ignoramous on Sunday January 15, 2023 @05:14PM (#34394159)
> CircleCI had negligent security design, but all its competitors are just as bad, to be fair.

Ominous. What about GitHub? The amount of secrets I trust them with is practically all my life force at this point.

Re:CircleCI says hackers stole encryption keys and customers... by lrvick on Monday January 16, 2023 @07:13AM (#34399833)
There is no evidence GitHub has any multi-party accountability for sysadmins or enclaves for secret management. You enter secrets into the GitHub Web UI in plaintext, which means at least some employees can access them in plaintext.

GitHub/NPM have historically failed to support supply chain integrity practices in their public offerings such as hardware anchored code signing, signed code reviews, reproducible builds, multi-party approvals, etc. It is reasonable to expect they are not doing any of that internally either.

Assume any secret you give GitHub will become public knowledge and act accordingly.

The good news is there is never a reason to trust a VCS or CI system with high value secrets. They should never ever need any power beyond running tests, accessing a test environment, or sending notifications.