News for hackers. Stuff that matters.
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @11:56AM (#37483211)
This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI.

I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up their credit cards, or better, their accounting PO process. You need a feature you withhold from the open source offering to make it appealing.

GitLab restricts CI/CD to paying customers. Travis/CircleCI limit on build time/credits unless also open source. Azure DevOps is the devil. ArgoCD is complicated. GitHub actions are nice if you have hardware to run the runners. Swarm’s of Jenkins is what most are used to in the enterprise.

As a former Director of DevOps, my immediate question will be what features make me want to buy vs host myself. I’m technically capable. Why should I buy with you when I can run my cloud and my DevOps pipelines for my workforce the way that makes the most sense for my business? Help me make the most sense.

Re:We built the fastest CI in the world and it failed by jangxx on Tuesday September 12, 2023 @12:36PM (#37483856)
You can self-host Gitlab CI runners and use them even in the free community version.
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @12:51PM (#37484102)
I know, this is what I’ve been doing for years. (edit on GitHub, not GitLab, didn’t know GitLab allowed community CI now)
Re:We built the fastest CI in the world and it failed by jangxx on Tuesday September 12, 2023 @12:54PM (#37484158)
Then I guess I misunderstood "GitLab restricts CI/CD to paying customers"?
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @01:21PM (#37484503)
Sorry, I misread your comment as GitHub, not GitLab. I was unaware you can run CI runners on GitLab community version.
Re:We built the fastest CI in the world and it failed by prometheon1 on Tuesday September 12, 2023 @02:10PM (#37485125)
I thought using your own runners for free in Gitlab was already an option before Github Actions even existed
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @02:20PM (#37485251)
When it was an open source project, yes. You are correct. Ever since GitLab went commercial, it's been behind a paywall. Apparently you can run runners on self-managed CE so I'm going to look into that.
Re:We built the fastest CI in the world and it failed by smashed on Tuesday September 12, 2023 @05:39PM (#37488829)
On Gitlab.com you need to input a valid credit card once in order to validate your account, but you can still get 400 minutes on shared runners for free.

This is used to combat abuse and crypto miners, they don't actually charge the card.

You can also connect your own private runner to either the saas or self-hosted gitlab-ce for free, you just have to provide your own runner.

Re:We built the fastest CI in the world and it failed by mindwok on Tuesday September 12, 2023 @06:12PM (#37489288)
GitLab is still an open source project (the core of it is MIT licensed) and using your own runners has always been an option. Also, free accounts on the SaaS version get free CI/CD minutes as well.
Re:We built the fastest CI in the world and it failed by nivenhuh on Tuesday September 12, 2023 @06:23PM (#37489399)
I run a self-managed CE GitLab with my own runners — works as expected (no payment required).
Re:We built the fastest CI in the world and it failed by bottled_poe on Tuesday September 12, 2023 @06:46PM (#37489639)
Same, been doing this for many years and upgrade frequently.
Re:We built the fastest CI in the world and it failed by jcrawfordor on Tuesday September 12, 2023 @07:19PM (#37490014)
(GitLab employee but familiar with this because I use it personally as well)

You can always connect your own runners to GitLab, whether SaaS or self-hosted, and whether free/CE or premium/ultimate/EE. GitLab SaaS also comes with free use of the central runners up to a quota. I'm not sure about the history here but as far as I know this has always been the case.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @09:45PM (#37491191)
Circa 2013? I remember vividly implementing gitlab at a certain cable company I worked at because they were too cheap for GitHub. I remember sometime in 2012 some merging of worlds of GitLab CI and GitLab.com, it was around that timeframe until 2015 or so when the unofficial runner because official and CI and Repos were merged? You work there, ask the oldest grey beard if this was the case. I could very well be mistaken and have false memory.
Re:We built the fastest CI in the world and it failed by jcrawfordor on Friday September 15, 2023 @05:50PM (#37529528)
you might be right, especially in that time period new features were often EE only initially and then merged into CE. When CICD was first created it was its own software project basically, but it may have required APIs that weren't merged into CE yet. To be totally honest I forgot that CICD was around that early on.
Re:We built the fastest CI in the world and it failed by Spivak on Tuesday September 12, 2023 @12:52PM (#37484125)
And honestly I can't even imagine ever using the public runners because they're permissionless. It's way easier to have them live in your own network and doling out permissions via tags that get picked up by different runners with different iam roles.
Re:We built the fastest CI in the world and it failed by chrisweekly on Tuesday September 12, 2023 @01:20PM (#37484500)
(mods: please forgive the off-topic personal question here)

Hi Spivak, You're not Lee Spivak (formerly of Upromise in the early 2000's), are you?

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @01:22PM (#37484520)
This is why some people don’t leave the house.
Re:We built the fastest CI in the world and it failed by stavros on Tuesday September 12, 2023 @09:04PM (#37490866)
They're too happy to have their username recognized by strangers on internet forums?
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @09:46PM (#37491196)
Well, in this case, leave the house…
Re:We built the fastest CI in the world and it failed by hhh on Tuesday September 12, 2023 @12:21PM (#37483630)
I feel almost certain someone could wrap ArgoCD into a product if they handled the 'patternization' of applicationsets. D2iQ already do it with Flux, but we bailed on D2iQ before we even got to try it.
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @01:28PM (#37484568)
ArgoCD is almost worthless if your DevOps teams are IaC wielders. Honestly, just a bash script to push a container to the pods is all that’s required for DevOps deployment. A build pipeline should be get artifacts -> put artifacts in container -> push container to registry -> trigger a pull on your pods. If you didn’t have a standard cloud architecture, I can understand the frustration and reasoning behind looking at ArgoCD. If you have a standardized cloud practice, it’s moot.
Re:We built the fastest CI in the world and it failed by hhh on Tuesday September 12, 2023 @05:52PM (#37489024)
My needs are more 1 application, 150 configurations, 150 clusters.
Re:We built the fastest CI in the world and it failed by dpeckett on Tuesday September 12, 2023 @12:54PM (#37484144)
I'm not sure feature withholding has traditionally worked out well in the developer space. The churn rate for tooling is so high that if Earthly were to intentionally cripple their product I'm sure the majority of developers would just move on to other inferior, but free, options (eg. taskfile). Maybe those that remain could be converted successfully but it's quite a gamble.

A great turn around story in recent times though is probably Docker. But that required them to abandon strict open source and make controversial license/product changes.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @01:19PM (#37484485)
Yeah, not suggesting feature withholding per se, but a differentiator on why one would buy vs run the OSS build.
Re:We built the fastest CI in the world and it failed by staunton on Tuesday September 12, 2023 @01:38PM (#37484715)
What's the difference?
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @01:42PM (#37484765)
plugins, hardware, hosting, support, there's a number of ways to offer a service for an open source project. Open Core models come to mind. Another is not having enterprise SSO as part of the core. Or not having clustering. Something that would make a VP think twice about adopting your open source solution vs using you as a vendor.
Re:We built the fastest CI in the world and it failed by justin_oaks on Tuesday September 12, 2023 @02:33PM (#37485496)
I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it.

See https://sso.tax/ for details but I'll quote this from it "SSO is a core security requirement for any company with more than five employees"

Re:We built the fastest CI in the world and it failed by rhaps0dy on Tuesday September 12, 2023 @02:42PM (#37485736)
But it should be OK to reserve SSO for paying customers.
Re:We built the fastest CI in the world and it failed by imiric on Tuesday September 12, 2023 @02:49PM (#37485872)
If a company requires SSO, they'll surely have a budget for paying for the products and services they use. The majority of OSS users won't have this requirement, and likely won't have the budget to pay for commercial features. So it doesn't make sense to give away a service that has a maintenance cost to everyone, as long as it's fairly priced.
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @03:06PM (#37486222)
If you have more than 5 employees, I’m sending you a sales rep to sell SSO to you.
Re:We built the fastest CI in the world and it failed by justin_oaks on Tuesday September 12, 2023 @05:20PM (#37488573)
I see, I see. The world isn't fair. We have 5 employees and we want SSO but don't need SSO. So we'll go without it. It's annoying for us but it allows your business to exist because enterprise needs SSO.
Re:We built the fastest CI in the world and it failed by mavelikara on Tuesday September 12, 2023 @09:19PM (#37490969)
I am going to the competitor. I am paying for the software, but I don’t want your “enterprise tier”, which is really designed for 1000+ employee companies.

BTW, I did just this this morning. The AE came back to offer the enterprise package for the price of the mid-tier package.

Re:We built the fastest CI in the world and it failed by Xymist on Wednesday September 13, 2023 @07:39AM (#37495395)
If I have to go through a sales rep to try or buy your product, I'm going with a competitor. Zero touch is a feature, upsell opportunities are cancer.
Re:We built the fastest CI in the world and it failed by BoorishBears on Tuesday September 12, 2023 @03:14PM (#37486358)
As others are pointing out, once you've grown big enough that you can't trust everyone with the keys go the kingdom: enterprise pricing is for you.

SSO Tax is a fun meme, but it misses the point: Whether it's 5 people or even 3, you've scaled your business enough to hire people who aren't in the inner circle.

That's a fair place to consider you an enterprise.

Re:We built the fastest CI in the world and it failed by justin_oaks on Tuesday September 12, 2023 @05:31PM (#37488739)
Not sure what you're talking about. It's not like without SSO everyone in the company has to all use the same password. Services allow people to have separate users with separate privileges.
Re:We built the fastest CI in the world and it failed by BoorishBears on Tuesday September 12, 2023 @06:17PM (#37489333)
It sounds like you're unaware of why SSO is considered a security feature at all them, but it's covered right on the site: https://sso.tax/

It's to allow centralized access management. Stuff like firing someone and revoking their access from one platform instantly, instead running around and changing permissions in every tool manually. Or ensuring people in department A can't be invited to some platform for people in department B in order to limit information access.

SSO tax is predicated on the idea that the moment you outgrow the informal arrangements and liberal access, you're really a business. Seems pretty fair?

Re:We built the fastest CI in the world and it failed by smaudet on Tuesday September 12, 2023 @07:14PM (#37489950)
> It sounds like you're unaware of why SSO is considered a security feature

Technically it's an anti feature...

The "feature" you are talking about is really identity management not sign on (which, btw, users have different identites, outside of the scope of a single company).

At its core it's just a delete function for a username. Putting that in a script with http access should be enough (and not put behind a ridiculous price tag).

Separate services otherwise are enough, no need for SSO.

Re:We built the fastest CI in the world and it failed by BoorishBears on Tuesday September 12, 2023 @07:58PM (#37490361)
You're barking up the wrong tree: I communicated how SSO's value prop is defined by essentially the entire SaaS industry... if you have a problem with that value prop, Okta alone is 13B worth of short potential for you once everyone realizes they could just use a script with http access.
Re:We built the fastest CI in the world and it failed by lelanthran on Wednesday September 13, 2023 @02:04AM (#37492773)
> At its core it's just a delete function for a username. Putting that in a script with http access should be enough (and not put behind a ridiculous price tag).

Can you explain more about how this works for SSO?

Let's say I'm trying to launch a product (which I am), and want to have SSO as part of the package (which I do), how do I implement what you just said?

I looked into SSO services and they're bloody expensive, so if I can cheaply do it myself, I will.

Re:We built the fastest CI in the world and it failed by BoorishBears on Wednesday September 13, 2023 @06:03AM (#37494646)
Their reply is not SSO, it's some toy alternative they're proposing that none of your customers would accept (like saying "Dropbox is just rsync")

SSO is hairy enough that you can't write it from scratch in any reasonable amount of time for what a typical SaaS needs.

There's OSS SSO you can host yourself that supports enterprise : https://www.keycloak.org/

If you're B2C Firebase Auth is cheap, and doesn't actually require hosting on Firebase

Re:We built the fastest CI in the world and it failed by lelanthran on Wednesday September 13, 2023 @06:23AM (#37494813)
> SSO is hairy enough that you can't write it from scratch in any reasonable amount of time for what a typical SaaS needs.

I expected so, looking at how much it costs to provide just SSO as a service.

Thanks for keycloak, didn't know about that.

Re:We built the fastest CI in the world and it failed by smaudet on Friday September 15, 2023 @08:44AM (#37522115)
The comment about being a 'toy service' is disingenuous...

The comment about customers is possibly valid, the difference entirely your use case. If you consume services as a corp, and your goal is to be able to decouple any employee quickly, then you do not need SSO, assuming each service you use allows you the option to delete by username (with proper auth, ofc).

SSO is only needed when you want to give a customer federated access (ability to assign roles without activation on their part). For an internal corp it is largely unnecessary and a convenience thing not a security one (it is much more secure to require different passwords for every service versus a global SSO credential that gets you everywhere).

If your "customers" are your own employees, then this is your decision not a use case or a business requirement.

Re:We built the fastest CI in the world and it failed by debarshri on Tuesday September 12, 2023 @03:22PM (#37486490)
Let me give a contrarian argument. I dont think it is awful to have sso reserved only for enterprise customers. I think enterprise customer who are going through compliance will definitely have to but into that tier therefore.

If you think you need SSO and but you do not have the budget for upgrading to their enterprise tier, may be you are not the right customer that they are targeting to sell. It is not you, it is them.

Re:We built the fastest CI in the world and it failed by justin_oaks on Tuesday September 12, 2023 @05:36PM (#37488797)
You're right. I was thinking from the position of the consumer for which SSO is both convenient and a security feature.

I wasn't thinking as the service provider that needs a way to force the folks with the big bucks to pay up.

It sucks for the little companies, but it may be necessary for the service to do so.

Re:We built the fastest CI in the world and it failed by lelanthran on Wednesday September 13, 2023 @01:08AM (#37492443)
> I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it.

SSO is expensive, adding maybe $10/user/month if the team adds in a purchased solution (passing the cost directly onto the client).

If the customer doesn't want to pay for it because they aren't an enterprise, you can't blame the supplier for not supplying it at a lower price than $10/user/month.

Re:We built the fastest CI in the world and it failed by caloique on Tuesday September 12, 2023 @05:25PM (#37488637)
Co-founder of BoxyHQ here - We've crafted an open-source enterprise SSO because we firmly believe that robust security shouldn't be a privilege limited to large organizations. Ideally, essential enterprise-level features like this should become commonplace for all.

While we acknowledge the reasons behind SSO being in the enterprise tier, we're all on a collective journey to enhance our security measures. Open Core models are indeed a good option (my preference), yet the dynamics vary across solutions and industries. It's up to each of us to explore, experiment, and discover what resonates with our market. In doing so, we can foster growth while maintaining our commitment to supporting the community in the long run.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @09:33PM (#37491097)
Well stated. I was merely showing how one could implement an open core strategy. I’m a firm believer in SSO for all (and not limited to social platforms or GitHub).
Re:We built the fastest CI in the world and it failed by imiric on Tuesday September 12, 2023 @01:50PM (#37484880)
For the open core model to work, the OSS product should be standalone and featureful. If users feel like it's just used to advertise a commercial product, it doesn't work. And the commercial product should offer value-added services and features that are not strictly required for the base OSS product. These should be features directed towards advanced or corporate users, and preferably something that competitors lack, or don't do a good job of.

It's very tricky to balance the open core model correctly, which is why not many companies get it right. You need to invest equally, if not more, in the OSS product, and make sure users are happy with it before you even mention a commercial addon. Grafana is another example of companies that understand and do this well.

Re:We built the fastest CI in the world and it failed by BlackjackCF on Tuesday September 12, 2023 @02:09PM (#37485110)
I think a nice compromise is the CircleCI/Travis model of providing limited build credits/hours. If you’re a small shop or single dev, you can use the core product for free, with the hope that you’d convert to a paying user if your business grows past the free constraint.
Re:We built the fastest CI in the world and it failed by tazjin on Tuesday September 12, 2023 @03:04PM (#37486173)
In this particular case, developers could also move on to a superior and completely open alternative (Nix), so making even a semi-closed alternative with much fewer people contributing etc. was already a hard sell.
Re:We built the fastest CI in the world and it failed by JustLurking2022 on Tuesday September 12, 2023 @03:58PM (#37487208)
Seriously, is there an email distro or something to notify Nix users when there's a new post to go troll? It's not suitable for 90%+ of users, which seems a far cry from superior.
Re:We built the fastest CI in the world and it failed by tazjin on Tuesday September 12, 2023 @04:01PM (#37487258)
It's just very noticeable when half of what the industry seems to spend its time on is solved problems. You and all the others have to learn why touching the hot stove is a bad idea the same way most children do. It's frustrating to observe.

It's fine to criticise Nix for its various flaws, but any suggestion of an "alternative" that has learned nothing from it is just a collective waste of time.

[dead]
Re:We built the fastest CI in the world and it failed by tazjin on Wednesday September 13, 2023 @05:21PM (#37501884)
I'm trembling at how much you're ignoring me!
Re:We built the fastest CI in the world and it failed by belmont_sup on Wednesday September 13, 2023 @08:49AM (#37496066)
I’ve recently had to learn some nix to build an IHP app (Haskell framework). Most hosting providers require a dockerfile and/or image. I couldn’t get nix to build an image on time within the hosting providers’ CI. So I gave up.

For any other project, I can spend half an hour max to get a decent deployable build. The trade offs is Nix doesn’t seem to work for me.

Re:We built the fastest CI in the world and it failed by pdimitar on Wednesday September 13, 2023 @09:59AM (#37496855)
> You and all the others have to learn why touching the hot stove is a bad idea the same way most children do. It's frustrating to observe.

You say that while your ecosystem still can't get away from cutesy childish terms like "flakes" and "pills". Newsflash: nobody cares. I want "a package" and "a system".

I am not impressed. Before you look down on others make your own thing look uber-professional and then uproot any and all nerd-friendly marketing from it, now and forever. "It was in fact just a phase, mom" -- that's the aura that Nix should radiate before I give it a second chance.

Nix is supposed to solve a very serious problem and if it actually succeeds in that technical goal, the next step is for them to act like it. Very often you're marketing to CTOs or other tech executives -- working programmers are only one chunk of your audience. So that ecosystem has to do better -- starting with you. Your two comments here are basically a text-book argument against using Nix.

I've been a CTO and VP of Eng and I have skipped projects that can't sound professional. I know I have missed out on good stuff; I am very well aware. But we all have only 24h in a day, and only so much energy. We have to filter. One of my filters is: perceived / marketed professionalism.

(To be fair to all sides, I like the nixos.org website. I don't like practically anything else about the ecosystem though.)

> It's fine to criticise Nix for its various flaws, but any suggestion of an "alternative" that has learned nothing from it is just a collective waste of time.

Let's start with that one: nobody needed one more programming language. Nobody will "see the light" if every new technology requires one more knowledge investment. Somebody must tell you that programmers, DevOps, full-blown Ops, CTOs etc. are already overworked and already have to know and regularly catch up with way too much.

If Nix is aiming to help people work less on very annoying problems then it should, you, actually make them work less.

Nix's marketing is a complete mess. "We want you to think less about A and B so here, now learn, X, Y, Z and you should still know A and B -- but it gets better after, one day!".

Nah.

It's fine that you found technology that you love but your bias is very clearly visible and you lack the introspection skills to see that Nix is not that impressive.

Give me a `curl ... | bash` installer plus instant onboarding plus zero upkeep (an oft complaint about Nix btw) and I am sold. Anything less than that, to me you are just one more voice in a huge crowd where everyone screams "use my thing, it's amazing!".

Differentiate yourself and people will flock to you without you having to lift a finger. The fact that this hasn't happened yet is a strong hint that Nix fans are refusing to notice and take a lesson from.

Re:We built the fastest CI in the world and it failed by tazjin on Wednesday September 13, 2023 @05:35PM (#37502018)
I tried to write a serious response to your comment, but as I got further into it and you got further into personal attacks and away from substance, I decided to delete it and I'm instead leaving you with this:

You should get off your high horse. Nobody cares about you having been a "CTO" of some random venture-backed company, it doesn't give you any credentials which you can fling at people to "win" debates online. 80% of the people on this forum have some useless title like that in their CV. You being able to write lots of words about irrelevant superficial topics doesn't mean that you have anything useful to say, even if you've previously been led to believe that these are equivalent.

I don't care where you flock. I publish all my Nix related work for free, I don't have a personal stake. Your decisions are your problem.

Re:We built the fastest CI in the world and it failed by pdimitar on Wednesday September 13, 2023 @05:40PM (#37502087)
Speaking of personal attacks, you chose only one part of my comment where I mentioned how I did some decision making, and ignored everything else.

Well, OK, now I know who is on the high horse so let's drop it here.

Shame that tech zealotry never dies.

Re:We built the fastest CI in the world and it failed by hnlmorg on Tuesday September 12, 2023 @04:15PM (#37487537)
> I'm not sure feature withholding has traditionally worked out well in the developer space.

It’s worked well for GitHub, Slack, and pretty much every service that only offer SSO behind enterprise subscriptions.

The trick is to offer enough to make people productive but not so much that there isn’t a worthwhile upgrade path.

Re:We built the fastest CI in the world and it failed by dpeckett on Tuesday September 12, 2023 @04:35PM (#37487848)
True (for web apps and multiuser applications).

Earthly is a single binary CLI on top of Docker buildkit so it's a bit trickier for them.

Re:We built the fastest CI in the world and it failed by hnlmorg on Wednesday September 13, 2023 @03:26AM (#37493340)
I’m not claiming it’s easy. Just that withholding features while still offering free tiers is a tried and tested business model that has demonstrated a lot of success for a lot of companies.
Re:We built the fastest CI in the world and it failed by edgyquant on Tuesday September 12, 2023 @07:42PM (#37490219)
Neither GitHub nor slack are open source
Re:We built the fastest CI in the world and it failed by hnlmorg on Wednesday September 13, 2023 @03:25AM (#37493333)
Both GitHub and Slack have free tiers. Same principle, different software license.
Re:We built the fastest CI in the world and it failed by edgyquant on Wednesday September 13, 2023 @05:28PM (#37501957)
It isn’t the same principle though. Slack and GitHub can sell data to make a profit etc but the problem with open source is people can just host it themselves meaning the creators don’t even have that option.
Re:We built the fastest CI in the world and it failed by hnlmorg on Thursday September 14, 2023 @02:31AM (#37505677)
Plenty of organisations prefer to have managed options and thus pay for SaaS, even from open source vendors, rather than hosting their own.

Also you’re getting too caught up in specifics and missing the broader point. You want open source examples? How about Nexus, Redhat, nginx, prettier much all of Hashicorps offerings, basically half of why AWS resell.

The real issue with commercial open source isn’t restricting access to customers, it’s restricting access to resellers. As demonstrated with the examples above.

Re:We built the fastest CI in the world and it failed by caseyohara on Tuesday September 12, 2023 @05:14PM (#37488479)
> I'm not sure feature withholding has traditionally worked out well in the developer space.

I think it's worked out well for Sidekiq (https://sidekiq.org). I really like their model of layering valuable features between the OSS / Pro / Enterprise licenses.

The trick is you have to do that very early on. Nerfing your open source offering by making some existing features paid-only is always going to get some backlash.

Re:We built the fastest CI in the world and it failed by ljm on Tuesday September 12, 2023 @05:41PM (#37488866)
The open source version has to be good and valuable enough for you to bring it in as a dependency in a basic setup, and then the paid versions provide support for larger scale setups where the requirements are a bit different.
Re:We built the fastest CI in the world and it failed by pjmlp on Wednesday September 13, 2023 @02:00AM (#37492751)
That is why most developer products have gone back to the timesharing days.
Re:We built the fastest CI in the world and it failed by isoprophlex on Tuesday September 12, 2023 @01:05PM (#37484317)
> Azure DevOps is the devil.

Amen to that!

Re:We built the fastest CI in the world and it failed by shandor on Tuesday September 12, 2023 @01:44PM (#37484795)
Is there an actual technical point beyond ”MS bad” for this sentiment?

Honestly asking, as I’ve been mostly hearing positive things about AzDO, and from developers of all people.

Re:We built the fastest CI in the world and it failed by WorldMaker on Tuesday September 12, 2023 @02:20PM (#37485262)
AzDO is "decent", depending on your needs, but it's in an unfortunate zombie state where Microsoft is supporting it just enough to keep it stable and keep certain enterprises happy and consistently insisting that AzDO has a roadmap and is still beloved, but it is very clear that all of the actual resources are going to the GitHub side of the house in 2023.

(One of the most recent signs of this crazy zombie state that a recent feature for AzDO was given the absurd only Microsoft could do brand name "GitHub Advanced Secure for Azure DevOps", and it is indeed a bundle of GitHub features several years old at this point finally provided to AzDO users. Other signs include all sorts of AzDO libraries and roadmaps and related repos openly hosted on GitHub with last updated dates in 2019 and 2020.)

From an outsider perspective, Microsoft probably has a lot of sentimental and internal engineering reasons it doesn't want to truly wrap up so much of its operations in North Carolina (AzDO's ancestral home going back to early days of TFS), but the writing seems to be on the wall that what's left is a skeleton crew mostly working for GitHub full time and then sort of applying things part time back to AzDO. It doesn't seem to me to be sustainable long term, and every day it feels more like an Old Yeller situation where Microsoft is just prolonging the inevitable and making the pain worse for everyone involved.

Re:We built the fastest CI in the world and it failed by hotnfresh on Tuesday September 12, 2023 @02:37PM (#37485595)
Ugh, they’re busy bringing bad project management to GitHub, in a quest to make it suck as much as all the others.

The old issues + tags + milestones was perfect. Now it’s the same needlessly-heavy thing as all the rest, with their reimagined “projects” thingy.

But then again I’ve not once seen a PM embrace any version of GitHub’s project management, with the only explanation forthcoming being “it’s confusing for non-technical users” (fucking how? More confusing than Jira or Asana? No friggin’ way) so maybe they have to shit it up and make it hard to use and easy to get lost in or miss info in, to get any traction with PMs.

Re:We built the fastest CI in the world and it failed by WorldMaker on Tuesday September 12, 2023 @02:50PM (#37485884)
It does seem a case where Microsoft dogfooding GitHub may be actively making it worse. They've done several BUILD and GitHub Universe videos "How Microsoft Uses GitHub to Build [X]" and their PMs certainly need all those complex features to track stuff and those videos are the closest to training materials on how to use some of it, and it is definitely a lot going on.

It definitely seems one of those sorts of "every system PMs touch is doomed to become unmanageable complexity" laws and GitHub "Projects" does seem to be exponentially closer to being "Jira 2: Electric Boogaloo" every day.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @02:32PM (#37485472)
This. AzDO will die. It will be replaced entirely by Github. It originally was only designed as a way to get people onto Azure using their already-existing MS partnership to give it away for free. The only purpose, drive adoption of Azure Cloud. In runners, resources, stickiness, and preventing abandoning ship.
Re:We built the fastest CI in the world and it failed by rhaway84773 on Tuesday September 12, 2023 @03:49PM (#37487017)
Azure Devops existed well before GitHub was owned by MS under Visual Studio Online moniker.

It was intended to be a SAAS version of the on Prem offerings MS has provided since before GitHub even existed.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @09:59PM (#37491282)
I know it’s history and I know it’s future.
Re:We built the fastest CI in the world and it failed by datadrivenangel on Tuesday September 12, 2023 @01:53PM (#37484926)
ADO makes me yearn for Jira.

The boards are terrible, the UI is hard, the permissions are incomprehensible, it doesn't even integrate that nicely with Azure.

On paper it does everything, it practice it can be made to do everything, but it's like getting citrus drenched sandpaper papercuts every day.

Re:We built the fastest CI in the world and it failed by shandor on Tuesday September 12, 2023 @02:00PM (#37485002)
Hah, thanks for that mental image.

As I said the positivive sentiments I heard came from developers, and TFA is about CI (rather than boards etc. that JIRA handles, so is there maybe anything remotely positive in how AzDO implements CI, git integration, testing, etc? Or is it just rotten to the core?

Hard to imagine someone would hate it more than Bitbucket Pipelines or, god forbid, Bamboo. Buuut I’m more than open-minded if need be.

Re:We built the fastest CI in the world and it failed by WorldMaker on Tuesday September 12, 2023 @02:22PM (#37485284)
AzDO Pipelines was directly the first draft of what became GitHub Actions, so technically it does have a lot to recommend it, especially if you learn the Multi-Stage YAML approach which does sort of directly feed into GitHub Actions for the eventual day you decide to migrate.
Re:We built the fastest CI in the world and it failed by datadrivenangel on Tuesday September 12, 2023 @04:20PM (#37487621)
Yeah, the Pipelines are okay all things considered.
Re:We built the fastest CI in the world and it failed by AaronM on Tuesday September 12, 2023 @01:32PM (#37484629)
Did you mean argowf instead of argocd? Argocd is pretty simple and only syncs k8 resources.
Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @01:46PM (#37484831)
I mean that node (not js) dashboard to build pipelines and make things look logical instead of writing 4 lines of bash. I'm not a fan of anything that adds complexity. ArgoCD/WF/ALL ARGO... is just a pretty dashboard over cli tooling.

   ./deploy.sh <repo>@<version> <pod-name>
This is the contract you should focus on. Whatever you need to make ./deploy.sh do this, do it.
Re:We built the fastest CI in the world and it failed by psYchotic on Tuesday September 12, 2023 @04:45PM (#37488018)
Isn't ArgoCD more of a GitOps tool? The pretty UI is mostly secondary to its main purpose for me, which is to keep the declarative "truth" in source control, and have ArgoCD be the control loop that keeps the cluster in sync with that truth. Accidentally nuked namespace? No worries, ArgoCD's (or whatever alternative, like flux) got your back!
Re:We built the fastest CI in the world and it failed by rrrix1 on Tuesday September 12, 2023 @05:54PM (#37489061)
Exactly this. The critical difference being who deploys and when/how your resources are deployed - a human, or a machine.

Having git be the source of truth of your production environment is a blessing and a curse which is directly related to the maturity of the deployment system.

If you have low confidence in your deployments (as in, you don't deploy very often and don't have full e2e tests & monitoring), GitOps is nightmarishly scary compared to classic, battle tested Ops team CLI scripts.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @10:04PM (#37491321)
I don’t accidentally do anything in production. Sorry. Nor do my teams or any developers who have access to my clouds. These kinds of failures don’t happen in my world. You will not have a mutable environment outside of data storage. Period. Nuke all you want, it will repair and redeploy itself.
Re:We built the fastest CI in the world and it failed by Klugistiono on Tuesday September 12, 2023 @05:44PM (#37488912)
Have you ever used ArgoCD?

No one EVER should run deploy.sh like how you propose it.

ArgoCD abstracts away access to a k8s cluster (login through dex).

It has strategies of how to deploy k8s resources like through retry, replace (which you would need to write manually in your deploy.sh).

It gives you debug output for your deploy process (yes thats a thing, you do not assume every deploy just works).

ArgoCD also can and should be used to reference secerts from secret store like vault.

And it gives you a history/audit.

Re:We built the fastest CI in the world and it failed by withinboredom on Tuesday September 12, 2023 @06:33PM (#37489495)
And fuck-us-all if you go on vacation.

In a small business (read: startup), `deploy.sh` is all you should have. When you go on vacation and it breaks, I just need to read a bit of bash/python/php/whatever-script-you-use.lang and fix the broken stuff.

Re:We built the fastest CI in the world and it failed by Klugistiono on Wednesday September 13, 2023 @05:59PM (#37502302)
I introduced ArgoCD to a project with 20 teams and the acceptance from developers was very high.

What they had befor was a terraform based pipeline (i would argue similiar to a deploy.sh) and they struggled with those scripts hard.

ArgoCD with dex is easily integrated in your auth system and you don't need to write some scripts to handle issues like login errors, timeouts etc.

I have seen plenty of bash scripts which do exactly what they should do with 0 error handling and it was always a pain in the ass to analyse and fix those scripts.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @10:02PM (#37491307)
I used it at a Fortune 500. I’ve used it at a Fortune 100. I’ve used it in a small startup. I stand by my statement. It solves problems self inflicted due to crappy cloud architecture and mutable environments. Running a simple deploy script means your DevOps pipelines and cloud are mature enough to handle rollbacks, failures, blue/green deployments, and any migrations applied after a data snapshot (whatever that looks like in your architecture).
Re:We built the fastest CI in the world and it failed by Klugistiono on Wednesday September 13, 2023 @05:56PM (#37502268)
Very weird that you have such an strong opinion of having some self written deploy script over a argocd setup.

ArgoCD is super easy to setup, has a declarative setup and is a great platform tool.

I introduced it to big systems and i really can't imagine how we would struggle if we would deploy those infra components with some bash scripts.

Could you be more specific?

Like assume you have some monitoring stack (kube-stack or grafana etc.) and either as a helm chart or as something else: What do you do when a CRD needs to be replaced? You write it in your deploy.sh script through some bash magic? Handling error cases in there?

What do you do when you want to see what you have deployed? Like checking if your test setup is rolled back? You always use kubectl to check some helm values? Or do you always annotate all pods with git commits which you then c&p?

A basic ArgoCD declarative setup takes perhaps a few hours if you do it for the first time, than you can give teams access to namespaces, they have log access, exec access if you provide a platform that is, in my opinion, quite a big advantage over just letting them use kubectl. Most developers i know do struggle with kubectl.

And for my own infra ( i also even setup argocd at home ) the dev cycle is much faster for me.

What "crappy cloud architecture" problem are you talking about specificly argocd apparently solves?

How do you test components? Like we have a test cluster and everyone uses argocd to just switch to a dev branch for a few days while testing. This works really well and you can switch back to main in argocd. And due to Argocd App Def you can even see all apps out of sync due to testing.

Re:We built the fastest CI in the world and it failed by ilyt on Tuesday September 12, 2023 @02:36PM (#37485588)
From my experience hosting our own instance of Gitlab + few CI/CD workers was few weeks of initial work (integrating with company's LDAP etc.) then around one ops-week per year worth of maintenance over last few years (50+ developers, dozens and dozens of projects).

Before that we had just "a git server + jenkins + some slaves" that took even less maintenance (althought I'd imagine more fumbling on dev side with job config)

Earthly isn't cheaper than any of those options and none of our clients use it (and we occasionally have to work on client's gitlab instances).

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @03:50PM (#37487045)
You just described a great starting point for a small shop, I would recommend this route 100%.

Initially your CI/CD needs to be close to the code. It needs to be close to developers. Not a DevOps department. It needs to be almost forgettable. Include a config in your repo, bam, welcome to the pipeline.

Where this breaks down is when you don’t have LDAP, you have Azure IDP + Amazon’s Cognito + Keycloak 3rd Parties + SAML (barf) + Form Login for contractors + JWT/OAuth for services.

This is the inflection point where you need DevSecOps + Architecture + Cloud Engineers to ensure everyone’s playing nicely. Enjoy your pods. Let us know when you need that VPN tunnel to your customers backend because they don’t trust SSL.

Re:We built the fastest CI in the world and it failed by ilyt on Wednesday September 13, 2023 @09:40PM (#37503907)
There are actually authentication providers that will pretend to be LDAP server for your apps, regardless of what auth backend you use.

But yeah, once your clients are big corporations you'd at least need to know how to implement some kind of cloud mess of authentication provider(s), still, doesn't need you need to infect your own stack with it.

Re:We built the fastest CI in the world and it failed by maccard on Tuesday September 12, 2023 @04:19PM (#37487612)
We use buildkite but if I was starting fresh now I would just use github with github actions.
Re:We built the fastest CI in the world and it failed by maratc on Tuesday September 12, 2023 @02:38PM (#37485642)
> Swarm’s of Jenkins is what most are used to in the enterprise.

Thanks, I thought we were one of the last places with that setup, nice to know this unholy contraption is actually somewhat standard.

Re:We built the fastest CI in the world and it failed by gabereiser on Tuesday September 12, 2023 @09:36PM (#37491112)
It’s the one true daemon of the enterprise. Unholy. Can’t die. You’ll find it in the darkest places. But it does it’s masters bidding.
Re:We built the fastest CI in the world and it failed by sanderjd on Tuesday September 12, 2023 @03:44PM (#37486908)
I know this is controversial, but I wish commercial "source available" were the more mainstream / not-shat-upon approach to services, rather than "open core".

I find it very frustrating to not be able to read the code or contribute bug fixes or self-host anything outside the "open core", and I have no need for any of the things that distinguish "open source" licenses from "source available" ones.

Re:We built the fastest CI in the world and it failed by dpeckett on Tuesday September 12, 2023 @04:52PM (#37488124)
Stallman was right. I think it's inevitable we'll see the next generation of developer focused tooling go in the direction of AGPL or source available (eg. BSL). Ironically enough Earthly started off as BSL.
Re:We built the fastest CI in the world and it failed by sanderjd on Tuesday September 12, 2023 @05:43PM (#37488894)
What did Stallman say about this?
Re:We built the fastest CI in the world and it failed by MaxBarraclough on Thursday September 14, 2023 @05:24AM (#37506696)
> I have no need for any of the things that distinguish "open source" licenses from "source available" ones

This strikes me as short-term thinking.

If you've ever used OpenSSH or MariaDB, say, then you've benefited from the right to fork, granted by Free and Open Source licences. [0][1] There are of course thousands of other examples.

This is what the FSF calls Freedom 3, [2] and what the OSI's Open Source Definition calls Criterion 3: Derived Works. [3]

Needless to say this kind of thing typically isn't permitted by source-available licences.

[0] https://en.wikipedia.org/wiki/OpenSSH#History

[1] https://en.wikipedia.org/wiki/MariaDB

[2] https://www.gnu.org/philosophy/free-sw.en.html

[3] https://opensource.org/osd/

Re:We built the fastest CI in the world and it failed by sanderjd on Thursday September 14, 2023 @02:19PM (#37512710)
I mean, I'm very glad that open source (and also free) software exists. I don't think there should only be software with source available licenses, and I certainly admire and appreciate the giant amount of foundational work that has been done under those licenses.

But I think it comes with its own trade-offs. For instance, I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.

And I lament that people seem to be less critical of entirely opaque proprietary software than of source available software. I think if the only viable choices are nonprofit+open and commercial+closed, then we are going to end up with more closed software that is of higher quality than its open alternatives.

In general my interest in this topic flows from dissatisfaction with funding models for open source software. I really hate that so much important software relies on under- or even entirely un-paid labor to create and maintain it. I'm not under illusions that alternative models are a panacea, but I'm also not satisfied with the status quo. I get paid well for my work, and I hate that so many people whose labor I rely on aren't getting paid well for theirs!

Re:We built the fastest CI in the world and it failed by MaxBarraclough on Thursday September 14, 2023 @04:11PM (#37514202)
> I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.

I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork.

The FOSS model has a good history of software project longevity, especially compared to, say, proprietary software developed by start-ups.

More generally, the rights granted by FOSS licences are a good (albeit imperfect) protection against software turning user-hostile. To that end, a FOSS licence might not be sufficient, but it seems to be necessary.

> people seem to be less critical of entirely opaque proprietary software than of source available software

The Free Software Foundation used to consider semifree software to be its own subcategory of non-Free software, but they no longer make the distinction. [0] This makes sense given the organisation's goals - they don't want to be seen to be giving partial credit to licences that, while not as bad as some others, still deliberately deny the user the '4 freedoms'.

I agree that non-Free-but-source-available is better than non-Free-and-source-unavailable, but there are good reasons the Free Software and Open Source software movements have the principles they have.

> my interest in this topic flows from dissatisfaction with funding models for open source software.

Agreed. It's frustrating that so few people or organisations think to contribute to FOSS projects, even ones they deeply rely on.

There are ways to monetise aside from patronage/donation, such as paid support and paid managed services, but unfortunately it seems these are rarely successful in generating appreciable income.

[0] https://www.gnu.org/philosophy/categories.en.html#Proprietar...

Re:We built the fastest CI in the world and it failed by sanderjd on Friday September 15, 2023 @12:58PM (#37526027)
> I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork.

Sorry, this was too implicit. What I was alluding to was the period of time when openssh had some showstopper security issues, and everyone realized it was being maintained by essentially one person with essentially no funding.

I don't want to downplay that the copyright owner of source-available software taking it in a bad direction or stopping maintenance or going out of business altogether is a big risk of that model (and of fully proprietary software as well). It totally is, and it is definitely something that the ability to fork solves.

But my point is that it's a trade-off. Both models have failure modes. And they both have (different) imperfect solutions for them.

> Agreed. It's frustrating that so few people or organisations think to contribute to FOSS projects, even ones they deeply rely on.

This is probably where our fundamental difference in philosophy comes from. I don't think it is at all the case that people don't "think to contribute to FOSS projects", it's that the incentives don't work out in favor of it, for most organizations. And I don't think the answer to that is to wish for altruism from organizations and free labor from individuals.

> There are ways to monetise aside from patronage/donation, such as paid support and paid managed services, but unfortunately it seems these are rarely successful in generating appreciable income.

Yeah, as I said, I find all of these solutions deeply unsatisfying. Patronage/donation has the "relying on altruism" issue. Paid support misaligns incentives, because if you manage to make perfect software, you've also managed to put yourself out of business, whereas if your software is difficult to use, that makes it lucrative. Paid managed services would be great, if Amazon couldn't just slap your code behind an official AWS service that 99% of people are going to use instead of your also-ran; but keeping them from doing that requires the restraint on commercial re-distribution that everyone seems to hate.

Re:We built the fastest CI in the world and it failed by AlphaSite on Wednesday September 13, 2023 @01:23AM (#37492520)
I think the main competitor is gitlab and GitHub, it’s very hard to compete with them. It’s not competing with earthly OSS. Personally the reason we haven’t used earthly satellites or CI is because we can’t use them inside the corporate firewalls, nor does it integrate with our existing CI tools super well, e.g. gitlab.

We use it and love it but it’s missing important enterprise features like that and SSO.

Re:We built the fastest CI in the world and it failed by XCSme on Thursday September 14, 2023 @08:48AM (#37508249)
This is why I don't believe in businesses that offer both open-source and a paid cloud version. Their business goal is literarily: make sure the open-source version is not so good and easy to use/install/maintain, so that people do need to use our cloud offering.