News for hackers. Stuff that matters.
Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @03:55PM (#36437861)
I would like to add the perspective of a small-business infrastructure guy here.

First things first: we're a Rocky Linux shop. About 100 instances, some bare metal, some VMs. A few local, a few at cloud providers like DigitalOcean. Held together by Ansible and kickstart installs.

Too often the discussion about RHEL clones like Rocky paints a picture of people like me as freeloaders. And some of the folks with a similar setup might be. But most small-time Linux admins I know are real open-source believers. I would really like to funnel some money into the platform we're using. First and foremost because it's the right thing to do, the people writing all that source code deserve it. But also because we need platform continuity, and being able to open a support case from time to time would also be great.

But it's really hard. I don't have the time and nerves to deal with the RHEL sales department, deal with the licensing, produce my own cloud images because DigitalOcean can't provide them, deal with our own internal purchasing process, deal with convincing my superiors that we need to.

Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

Re:Dear Red Hat: Are you dumb? by kbenson on Thursday June 22, 2023 @04:04PM (#36438033)
We're the same, but a little larger. Don't let anyone convince you you're a freeloader. If you're a freeloader, then Red Hat are freeloaders for relying on the Linux Kernel and all the open source software in their distros as well.

What we both are are power users that don't really need their support. Our company has an entitlement so we can use the KB articles (to our benefit) and view/submit bug reports to them (which is to their benefit). I have no problems telling them that (and for the most part did last week), and if someone on the call were to argue me on that point, I'd have happily gotten into it with them. I know they're adding a lot to linux and open source, and pay for a lot of developers to maintain and add to projects (including Linux itself), but that's just the point, they're adding to something that already exists that they use. They don't own it, and they can't expect to require payment for it. They aren't Microsoft, and the only reason they've ever been able to compete in that space is because they're standing on the shoulders of what the open source community has made possible.

It's a symbiotic relationship, and they can mess with it at their peril.

Re:Dear Red Hat: Are you dumb? by op00to on Thursday June 22, 2023 @04:18PM (#36438269)
> then Red Hat are freeloaders for relying on the Linux Kernel

"freeloaders"

Most Active Linux 6.1 Employers [1]

By Changeset

[...]

7. Red Hat 672 4.8%

By Lines Changed

[...]

10. Red Hat 24073 3.1%

Most active employers, 5.16 through 6.1

By changesets

3. Red Hat 4916 5.7% (behind Google and "Unknown")

By lines changed

8. Red Hat 202698 3.3%

[1] https://lwn.net/Articles/915435/

Re:Dear Red Hat: Are you dumb? by autoexecdotbat on Thursday June 22, 2023 @04:36PM (#36438585)
> If you're a freeloader...

It's dishonest to leave that conditional off the quote. The work and risk/stress involved in running a supportless shop is quite high. So if they are freeloading by using Rocky then Red Hat is also freeloading.

I do think it's funny that they weren't even number 1 on that list though. Guess they are at least getting a lot of volunteer work from other companies!

Re:Dear Red Hat: Are you dumb? by op00to on Thursday June 22, 2023 @08:04PM (#36440976)
The others are almost all hardware manufacturers doing new hardware enablement.
Re:Dear Red Hat: Are you dumb? by kbenson on Thursday June 22, 2023 @04:42PM (#36438663)
It's hard to tell without more context of your own whether you're supporting what I said, or refuting the argument that Red Hat are freeloaders, which is an argument I did not make.
Re:Dear Red Hat: Are you dumb? by dralley on Thursday June 22, 2023 @04:33PM (#36438535)
There's also different types of contributions. Hardware vendors like Intel and AMD are mostly contributing their own hardware drivers (and a lot of that, especially in AMD's case, is just autogenerated hardware definitions), whereas Red Hat is mostly contributing to core kernel functionality.
Re:Dear Red Hat: Are you dumb? by freedomben on Thursday June 22, 2023 @04:26PM (#36438428)
and that's just the kernel. Red Hat also runs or contributes to a bunch of other projects that are significant, especially relating to containers.
Re:Dear Red Hat: Are you dumb? by StillBored on Thursday June 22, 2023 @05:37PM (#36439399)
Not just containers, all the core linux stuff. gcc, binutils, gnome, systemd, etc.

One of the major differences between buying a support contract with RH and buying one from literally any other vendor is that RH doesn't sell support for things without having someone active in that project's community.

AKA, there is a good chance the gatekeeper that your !RH support provider is talking to is a RH employee or funded by RH. Put another way, rocky can't afford to hire 1k engineers to maintain all those packages, nor can anyone else so your trusting that the bug you want fixed can be fixed by a drive by contributor. This is part of the reason that RHEL has such a small package list too, just about every single package in that list has someone working upstream.

So, you won't get stuck like i did 15 years ago with a glibc bug that the company your paying a support contract to can't fix. Sure that distro patched their own version, but the change got dropped a couple years later during a rebase, and it was the only distro with the fix. Meaning our product only ran on that specific distro + version until we reopened the bug with RH and they actually got it fixed upstream.

Re:Dear Red Hat: Are you dumb? by kbenson on Thursday June 22, 2023 @05:55PM (#36439613)
To be clear, you're less likely to get stuck like that. There's plenty of bugs open that Red Hat doesn't fix because it's either too problematic or affects too few people. Sometimes that's eventually solved with a point release and them rebasing to a newer package version which includes an upstream fix, sometimes it just persists. I'm not saying it's common, but it's happened, and happened to us. Encountering a bug and finding the KB article with only workarounds and then a bugzilla entry that's been open for a couple years isn't exactly common, but I also wouldn't count it as all that rare in my experience.

Also to be clear, I'm wasn't making a case abovec that Red Hat doesn't provide a lot of benefit to the open source community, just that Red Hat could not exist without that same community. I do not begrudge Red Hat making money. I do think they're being extremely shortsighted if and when they shut off the larger open source community's uses of their core OS product. They're making their whole ecosystem much harder to get talent for if they do that, and it's not like entities running Rocky or Alma are just going to spring for RHEL. We trade on our staff's skill as System/Linux Administrators, and the core OS is not as important as all that. It's like if Python started requiring licensing fees. You'd get some short term return on that and some larger entities will just pay, but you'd see python usage in the general community plummet, with all that entails about its future.

Maybe to their eyes it makes more sense in this current world of startups and VC to just exclude the smaller players, since those companies that grow organically to a size where Red Hat's pricing makes sense are much rarer?

Re:Dear Red Hat: Are you dumb? by nisegami on Friday June 23, 2023 @08:38AM (#36446019)
>Not just containers, all the core linux stuff. gcc, binutils, gnome, systemd, etc.

Depending on who you're talking to, this may not be a good thing.

Re:Dear Red Hat: Are you dumb? by justinclift on Thursday June 22, 2023 @06:32PM (#36440082)
... and looks like they're locking many of those changes to the source code behind their own paywall.

While it may technically be legal (ie crafted by lawyers to make sure of it), that still seems like a pretty shitty thing to do.

Re:Dear Red Hat: Are you dumb? by fariszr on Thursday June 22, 2023 @07:15PM (#36440535)
Red hat upstreams a lot of their patches, much more than Debian or Ubuntu.
Re:Dear Red Hat: Are you dumb? by justinclift on Thursday June 22, 2023 @10:13PM (#36441996)
I'm aware, I used to work for Red Hat (2010-2015). :)
Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @04:21PM (#36438314)
Thanks for the feedback!

We've just about finished transitioning to 100% Rocky, coming from a 50/50 setup with many of the bare metal installs being Ubuntu. But Ubuntu left a real bitter taste in my mouth with the snap enforcement (I can't even start this snap thing in my Ubuntu Docker container? And you took away the native apt? Wow.), so we started a migration to 100% Rocky.

Telling my manager that we'll have to work something out after we just about finished working something out will be my own peril as well. Let's see where this will end.

Re:Dear Red Hat: Are you dumb? by freedomben on Thursday June 22, 2023 @04:28PM (#36438450)
I would wait a bit to see how this shakes out before raising the alarm. I'm waiting to see what Alma/Rocky do, as well as holding out a small amount of hope that RH and others will agree to something.
Re:Dear Red Hat: Are you dumb? by kbenson on Thursday June 22, 2023 @04:52PM (#36438795)
Yeah, we were late on CentOS 8, and had just put 3-4 systems and VMs into production when the Stream announcement hit, which made us stop in our tracks. We eventually accepted Rocky and started actually getting newer systems in place once 9 was out, so we just jumped to using 9 and in-place upgraded our Rocky 8 systems to 9.

We're only about a quarter migrated though, so still have a few hundred Centos 7 boxes to eventually get off of, and we'll see if that ends up being Rocky or something else.

I will note that this episode of The FOSS Pod[1] with a Debian lead actually made me think switching to Debian itself way back in the past (since we've been doing this for almost three decades now) would have been a safer path. I highly recommend it, it cleared up a few misconceptions I had as a Red Hat/CentOS user about what Debian is really attempting and why, and how that would fit with our needs. For example why their goal of pure open source licenses isn't an ideological choice, but a choice to benefit the Debian ecosystem, since the goal is to make downstream distros of Debian easy and without problem (e.g. no extra legal review needed based on licensed included).

1: https://fosspod.content.town/episodes/debian-linux-with-jona...

Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @05:19PM (#36439173)
Same here, we still have 2 or 3 CentOS 8 to Rocky conversions to go from before we even started replacing Ubuntu!

I have to admit, while I share a lot of the values of Debian, I never really liked the product. Especially the apt package mangler, err, manager. They've also carried some Debian in-house patches that really caused me some grief in the past.

I've started running FreeBSD on my personal servers. Unfortunately it's not an option at work.

Re:Dear Red Hat: Are you dumb? by kbenson on Thursday June 22, 2023 @05:44PM (#36439485)
I've never really Debian it in the past, so can't comment on the reality, just what I've heard, and even my Ubuntu experience is lacking. I'm running it as a virtual desktop now, but I try to keep my use of it to an MUA, web browser and terminals so I don't have to spend the time to become familiar with it, given it's got no other use for me so far.

Our director loves FreeBSD, and we actually use it for firewalls, so it's not a spectacularly hard sell. I'm not sure it's significantly different from the problems of Stream to make it worthwhile, at least not without some external patching and support entity in addition, but it's been a long time since I looked at the specifics of their packaging and ports systems. I don't actually mind something like Stream for a personal server actually. It's just when trying to manage a fleet in the tens and hundreds when it becomes a real liability and increased possible admin support load IMO.

Re:Dear Red Hat: Are you dumb? by bayindirh on Thursday June 22, 2023 @07:43PM (#36440802)
Asking from curiosity, what is the problem you had with apt? I'm using Debian close to 20 years now, and liked Debian and .deb a lot, more then RPM even.

Honestly, having built some packages and maintaining a derivative provided me more insight than a regular user, but even before that I was using Debian without any problems.

BTW, Ubuntu is not Debian with a different UI. They're pretty different on fundamental things, even.

Re:Dear Red Hat: Are you dumb? by folmar on Sunday June 25, 2023 @06:14PM (#36473171)
A broken pre/post install script (often the problem is not expecting the current state of your machine) that fails while running `apt-get install -f` it the common culprit.
Re:Dear Red Hat: Are you dumb? by mwcampbell on Thursday June 22, 2023 @06:42PM (#36440187)
Just curious, why didn't you migrate from Ubuntu to Debian? Wouldn't that have been a smaller change?
Re:Dear Red Hat: Are you dumb? by avhception on Friday June 23, 2023 @04:03AM (#36444095)
Because we already had a solid base of RockyLinux VMs, so standardizing made sense. And I personally dislike apt / Debian.
Re:Dear Red Hat: Are you dumb? by izacus on Thursday June 22, 2023 @05:05PM (#36438999)
Did you really compare our freeloading shop against one of the biggest software contributors to Linux and supporting ecosystem? RH is a top 5 contributor of patches across the most important core components.
Re:Dear Red Hat: Are you dumb? by kbenson on Thursday June 22, 2023 @07:51PM (#36440866)
Red Hat adds to the open source projects, they didn't write them from scratch (in most cases). Using freely provided software and building your business on it is perfectly acceptable, whether it's us using Red Hat's sources or Red Hat using the Linux Kernel sources and hundreds to thousands of other open source projects. We extend the OS where it doesn't fit our needs, and submit patches and feedback and bugs to both Red Hat through a subscription for a system we have for just that purpose, as well as to the upstream projects RHEL is based on, just as Red Hat contributes to the upstream projects.

So yes, I made that comparison. If you think a company using a RHEL derivative product is freeloading, I'm not sure how you can believe Red Hat isn't also freeloading on the open source community, but feel free to explain you position.

I, for one, think neither is freeloading, but just making use of the open source community to provide extra value for ourselves and our customers.

Re:Dear Red Hat: Are you dumb? by jzb on Thursday June 22, 2023 @04:23PM (#36438348)
"Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing."

This is a two-way street. It's not all on Red Hat. I don't know to what extent Red Hat has or hasn't tried to do a deal with Digital Ocean but it's not as simple as Red Hat just deciding to do it. There's a fair amount of certification work that DO would have to do as well as hammering out the legal / business agreements.

The "comparable" part exists with many cloud providers, so if you're not married to Digital Ocean then that part should be doable. Red Hat also has some programs around subscription portability.

However, I do get it that the subscription process introduces friction and for small shops that have a patchwork infrastructure as you describe, it's going to be difficult. Some of that is on Red Hat to solve if they want your money, but not all of it. They can't do it all themselves by fiat if Digital Ocean isn't willing to meet them halfway.

Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @04:38PM (#36438615)
We're not married to DigitalOcean, but changing cloud providers also takes time and investment we'd rather spend elsewhere. And we'd still have to figure out what to do with the bare metal instances, container images and whatnot. The friction is very real.
Re:Dear Red Hat: Are you dumb? by ikiris on Thursday June 22, 2023 @06:49PM (#36440262)
you have to understand, why would digital ocean want to deal with this when they can use actual open source options without the overhead?
Re:Dear Red Hat: Are you dumb? by avhception on Friday June 23, 2023 @04:09AM (#36444123)
Because they might some day find themselves in a tight spot if all of those open source options have become a half-broken wasteland because nobody invests time/money.
Re:Dear Red Hat: Are you dumb? by ikiris on Friday June 23, 2023 @09:19PM (#36454462)
being afraid of all free linux distros going away because no one invests in open source is some epic level FUD.

How exactly do you think those companies even came to exist again?

Re:Dear Red Hat: Are you dumb? by jehb on Thursday June 22, 2023 @04:08PM (#36438110)
> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

I think they do this on AWS and Azure (and maybe GCE?). DO may just not see enough of their enterprise customers asking for it for them to dispatch the lawyers and negotiators to make this happen.

Disclosure: Ex-Red Hatter, but never got anywhere close to anything related to sales or licensing.

Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @05:31PM (#36439312)
Thanks for the hint! I will keep it in mind. Still waiting for the situation to really calm down. We'd rather not spend precious time and effort on changing clouds, and even if we did we'd still need to figure out the situation regarding bare metal and container images.
Re:Dear Red Hat: Are you dumb? by progmetaldev on Thursday June 22, 2023 @06:06PM (#36439777)
Most likely switching from DO to AWS or Azure is going to greatly increase your costs
Re:Dear Red Hat: Are you dumb? by freedomben on Thursday June 22, 2023 @04:25PM (#36438393)
From your description, IMHO you're not a freeloader, especially since you want to contribute. You are the exact type of people that I think Red Hat is harming themselves with.

I'm in basically the same boat, currently using Linode. At other companies I've made decisions to buy RHEL and OpenShift, and would again because I've been pretty happy (we'll see how this current thing shakes out).

I also went to work for Red Hat (and took a pay and title cut) largely because I had so much love for them. I'd been using Fedora and CentOS (pre-stream) for many years. The expertise that I gained from the experience also let me hit the ground running in my new job, and I made them a lot of money. Had CentOS not existed, I would never have learned their ecosystem and certainly wouldn't have gone to work there. That's profit that is hard to measure, and it is hard to directly tie it to somebody's commission or KPIs, but I'm sure it's not unsubstantial. Many of the people I worked with had a similar backstory!

I would also say (at least when I worked there) I didn't know anybody (who wasn't a sales person) who would have called us free loaders. Quite the opposite, they considered us part of the community.

Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @05:36PM (#36439395)
This is essentially my backstory as well. I'm typing this on a Thinkpad running Fedora. RedHat seems to be focused on big corp and government customers. But even they are not an island. Loosing the grassroots will really hurt them in the long term I think.
Re:Dear Red Hat: Are you dumb? by AlgorithmicTime on Thursday June 22, 2023 @04:16PM (#36438250)
>Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

We've tried to make deals with RH in the past, but they will only let us sell RHEL VMs with licenses if it's running on a RHEL hypervisor... which none of our platform is using at all.

Re:Dear Red Hat: Are you dumb? by freedomben on Thursday June 22, 2023 @04:33PM (#36438534)
I think this sucks, but for a little bit of defense of RH on this the entire support stack is built around assuming a RHEL host (meaning you're using a supported and exact version of the hypervisor and associated tools). Adding support capabilities for a non-RHEL host would be a major task, and not one they are well equipped to do.

Now that said it's a bit dumb because with most things it's not hard for support to tell your customer, "yep that's a problem in the guest" and help, or "that's a problem with the host, contact DO" or whoever.

But in general, when Red Hat says something is "supported" they are guaranteeing a lot about it. They'll literally fix the code if something is wrong. There's also a culture internally of "be careful helping with something that is 'unsupported'" because customers have previously accused RH people of messing stuff up and demanding they fix it. I helped with stuff like that most of the time when I worked for RH, but I was always very clear with the customer that "this isn't Red Hat, this is me helping as a friend and fellow Linux nerd" and the rapport with the customers was always good enough that I wasn't worried.

Re:Dear Red Hat: Are you dumb? by progmetaldev on Thursday June 22, 2023 @06:12PM (#36439841)
As someone who has done this, and benefitted from others doing this, I think the open sharing is probably the number one reason to use these types of solutions. You get people that are in their own niche and know specific parts of the system inside and out. That builds credibility and increased profits, but the companies are too nervous to stand behind their people.
Re:Dear Red Hat: Are you dumb? by bonzini on Thursday June 22, 2023 @06:22PM (#36439961)
> you're using a supported and exact version of the hypervisor and associated tools

Having exact versions is not a problem (any version of RHEL guests are supported on any RHEL hosts past and future, almost). Plus Windows, ESX, Amazon and Google clouds, and a few more.

Rather, the problem is having _up to date_ versions; with RHEL you can often ask the customer to reproduce on an up to date host if possible, with other distros the support machinery isn't there and you need someone to talk to at the cloud provider, so that they can debug stuff that breaks only in their environment and can't be reproduced. Otherwise you cannot guarantee the level of support that you mention. Very few cloud provider can provide that, I am not even sure that IBM's cloud made the short list.

Re:Dear Red Hat: Are you dumb? by XorNot on Thursday June 22, 2023 @06:45PM (#36440222)
Isn't long term support Red Hat's thing? The ask isn't to support some random system, it's for RH to put out an official image and distro they support on cloud providers and bake the licensing costs into the hourly runtime fees.

This is exactly how you consume Windows on AWS and it's a huge incentive to do it that way because it takes licences management out of consideration.

Re:Dear Red Hat: Are you dumb? by bonzini on Thursday June 22, 2023 @06:49PM (#36440270)
The cloud provider is also running software, the cloud doesn't run on magic dust. Red Hat doesn't supports RHEL on your blood if they don't believe they can trust you to help supporting RHEL, the way their customers expect. The easiest way is to run RHEL as the bare metal OS but it's not the only way.

> This is exactly how you consume Windows on AWS and it's a huge incentive to do it that way because it takes licences management out of consideration.

It's also how you consume RHEL on AWS, GCE or Azure.

Re:Dear Red Hat: Are you dumb? by bonzini on Friday June 23, 2023 @02:57AM (#36443698)
Blood -> cloud of course.
Re:Dear Red Hat: Are you dumb? by avhception on Thursday June 22, 2023 @05:48PM (#36439533)
That's sad.

And we'd have to figure out what to do with the bare metal instances and containers, too.

Re:Dear Red Hat: Are you dumb? by jsight on Thursday June 22, 2023 @04:50PM (#36438766)
Honestly, there are a ton of people like you out there. A big selling point for Red Hat has always been their contributions to the community and that funding them would help to drive that.

It is a big motivator for people transitioning from your state to becoming a customer.

Re:Dear Red Hat: Are you dumb? by Twirrim on Thursday June 22, 2023 @05:06PM (#36439013)
> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

Most of the major clouds have this pre-existing capability to charge more and revenue share the extra through their marketplace services etc; and they're in active use by other vendors. It's often a good source of revenue for both the cloud company and the vendor, and helps the customers by providing them with prebuild solutions that they don't have to build themselves (e.g. web application firewall vendors can/do provide closed source images that customers can use that way). It's not like this is expecting cloud companies to build whole new services from whole cloth just to support Red Hat.

Re:Dear Red Hat: Are you dumb? by dralley on Thursday June 22, 2023 @05:48PM (#36439534)
>Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

This 100% already exists, but I don't believe Digital Ocean is one of the supported vendors.

Re:Dear Red Hat: Are you dumb? by bayindirh on Thursday June 22, 2023 @06:00PM (#36439681)
You with 100 instances is pretty sizeable. On the other hand, I know research infrastructures (read: thousands of instances around Europe) built with CentOS 7, Alma & Rocky.

These guys are also power users and developers who need no support.

This is a massive blow by RedHat, and I'm not sure that they're aware of the size of the install base they are attacking to.

Re:Dear Red Hat: Are you dumb? by yrro on Monday June 26, 2023 @07:43AM (#36478442)
> Please, dear Red Hat, just give me a DigitalOcean VM (or something comparable) that costs a few dollars more and has automatic RHEL licensing. Make deals with the VM providers, work something out.

This exists, but probably only for the larger cloud providers and not e.g., DigitalOcean: https://azuremarketplace.microsoft.com/en-GB/marketplace/app...