Sorry, but what's the hard part to understand? Open source maintainers end up in this position because they are nice, helpful people who like using computers to solve problems for others. People who spend years on a project and then see a bigger problem arise don't suddenly turn that off. With the bigger problem, they'll want to work harder, not just hoist a middle finger and go binge Netflix without a care in the world.
But I totally agree with you on the CTOs, etc. I don't expect random programmers who like working on logging to also be good at solving complicated sociotechological problems around paying for global infrastructure. But it boggles my mind that none of these richly rewarded, supposedly brilliant experts at organizing engineers has gotten out in front of this. If not out of community spirit or social responsibility, then out of pure self interest.
Indeed. Each of them has had to spend the last few days madly trying to fix this problem to avoid exposing exposing their infrastructure. Each has been, in some way, replicating the wheel to do so. I'm curious how many will actually submit their findings to the original OSS so others can learn from their experience?
There's always resources to put a fire but rarely enough to install a sprinkler system.
There is a perfectly healthy, acceptable, middle ground between those two extremes, however.
Feel free to prove me wrong by pointing to the major open-source project you've led for years and how good you have been at ignoring problems with it.
I have maintained open-source projects myself for about 15 years, and it is completely different from sysadmin alarms. I can do it on my time, and even if with vulnerabilities I do it on the time I have previously allotted for doing it, since my family and my job comes first. And no, I won't dox myself. There, I "just" said "no" to you. That's a healthy way of establishing boundaries.
By the way, feel free to prove that I'm wrong in saying that by pointing to any law that says I have to do otherwise.
And no, just because people have unhealthy behaviours doesn't make it a rule or a law.
I agree that unhealthy behaviors don't make it a rule. But deep-set behaviors don't change overnight. And it's not clear to me that people who are perfectly healthy would ever put themselves in the situation of maintaining important infrastructure for free. As mentioned elsewhere, I've closed down an open-source project of mine because it grew too big and became more of a drain than a pleasure. But if everybody did that, we'd have a lot fewer open-source projects. So I think your breezy "just" only works for the kind of people who would never have ended up with this problem in the first place.
Rather than giving up and closing a project, my answer to abuse and unreasonable demands is to be liberal with ignoring and blocking. It's not my fault that someone chose to berate me, but it's my choice to keep giving them a way to do it. But most of the time, a simple "I will do it when I have time" is more than enough. I believe in carefully curating the online places I manage in order to maintain a healthy atmosphere. I choose health and well-being above "popularity at all costs".
About the projects, I'm not gonna dox myself, and I don't think the cross-examination is warranted or even in the spirit of this site. I believe my post stands by itself. You can doubt all you want, but that only solidifies my belief I'm doing the right thing.
I am saying you are delusional to talk as if what's easy for you is what's easy for others. And that you're failing to consider that people who are in XKCD's "random person in Nebraska" bucket [1] are much less likely to have good habits and healthy boundaries in the first place, because people like that end up bailing much earlier.
The myth of the hero maintainer must die.
Binging Netflix is completely acceptable if you're in your free time. EVEN if the package has a RCE vulnerability. And most maintainers of popular projects will agree with that.
I'm glad it's easy for you. But therapists can spend years working with people on establishing and maintaining boundaries in important relationships. You can't just handwave the difficulty away. Especially when so many tech jobs discourage having a good sense of boundaries in the first place. E.g., all the startups where people are expected to be bought in to a vision of changing the world, working crazy hours. All the bosses that talk about their teams being like families. All the places that talk up commitment and loyalty and going the extra mile. Which again you can see a flood of if you care to look: https://www.reddit.com/r/antiwork/
> And most maintainers of popular projects will agree with that.
[citation needed]
I'd likewise ask for a citation of some maintainer agreeing that watching Netflix on their free time is akin to "giving the middle finger" to users. That was one of the most toxic phrases related to OSS maintainer work I've ever seen on this website, and those views should not be spread as if they were acceptable. I stand by my opinion that the "work harder" vs "watch Netflix" is a completely fabricated dichotomy, and there is a very viable middle ground available (and necessary) for everyone.
FAANG engineer here, and one who had to work extra hours to redeploy services with the log4j vulnerability fix. I'm not sure you understand the scope and constraints of this sort of problem. Log4j's maintainers have a far more difficult and challenging job than FANGs or any other consumer of a FLOSS package, who only need to consider their own personal internal constraints, and if push comes to shove can even force backwards-incompatible changes. The priority of any company, FANG or not, is to plug their own security holes ASAP. Until that's addressed the thought of diverting resources to fix someone else's security issues doesn't even register on the radar. I mean, are you willing to spend your weekend working around the clock to fix my problems? Why do you expect others like me to do that, then? Instead I'm spending a relaxing weekend with my family with the confort of knowing my service is safe. Why wouldn't I?
I'm saying that your company's CTO, especially one with a very large companies, could likely identify two or three engineers who they pull into a meeting and say "reach out to these guys and get them whatever they need. Here's my cell, call me the moment you need the plane or additional resources."
Seriously, if a CTO has a budget of a few hundred million dollars and thousands of dedicated employees, how hard is it to throw a few crumbs to the open source community to change this situation from being one of a burden on a volunteer effort to, instead, one where they feel like they're in the middle of an international event where their knowledge and services are vital to keeping the internet alive?
Again, I'm exaggerating, but you see where I'm going with this. It's a missed opportunity for some seriously great PR out of a seriously bad situation.
It's not hard to extend this to the kind of software security flaw here. If I'm a developer on a package with a critical security vulnerability that needs to be fixed now, sending me extra developers who know absolutely nothing about the code I'm working on isn't going to be helpful--it's just going to waste my time trying to bring them up to speed (or more often, telling them to just go away). If I actually need help, I'll ask the people who I know can help me; trying to sift through unsolicited help to figure out who actually has the skills to do so would take too much time.
So think hard about what help Google et al could actually be providing to help log4j here. If you have to resort to clear exaggeration to find examples... maybe that's a sign that there actually isn't all that much that they could be doing that would actually be helpful.
I think you make a perfectly valid point and one that shouldn't be overlooked. How about this:
"Here's a $100K and an isolated penthouse suite down the road rented for the month where you can focus on fixing the problem and not be interrupted by screaming children. Here's a phone number if you need to delegate any specific tasks to additional teams."
Incentive to help. No added pressure. Just one practical example.
If children and family were viewed as too much of a distraction, I’m pretty sure the CTO (in this scenario) would simply choose a developer who lacks those distractions.
Let’s say the engineers chosen do have family. Why wouldn’t the company just comp a room in a local hotel?
In this instance either would be reasonable.
Have a team handle all github issues and media inquiries. Another team focus on initially evaluating all incoming pull requests to check for egregious errors or applicability.
Only after making it through the gauntlet would the original maintainers need to read and/or respond to them.
Especially when such overwhelming public attention and pressure overtakes a relatively small team like this one.
As with openssl, what needs to be done is that these volunteers need to be given cash so this is more than just a volunteer project. If a particular corporate entity doesn’t want to sponsor some of the maintainers to work on it full time, then the project needs full-time sponsorship by the Linux Foundation, ASF or under the OpenJDK.
But that is not to say that nothing is being done. At Google, at least, there are organized efforts staffed with plenty of people that are trying to solve the much much much bigger problem of "secure all of our open source dependencies and all future dependencies" rather than the individual problem of "secure this one dependency."
And PR? Google has been running projects like OSSFuzz for years and I haven't really seen it materialize as a large amount of positive PR, even in the tech community.
Google's Project Zero is both very helpful and gets them A LOT of PR, both tech and mainstream.
If all your IT projects have an RCE vulnerability that’s relatively easy to exploit, that should keep you up at night.
The actual solutions here are at a more abstract layer than individual vulns.
A week ago, this vulnerability might have been known at most to a few three-letter agencies. Today, every two-bit script kiddie will be trying to exploit it. It's not hard to see how the situation has changed.
Apache has processes for this, and project maintainers pointed people in that direction repeatedly (e.g. https://github.com/apache/logging-log4j2/pull/608#issuecomme..., https://github.com/apache/logging-log4j2/pull/608#issuecomme...).
The Apache foundation receives funding from a large number of organizations already: https://www.apache.org/foundation/thanks.html
Perhaps the right question to ask here is: what did Apache do to help their members in this event?
You can ask this question of the Apache foundation independently, without adding pressure on the project maintainers at this time.
"Let's get 3 of your best guys in touch with the guys over at log4j...."
As if you can just wire up some engineers to reach out to each other on unrelated projects and magic will happen.
Adding random new people generally does not immediately speeds up the process, but it is actively harmful in the middle of crisis.
Surely the difference is you are getting paid, and if your boss says, help these guys out, you can do it? As opposed to some guys with jobs who have a project on the side. The big guys could even do something like offer to pay the maintainers and maybe they can take leave or something.
I agree with both sentiments. The big guys are under no obligation to fix an issue in some library they happen to use. But the log4j guys are under even less obligation when they do it in their spare time.
Everyone should enjoy their weekends.
No, I'm not getting paid. What leads you to believe in that? My targets are defined yearly and are very well defined, and patching random FLOSS projects is not one of them. And what leads you to believe that others, such as my boss, don't have their own milestones to meet, and instead take random FLOSS requests from random people on the internet?
A FANG is not a magical entity where any engineer can drop everything they're doing at the drop of a hat to work on external projects, let alone one whose only possible outcome is at best total indifference and at worse we get the company to own a problem affecting everyone for no reason whatsoever.
Something I believe we agree on is that it is in the interest of large tech companies to spend time fixing critical security bugs in their own programs, regardless of who originally wrote the malfunctioning code and for whom said code was written.
One way to fix those bugs would be to create a patch for the external OSS library in instances where such a library is the origin of the vulnerability. This is especially practical when that library is used heavily as a basic piece of the company's common software development framework.
GP appears to be arguing that these patches should be upstreamed instead of simply being maintained internally until the bug is patched by someone else in the OSS community.
If all our projects suffer from a known RCE vulnerability, I’m fairly certain my boss will be happy for me to drop everything to get it resolved.
That’s not nearly at FAANG, but it’s certainly not an undue burden on the schedule.
If the big guys don’t want to do that, fair enough. But the open source maintainers are not under any obligation to work to anyones time lines either.
Your "(...)" elides the word "help," which completely changes the meaning of the quote, and your reply is constructed uncharitably as if that word wasn't in the original statement.
> Log4j's maintainers have a far more difficult and challenging job than FANGs
You are saying that the companies that built advanced ML-based Chess/Go engines like Alpha Zero/Go can't solve a simple logging bug involving string substitution?
If your company ends up using the product in all your teams/project and products wouldn't it be in the company's interest to keep the product safe?
How do we know you're not a CTO/C--/manager in your 'faang' just taking this opportunity to bitch about how bad and unreliable open source is? You do have a track record when it comes to this.
> I mean, are you willing to spend your weekend working around the clock to fix my problems?
Wow, that's cynical even for a 'faang' dude.
But as a professional representative of your org. surely you'll recognized the unsustainability of the situation and that it's far from ideal even in the pure self-interest of the company in question.
Agreed, while reading it I also disagreed at this point:
>> the maintainers of log4j would have loved to remove this bad feature long ago, but could not because of the backwards compatibility promises they are held to.
Nobody is holding them to anything. If they want to remove an old feature, go right ahead. If those using it think it's that important they can fork the project and maintain it themselves. Oh right, that would take effort or money.
I don't get this argument. Part of sharing your work is making sure what you put out is actually helpful to people. If they remove features people really like, then the library won't be as helpful - so it's perfectly fine for the OG devs to maintain this feature. The same thing with "scrambling" to fix - that could be because a sword is hanging over your head, or because you care about the people who use your work. Thinking this way, I can perfectly see them working hard to fixing this bug.
Developers are social creatures like anyone else and like validation and recognition from peers, that is understandable but not an excuse.
It is no different from being able to say no when your boss ask you to build something you don't believe in or is against your morals, it is tough but our choice nonetheless.
Their choice then to support this feature, and now to patch it. They could have no either time, no point in complaining about it after making the choice.
No, people get to decide for themselves what is helpful to them. Assuming developers want to make the best tool they can, they still have to do so within the resource constraints they have. Dropping a feature or ignoring requests is part of that.
If so, I really wouldn’t hold someone to any backwards compatibility promise if security is a concern.
How much you paid for it? Money back guaranteed. If you paid $0 then it you are guaranteed to get $0 money back in the case of an issue.
A past organization of mine had a data breach (the kind that ended up making the news everywhere). A few people left (probably making it worse with all the turnover there), but I would be surprised if anything really changed in that organization.
I'll note that there's been more investment in security the last 4-5 years. Most B2B companies do a SOC2, and early on, so there tends to be a baseline of competence.
If the RCE had been responsibly disclosed instead of via tweets and PR comments, maybe there wouldn't have had to be so much scrambling. And indeed maybe ASF could have found corporate OSPOs to help with remediation.
There are lots pixels being spilled on how the users of open source software should be paying for it (?), but I haven't seen much criticism of the vulnerability not being responsibly disclosed.
I wonder if this wasn’t intended for something bigger and they just got caught when testing this out in the field.
If you sent a message to everyone with a client (that logged that message), everyone with a client would at least ping back to the ldap server.
The issue where this was introduced was: https://issues.apache.org/jira/browse/LOG4J2-313
That's from 2013 (Minecraft was 2 years old then) On the other hand, nothing with it really happened until someone asked if it was a security vulnerability - https://github.com/apache/logging-log4j2/pull/608#issuecomme...
And then all hell broke loose.
Simply said, this was a feature that was intended to make it easier for one company to use their structure with ldap lookups for where/how to log. The author of the change did what many people encourage others to do when working with open source "here's some code that I wrote, I'm contributing it back upstream."
If this was part of something "bigger", it sat quiet for the better part of a decade.
No consideration, no discussion, no security analysis, just “JDNI is cool, can I hav plz? Ofc!”
Did none of these people consider what JDNI is designed to do?
Did none of these people consider what side-effects are appropriate within a logging library?
As an open source developer working on a project that hadn't even been formally released, I'd be quite pleased to have someone else contributing the features that they found useful back upstream in an effort to make it a better project.
Yes, this is what jndi is supposed to do. Was it done as best as it could be? Probably not. But it isn't something that's only in log4j2
http://logback.qos.ch/manual/loggingSeparation.html#ContextJ...
https://dennis-xlc.gitbooks.io/the-logback-manual/content/en...
But I'm not going to fault a solo developer of some beta software in the world of 2013 for not rejecting a patch because every angle wasn't thought out.
Then three days ago someone asked "Is it a security vulnerability" and then everything happened.
This wasn't any great "there's a security bug that needs to be patched yesterday" and then a fix and release but rather an after the fact, when looking at it, was it realized the scope of the issue.
Personally I'd like something like a security health card/metric on opensource libaries that we could tie into CI systems/pull requests or something
in the past there were so few libarries it wasn't as daunting
I'd be able reason about stuff like libpng, libttf ..etc and think about them or even support them but now some projects are massive hodgepodges of thousands upon thousnads of packages
That ("...private jets..") doesn't happen because the solution isn't exactly the hard part, and the unpaid original maintainers are doing them anyway.
In all seriousness, taking actions like I identified above would cost the companies virtually nothing but result in huge long-term benefits by signaling to the rest of the open source world that "we love your work and will be right beside you helping if the chips are down."
This is, of course, not a suitable compensation model for popular open source projects. Thats a separate conversation.
But it would at least be something.
Open source contracts essentially state that companies can use the product with no relevant obligations, so they do. The "huge long term benefit" you claim can't be reliably translated onto a balance sheet, so it isn't.
And when companies do get involved, it's often explicitly for their direct commercial benefit, like Amazon's ElasticSearch distribution.
There's also a race to the bottom aspect to it. If an open source package e.g. charges for commercial use, something more free is likely to replace it.
FSF: Pardon me, here's a little process that encourages sharing so that everyone can...
Corporations: gnawing on a giant proprietary turkey leg Doesn't Mesh Well nom nom nom No Standard Mechanism nom nom nom
**
Corporations: yelling through a mouthful of turkey What in sam hell is Linux why does IBM have a billboard about Linux why don't we have Linux?
Underling: Sir that's open source, our standard licensing mechanism wouldn't...
Corporations: belching through a mouthful of turkey change the goddamned mechanism I want linux and pass me that mayo
But they're following the process. What the comment I replied to was saying was that they should go beyond it. I'm pointing out that the lack of standard processes for doing that is what prevents it from happening.
That's just the reality of the situation. Yelling at clouds isn't going to change that.
I'm not sure you understand what you are asking, and I'm kind of dumbfounded by the sense of entitlement of your request. You are expecting others like me to be forced to work weekends on a problem that doesn't concern me (because my service is already patched) for absolutely nothing in return, and instead risking owning a problem and the blame of not coming up with a one-size-fits-all magic bullet.
All downsides and absolutely no upside at all, for my employer and let alone for myself.
Let me ask you this: how much of your personal time did you invested in coming up with a fix for this vulnerability? And yet you feel entitled to demanding this from others?
from: https://github.com/apache/logging-log4j2/pull/608#issuecomme...
If you read more of the history of the release you'll find that additional vectors were found in the RC process that were also fixed.
molehill meet mountain.
I poked some fun at the issue, because this is in many ways an amusing issue - a feature that would rarely be added in more recent times, but at the time it was introduced we as an industry considered the whole space differently. I'm sure the maintainers are having a tough time, and theres no need to point fingers at them, hell there's no need to point fingers at the implementor of the feature. It's a mistake in retrospect but that doesn't make anyone unworthy of respect.
The actions you identified make a lot of assumptions, some are exclusionary, some are potentially offensive: - not everyone lives in the same country - not everyone can travel on a whim - the developers don't need to be close to you or their customers unless they want to be. - the project is not in some dire state in need of "saving"
As an Apache project there is a foundation that can help organize funding, and so if there's a funding problem with the project the discussion should start there. Yes, open source at large is underfunded, but this isn't a standalone project on a personal git host. Apache has some (probably not enough) funding, but most importantly it has the industry contacts and relationships to do better here.
There's a problem with "doing better" though, which I rarely see come up in these conversations. There are lots of libraries, such as log4j, that don't necessarily need full time staff or full time funding. They need spurts of funding at key times, to handle the trickle of regular but infrequent patches, to roll releases periodically, and occasionally such as this to dedicate some significant time to handling an exceptionally rare event. What this requires, in my opinion, more than arbitrary dollars, is a slush fund of professional time sponsorship, for employers of key contributors to be ready to make space available during work hours for this work, without adding any more pressure to the situation. Depending on the situation this may or may not require additional funding, but for a healthy ecosystem finding ways to arrange for this, and helping employers be comfortable with it is the step necessary to address the wide scale problem of small to mid size projects burning people's personal time in unplanned ways.
> "Log4j maintainers have been working sleeplessly on mitigation measures; fixes, docs, CVE, replies to inquiries, etc. Yet nothing is stopping people to bash us, for work we aren't paid for..."
I guess that sounded to me like a very small group of isolated volunteers struggling to handle a lot of press and attention along with demands from numerous loud users. I thought perhaps they could have benefited from having some additional support.
My apologies if I unintentionally offended.
Quite the opposite.
The only way you can make this be less stressful for the maintainers is for them to delegate it all to some corporation to handle and go shopping, but this effectively means they are no longer the maintainers of that project.
Surely there is the possibility that these million dollars are just donated to the maintainers as compensation for a prompt resolution, but I don't think that is so easy to pull off. CTO may have some staffing but paying large sums to external contractors on a short notice is not something easy to pull off in any large organization.
No. I think this whole line of reasoning doesn't resonate much with reality.
Oddly enough, I hear the same thing from boss all the time.
And yes, I do think this, "but everyone else is doing the same thing so it isn't really our fault" attitude is a problem.
Google doesn't even use log4j. What are you talking about? The spirit of open source does not dictate that the richest companies automatically shoulder the burden of maintenance of projects they do not even use. Google already has initiatives like Summer of Code that help open source projects it does not use, and I think it's perfectly fine to draw the line there.
> divert the best engineers they had
So the lessons from the mythical man-month are forgotten here. At this point I don't think adding more manpower helps.
Also, the reason the maintainers are rushing to fix it is: they're worried about losing "market share". Having been in open-source circles for a long time, maintainers care GREATLY about how many users they have. They just like watching their download stats go up every year. Even if it beings them no financial rewards. It's a sort of addiction.
They do. Until they don't.
That inevitable day when they get yelled at in a github issue thread by a user who didn't bother reading the documentation, while staring at their kid in the living room playing video games and start wondering to themselves "why am I doing this hobby in my spare time again?"
Mild dopamine hits to affirmation-addicted programmers is not the sturdiest foundation upon which to build enterprise-grade software libraries.
Many serious contributors eye all these aveneues to leverage their popularity, it is just not affirmation, it is also ability to monetize it
Anything sufficiently at scale needs a set of maintainers that the commercial tech companies would then collaborate with to get the PRs going.
Otherwise if everyone's just panicking and rushing to submit PRs, they'll inundate the maintainer. There's also no guarantee that even the best engineers at these companies are intimately familiar with the project, and might introduce regressions or other vulnerabilities in the process.
Anyway I do agree companies should be working with OSS devs, but it shouldn't be rushed or knee jerk. It should be collaborative and measured.
For the general case, the problem is that a reporter might report the vulnerability to the open source project, then the project needs to keep it a secret while they make a fix. There isn't a great way to leverage these stakeholders. It's obviously different for something like Android that is open source, but clearly Google.
Drive-by pull requests during a highly visibile emergency are rarely useful.
Maybe it makes more sense to fund system-wide efforts?
You'd think, in the spirit of open source, these multi-billion dollar companies--like Apple and Google and Amazon--would recognize the danger and immediately divert the best engineers they had to help this team identify and mitigate the problems. They should have been buried in useful pull requests.
For that matter, they should have really picked them all up in private jets and flown them to neutral working space with those engineers for a one or two week hackathon/code sprint to clean up the outstanding issues and set the project on a sustainable path. To get those maintainers there they should offer a six figure consulting fee and negotiate with their current employers to secure their temporary help.
I can't believe these folks just get abandoned like this while CEOs/CTOs from rich companies wring their hands wailing about the problems and not offering solutions.