https://raw.githubusercontent.com/QiushiWu/qiushiwu.github.i...
> We send the emails to the Linux communityand seek their feedback. The experiment is not to blame any maintainers but to reveal issues in the process. The IRB of University of Minnesota reviewed the procedures of the experiment and determined that this is not human research. We obtained a formal IRB-exempt letter. The experiment will not collect any personal data, individual behaviors, or personal opinions. It is limited to studying the patching process OSS communities follow, instead of individuals.
I'm not sure how it affects things, but I think it's important to clarify that they did not obtain the IRB-exempt letter in advance of doing the research, but after the ethically questionable actions had already been taken:
The IRB of UMN reviewed the study and determined that this is not human research (a formal IRB exempt letter was obtained). Throughout the study, we honestly did not think this is human research, so we did not apply for an IRB approval in the beginning. ... We would like to thank the people who suggested us to talk to IRB after seeing the paper abstract.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
That's not really what they did.
They sent the patches, the patches where either merged or rejected.
And they never let anybody knew that they had introduced security vulnerabilities on the kernel on purpose until they got caught and people started reverting all the patches from their university and banned the whole university.
https://www-users.cs.umn.edu/~kjlu/papers/clarifications-hc....
> (4). Once any maintainer of the community responds to the email, indicating “looks good”, we immediately point out the introduced bug and request them to not go ahead to apply the patch. At the same time, we point out the correct fixing of the bug and provide our proper patch. In all the three cases, maintainers explicitly acknowledged and confirmed to not move forward with the incorrect patches. This way, we ensure that the incorrect patches will not be adopted or committed into the Git tree of Linux.
Seems like there are some patches already on stable trees [1], so they're either lying, or they didn't care if those "don't merge" messages made anybody react to them.
1 - https://lore.kernel.org/linux-nfs/CADVatmNgU7t-Co84tSS6VW=3N...
The main culprit seems to be only Quishu Wu. He is also the one who wrote the paper.
>> The work taints the relationship between academia and industry
> We are very sorry to hear this concern. This is really not what we expected, and we strongly believe it is caused by misunderstandings
Yeah, misunderstandings by the university that anyone, ever, in any line of endeavor would be happy to be purposely fucked with as long as the perpetrator eventually claims it's for a good cause. In this case the cause isn't even good, they're proving the jaw-droppingly obvious.
..."Because if we're lucky tomorrow, we won't have to deal with questions like yours ever again." --Firesign Theater, "I Think We're All Bozos on the Bus"
Yet we do nothing about it? I wouldn't call that jaw-droppingly obvious, if anything, without this, I'm pretty sure that anyone would argue that it would be caught way before making it way into stable.
What they do almost universally lack is enough people making positive contributions (in time, money, or both).
This "research" falls squarely into the former category and burns resources that could have been spent on the latter.
Yes, that's the whole point! The real malicious actors aren't going to notify anyone that they're injecting vulnerabilities either. They may be plants at reputable companies, and they'll make it look like an "honest mistake".
Had this not been caught, it would've exposed a major flaw in the process.
> ...until they got caught and people started reverting all the patches from their university and banned the whole university.
Either these patches are valid fixes, in which case they should remain, or they are intentional vulnerabilities, in which case they should've already been reviewed and rejected.
Reverting and reviewing them "at a later date" just makes me question the process. If they haven't been reviewed properly yet, it's better to do it now instead of messing around with reverts.
While true, it's simply not acceptable to abuse trust in this way. It causes real emotional harm to real humans, and while it also may produce some benefits, those do not outweigh the harms. Just because malicious actors don't care about the harms shouldn't mean that ethical people shouldn't either.
Given the completely unavoidable limitations of the review and bug testing process, a maintainer has to react very differently when they have determined that a patch is malicious - all previous patches past from that same source (person or even organization) have to be either re-reviewed at a much higher standard or reverted indiscriminately; and any future patches have to be rejected outright.
This puts a heavy burden on a maintainer, so intentionally creating this type of burden is a malicious action regardless of intent. Especially given that the intent was useless in the first place - everyone knows that patches can introduce vulnerabilities, either maliciously or by accident.
The vast majority of drunk drivers never kill anyone.
> Sending a malicious patch (one that is known to introduce a vulnerability) is a malicious action.
I disagree that it's malicious in this context, but that's irrelevant really. If the patch gets through, then that proves one of the most critical pieces of software could relatively easily be infiltrated by a malicious actor, which means the review process is broken. That's what we're trying to figure out here, and there's no better way to do it than replicate the same conditions under which such patches would ordinarily be reviewed.
> Especially given that the intent was useless in the first place - everyone knows that patches can introduce vulnerabilities, either maliciously or by accident.
Yes, everyone knows that patches can introduce vulnerabilities if they are not found. We want to know whether they are found! If they are not found, we need to figure out how they slipped by and how to prevent that from happening in the future.
That is a complete misunderstanding of the Linux dev process. No one expects the first reviewer of a patch (the person that the researchers were experimenting on) to catch any bug. The dev process has many safeguards - several reviewers, testing, static analysis tools, security research, distribution testing, beta testers, early adopters - that are expected to catch bugs in the kernel at various stages.
Trying to deceive early reviewers into accepting malicious patches for research purposes is both useless research and hurtful to the developers.
But the linux kernel is NOT a security product - it is a kernel. It can be used in entirely disconnected devices that couldn't care less about security, as well as in highly secure infrastructure that powers the world. The ultimate responsibility of delivering a secure product based on Linux lies with the people delivering a secure product based on the kernel. The kernel is a essentially library, not a product. If someone is assuming they can build a secure product by trusting Linux to be "secure" than they are simply wrong, and no amount of change in the Linux dev process will fix their assumption.
Of course, you want the kernel to be as secure as possible, but you also want many other things from the kernel as well (it should be featureful, it should be backwards compatible with userspace, it should run on as many architectures as needed, it should be fast and efficient, it should be easy to read and maintain etc).
Well, in real life, you can't go punch someone in the face to teach them a "point". If you do so, you'll get punished.
> Reverting and reviewing them "at a later date" just makes me question the process.
I don't think anybody realistically thought that the kernel review process is rock solid against malicious anyway. What exactly does the paper expose?
This just turns the researchers into black hats. They are just making it look like "a research paper."
How is this not human research? They experimented on the reactions of people in a non-controlled environment.
It doesn’t just “involve humans” it is first and foremost the behavior of specific humans.
> but the humans (reviewers) are not the research subject.
The study is exactly studying their behavior in a particular context. They are absolutely the subjects.
This study does not care about the reviewers, it cares about the process. For example, you can certainly improve the process without replacing any reviewers. It is just blatantly false to claim the process is all about humans.
Another example, the review process can even be totally conducted by AIs. See? The process is not all about humans, or human behavior.
To make this even more understandable, considering the process of building a LEGO, you need human to build a LEGO, but you can examine the process of building the LEGO without examine the humans who build the LEGO.
This was all about the reaction of humans. They sent in text with a deceptive description and tried to get a positive answer even though the text was not wholly what was described. It was a psych study in an uncontrolled environment with people who did not know they were participating in a study.
How they thought this was acceptable with their own institutions Participant's Bill of Rights https://research.umn.edu/units/hrpp/research-participants/pa... is a true mystery.
"Process" in this case is just another word for people because ultimately, the process being evaluated here is the human interaction with the malicious code being submitted.
Put another way, let's just take out the human reviewer, pretend the maintainers didn't exist. Does the patch get reviewed? No. Does the patch get merged into a stable branch? No. Does the patch get evaluated at all? No. The whole research paper breaks down and becomes worthless if you remove the human factor. The human reviewer is _necessary_ for this research, so this research should be deemed as having human participants.
How was this study conducted? For every patch that the researchers sent, what process did it go through?
The answer is, it was reviewed and accepted by a human. That's it. Full stop. There's your human subject right there in the middle of your research work. It's not possible to conduct this research without that human subject interacting with your research materials. You do not get to discount that human participation because "Oh well we COULD replace them with an AI in the future". Well your study didn't, which means it needs to go through the human subjects review process.
When you claim that this study was about a process, you're literally taking the researchers side. That's what they've been insisting on as the reason why this study is ethical and they did not need to inform or obtain consent from the kernel development team. That's the excuse they used to get out of IRB's review process so they can be considered "not a human subjects research". That's the excuse they needed so they can proceed without having to get a signed consent form. They did all of this so they could conduct a penetration test without the organization they were attacking knowing about it.
You don't seem to be able to comprehend why or how the maintainers feel deceived here, or that their feelings are legitimate. If you did, you wouldn't keep banging on about "oh this is just a process study, the people don't matter, it's all isolated from humans". Funny enough, the people who DID interact with this research DID feel they mattered and DID feel deceived. The whole point of IRB was to prevent exactly this; researchers conducting unethical research which would only come to light after the study concluded and the injured parties complained (and deceit IS a form of harm). For research which is supposed to be isolated from humans and thus didn't see the need in obtaining a signed consent form, that's not really the outcome you expect to see if everything was on the up and up. Another form of harm from this study, the maintainers now have to go over everything they submitted again to ensure there's nothing else to be worried about. That's a lot of wasted man hours and definitely constitutes harm as well. All of University of Minnesota now has less access to the project after getting banned, even more collateral damage and harm caused to their own institution.
Let's be honest. If the researchers were able to sneak their code into a stable, or distribution version of the kernel, they'd be praising themselves to high heaven. Look at how significant our results were, we fucked up all of Linux! Only reason they didn't is because at least they can recognize that would be going a step too far. They're just looking for excuses to not get punished at this point. Same with the IRB. The IRB is now trying to wiggle out of the situation by insisting everything is ok. The IRB is also made up of professors who have a reputation to maintain! They know they let something through that should never have been approved in it's current form. Most human subject research NEVER get this kind of blowback and the fact that this one did means they screwed up and they know it.
No ethics review board considers a multi page, multi forum, lengthy discussion on the ethics of a study they approved as a good sign. Honestly, any study that gets this much attention would be considered a huge success in any other situation.
Thats not the correct or relevant criteria. If you were correct, testing airport security and testing AntiMoney Laundering checks at a bank would amount to human experiments. In fact its hard to think of any study of the real world that would not become a "human experiment".
"When you claim that this study was about a process, you're literally taking the researchers side."
Thats some seriously screwed up logic right here.
"Weinstein was a Nazi and a serial killer, if you disagree with me you are taking his side"
It's easy to think of studies that don't involve humans so that statement is just wilful obfuscation. Physics, chemistry, heck lots of biology, and of course computer science are primarily made of studies on objects rather than people. Of those that are done on people they are almost always done on people who know they are the subject of an experiment. Very few studies are like this one.
Studies of airport security are done all the time, thats how we know its terribly ineffective. The staff of the airport are not told about them, they are not human experiments.
The experiments on people have a spesific definition that goes beyond "a human is present"
Perhaps a similar approach that allows randomness with some sort of agreement with the maintainers could have prevented this issue while preserving the integrity of the study.
It is self-evident that this study tangibly involved people in the scope, those people did not provide consent prior, and now openly state their grievances. It is nothing short of arguing in bad faith to claim otherwise.
Maybe the stated aim of the research was to study the process. But what they actually did was study how the people involved implemented it.
Being publicly manipulated into merging buggy patches, and wasting hours of people's time are two pretty obvious effects this study had that could cause some amount of distress and thus it cannot be dismissed as simply "studying the process".
It is likely the professor involved here will be fired if they are pre-tenure, or sanctioned if post-tensure.
Of course, there are other ethical and legal requirements that you're bound to, not just this one. I'm not sure which requirements IRBs in the US look into though, it's a pretty murky situation.
It seems to qualify per §46.102(e)(1)(i) ("Human subject means a living individual about whom an investigator [..] conducting research: (i) Obtains information [...] through [...] interaction with the individual, and uses, studies, or analyzes the information [...]")
I don't think it'd qualify for any of the exemptions in 46.104(d): 1 requires an educational setting, 2 requires standard tests, 3 requires pre-consent and interactions must be "benign", 4 is only about the use of PII with no interactions, 5 is only about public programs, 6 is only about food, 7 is about storing PII and not applicable and 8 requires "broad" pre-consent and documentation of a waiver.
It's not worth arguing about this; if you care, you can try to change the law. In the meantime, IRBs will do what IRBs do.
Since IRBs exist to minimize liability, it seems like that would be that fastest route towards change (assuming you have legal standing )
Frankly universities and academics need to be taken to court far more often. Our society routinely turns a blind eye to all sorts of fraudulent and unethical practices inside academia and it has to stop.
I had a look at section §46.104 https://www.hhs.gov/ohrp/regulations-and-policy/regulations/... since it mentioned exemptions, and at (d) (3) inside that. It still doesn't apply: there's no agreement to participate, it's not benign, it's not anonymous.
IRBs are like the TSA. Imposing annoyance and red tape on the honest vast-majority while failing to actually filter the 0.0001% of things they ostensibly exist to filter.
I'm guessing it passed for similar reasoning, along with the reviewers being unfamiliar with how "vulnerabilities are injected." To get the bad code in, the researcher needed to have the code reviewed by a human.
So if you rephrase "inject vulnerability" as "sneak my way past a human checkpoint", you might have a better idea of what they were actually doing, and might be better equipped to judge its ethical merit -- and if it qualifies as research on human subjects.
To my thinking, it is quite clearly human experimentation, even if the subject is the process rather than a human individual. Ultimately, the process must be performed by a human, and it doesn't make sense to me that you would distinguish between the two.
And the maintainers themselves express feeling that they were the subject of the research, so there's that.
What makes people revieing linux kernel more 'human' than any of the above?
Main point is that IRBs were created in response to some highly unethical and harmful "studies" being carried out by institutions thought of as top-tier. Now they are considered to be a mandatory part of carrying out ethical research. But if you think about it, isn't outsourcing all sense of ethics to an organization external to the actual researchers kind of the opposite of what we want to do?
All institutions tend to be corruptible. Many tend to respond to their actual incentives rather than high-minded statements about what they're supposed to be about. Seems to me that promoting the attitude of "well an IRB approved it, so it must be all right, let's go!" is the exact opposite of what we really want.
All things considered, it's probably better to have something there than nothing. But you still have to be responsible for your own decisions. I bamboozled our lazy IRB into approving our study, so I'm not responsible for it being obviously a bad idea, just isn't good enough.
If you think about it, it's actually kind of meta to the code review process they were "studying". Just like IRBs, Code review is good, but no code review process will ever be good enough to stop every malicious actor every time. It will always be necessary to track the reputation of contributors and be able to mass-revert contributions from contributors later determined to be actively malicious.
Eventually, the IRB, unhappy at his behavior, said he couldn't do the experiment. He left for another institution (UC San Diego) immediately, having made a deal with the new dean to go through expedited review. It was a big loss for Boulder and TBH, the IRB's reasoning was not sound.
This research had the potential to cause harm to people despite not being human research and was therefore ethically questionable at best. Because they presented the research as not posing potential harm to real people that means they lied to the IRB, which is grounds for dismissal and potential discreditation of all participants (their post-graduate degrees could be revoked by their original school or simply treated as invalid by the educational community at large). Discreditation is unlikely, but loss of tenure for something like this is not out of the question, which would effectively end the professor's career anyway.
I don't buy it, and you fail to back that claim up at all.
As to the backdated review now being undertaken, as far as I'm concerned that decision is squarely on the maintainers. (Honestly it comes across as an emotional outburst to me.)
Nor is breaking my computer monitor.
Theft was meant as an example of a type of harm, not a complete list of all types of harm.
Something doesn’t have to be illegal to be harmful.
Sure, but I'm still going to be pretty annoyed with you. And if you've wasted my time by messing with a system or process under my control then I'm probably going to block you from that system or process.
As a really prosaic example, I've blocked dozens - if not hundreds - of recruiter email addresses on my work email account.
It seems very possible to me that an IRB wouldn't have accepted their proposed methodology if they hadn't received an exemption.
Is there anyone on hand who could explain how what looks very much like a social engineering attack is not "human research"?
First of all, this is completely irresponsible, what if the patches would've made their way into a real-life device? The paper does mention a process through which they tried to ensure that doesn't happen, but it's pretty finicky. It's one missed email or one bad timezone mismatch away from releasing the kraken.
Then playing the slander victim card is outright stupid, it hurts the credibility of actual victims.
The mandate of IRBs in the US is pretty weird but the debate about whether this was "human subject research" or not is silly, there are many other ethical and legal requirements to academic research besides Title 45.
Right. It's not just human subjects research. IRBs vet all kinds of research: polling, surveys, animal subjects research, genetics/embryo research (potentially even if not human/mammal), anything which could be remotely interpreted as ethically marginal.
It's a real shame because the university probably has good, experienced people who could contribute to various OSS projects. But how can you trust any of them when the next guy might also be running an IRB exempt security study.
I don't think code commits to the Linux kernel make it to live systems that fast?
I do agree with the sentiment, though. It's grossly irresponsible to do that without asking at least someone in the kernel developer's group. People don't dig being used as lab rats, and now the whole uni is blocked. Well, tough shit.
That'd be great, yup. And the linux kernel team should then strongly consider undoing the blanket ban, but not until this investigation occurs.
Interestingly, if all that happens, that _would_ be an intriguing data point in research on how FOSS teams deal with malicious intent, heh.
I think the real problem is rooted more fundamentally in academia than it seems. And I think it has mostly to do with a lack of ethics!
We presented students with an education protocol designed to make a blind subset of them fail tests. Then measured if they failed the test to see if they independently learned the true meaning of the information.
Under any sane IRB you would need consent of the students. This is failure on so many levels.
(edit to fix typo)
I hope we hear from the IRB in about a year stating exactly what happened. Real investigations of bad conduct should take time to complete correctly and I want them to do their job correctly so I'll give them that time. (there is the possibility that these are good faith patches and someone in the linux community just hates this person - seems unlikely but until a proper independent investigation is done I'll leave that open.)