adminprof

I think you're missing two fatal problems in this "publish all raw data and code" mindset. I don't think the desire of commercialization is high on the list of fatal problems preventing people from publishing data+software.

1) How do you handle research in domains where the data is about people, so that releasing it harms their privacy? Healthcare, web activity, finances. Sure you can try to anonymize it, anonymization is imperfect, and even fully anonymized data can be joined to other data sources to de-identify people; k-anonymity only works in a closed ecosystem. If we live in a world where search engine companies don't publish their research because of this constraint, that seems worse than the current system.

2) How does one define "re-runnable processing"? Software rots, dependencies disappear, operating systems become incompatible with software, permission models change. Does every researcher now need a docker expert to publish? Who verifies that something is re-runnable, and how are they paid for it?

It seems like it's just that the goals are different. Computer science researchers could also make the opposite case, "The underwhelming impact of software development on producing state-of-the-art computer science discoveries."

I think your comment is pretty spot on. Just three things I'd add to, are that 1) the advisor is not intentionally downgrading your letter to "just good", but it's that they're obligated to write a better than "just good" letter for someone who has been more productive both in terms of research and in being a leader in the community (these two things go hand in hand). Writing the same letter for two students regardless of what they accomplish would be unfair.

And 2) sometimes getting more funding is simply not possible, as in the advisor has basically reached the limit of what they can do. There's a limit to the number of proposals that one can submit and the number of calls that fit their research agenda. So what I think you're missing is that if an advisor has less funding, there's going to be more pressure to finish sooner and less freedom to explore ideas beyond what's written in a previous grant proposal.

3) I've never heard of a tenured professor that concerned about their publication rate. In fact, most of them don't even update their CVs or websites with the last few years of papers. It's always the student who is trying to get more papers.

Students are users of a class, as much as a dentist is a user of a dental software application. Neither were involved in the making of the class or software. Both were delivered the experience as designed by the course staff (instructor and/or TAs), or software team. Both only see the final product after months or years of development, done by specialists trained for many years even before that.

The error you're making is stating that "students are involved in the entire process" which is laughable. Many classes have gone through years of iteration, and even new courses take many months to develop before students set foot in the classroom, not to mention the years of experience and education needed to get the instructor to the point that they can even make a class in several months.

I'd like to see in a forum for dentists a thread where they offer clever suggestions for how to change software development. Clearly, all these dentists have used plenty of software before.

They would have seemingly great ideas like "programmers should get 10% of their pay docked each time I encounter a bug in a program" or "the real solution is to hire someone to test the program from start to finish before releasing it".

I bet programmers on Hacker News would be livid upon hearing these suggestions, but seem to have no problem announcing their clever solutions about other disciplines (not excluding myself).

Simply, because using absolute points would lead to debate over every single decimal of a point. A student with 89.95% would be determined to fight to get 90.05% and then continue to fight to get 90.15% so it only makes the problem worse.

This doesn't work when I've tried it. How many students or times have you implemented this policy? First, it doesn't make sense when the regrade is most objective (like points were calculated wrong, or the grader didn't see something that the student wrote). And if you say that it doesn't apply for straightforward grading mistakes, then you get emails asking you whether something is a grading mistake or has the chance of lowering a grade.

And I've tried this policy before, and got students who wrote in my course evals something like "the professor intentionally tries to scare students from asking for regrades by threatening to lower their grade even more." And then what about when you are still asked for a regrade (which in my experience was not zero, but maybe about a third or half as much as without this policy). In those cases, you end up doing way way more, so the level of effort actually increases.

> Please don't use such circular logic. We're debating whether the research properly qualifies as human subject research; we're not debating about what the IRB actually decided on that question, because they may have gotten it wrong.

Yes that's what we're debating. But you used the word "exemption" which has a specific technical meaning in human subjects research, and I'm saying that it's not an exemption. There are 8 tests for exemption, and I'm pointing out that this is not an IRB exemption.

> The policy itself is certainly the intended subject of the research. But the methods they've chosen mean they are also collecting and analyzing information about the responses of real live humans to their interactions and interventions, and that qualifies this as human subject research irrespective of the naive intentions of the researchers. Having a non-human subject does not preclude also having a human subject.

Do you have a source for this interpretation? It sounds like this is your interpretation, but not the federal one. Following your interpretation, surveys of companies (e.g. emailing [email protected] to ask how many employees they have) would fall under the definition of human subjects.

Thanks for the continued conversation, but I think this is my last comment. Nothing personal, but this is a bit exhausting. It seems like you're debating two other people on this forum about this exact definition, and you might consider that maybe you're just wrong about your interpretation?

Here's one final source, if it helps provide closure:

To meet the definition of human subjects, you must ask “about whom” questions. Questions about your respondents' attitudes, opinions, preferences, behavior, experiences, or characteristics, are all considered “about whom” questions. Questions about an organization, a policy, or a process are “about what” questions.

https://campusirb.duke.edu/resources/guides/defining-researc...

Not at all,

1) this is clearly not an exempt study, which is a category of its own that the IRB reviews and makes a judgment on. The authors would immediately have been able to point out the protocol number of the exempt study if it were exempt. Rather it's not considered human subjects as the authors clearly state on their FAQ.

2) it seems like you're thrown off by the example, because if you ended your sentence as "The bit you've quoted is intended to clarify that "about whom" means the subject is the patient" then we would be in agreement, and it'd be more obvious that the subject is, in fact, the website's policies/procedures. Here's an excerpt from the written text of the common rule,

"“About whom” – a human subject research project requires the data received from the living individual to be about the person."

https://hso.research.uiowa.edu/defining-human-subjects

He seems to have good intentions, but does not seem to have knowledge of IRB which may make this situation worse.

Specifically, he confuses "does not constitute human subjects research" with "exemption" which is a pretty big difference and anyone who works with human subjects should know this.

From his Twitter thread, "Update: They are now saying they have an exemption. They have not made any forms available or explained the lack of informed consent."

Exemptions are protocols that have been reviewed, and deemed exempt based on one of 8 very specific criteria. Studies deemed not constituting human subjects research are returned by the IRB, and not considered reviewed.

Given that the authors actually said "...to the Princeton University Institutional Review Board, which determined that our study does not constitute human subjects research" this is clearly NOT an exemption, and informed consent is not a consideration as far as the IRB is concerned.

This is actually the most technically correct answer on this page. Everyone is going by their own opinions about definitions of what constitutes human subjects research, rather than starting from the primary sources. IRB guidelines are dictated by the federal government "common rule", a common standard adopted by all institutions that receive federal funding.

"about whom" is a key criteria from the federal government to determine whether something fits the definition of human subjects research. Here's a quote from HHS:

"The phrase ‘about whom’ is important. A human subject is the person that the information is about, not necessarily the person providing the information. In the case of biospecimens, the human subject is the person from whom the specimen was taken."

https://www.hhs.gov/ohrp/sites/default/files/OHRP-HHS-Learni...

Reading that, it's clear that the Princeton study does not fit the definition of human subjects research. The complainants may be able to sue for damages to the university, but not because the study was improperly classified as human subjects.

Sure, as someone who has also been on both sides, that's not what I was describing at all though. None of the examples I gave are handled by an editor, except maybe "managing issues dealing with misconduct" depending on the situation, but maybe by the publisher's legal team. But the other issues are not handled by an editor in my experience.

So I think your assumption that academic publishing is "a website hosting up PDFs" is the source of confusion for "what justifies this much money." Just like AirBnB isn't just a website hosting JPEGs, while being worth 129 billion dollars. Hosting PDFs can already be done by arXiv or Google Drive or Github as you said.

Customer support is for peer reviewers who can't log into their account, for managing issues dealing with misconduct, for handling issues with payments, for post-publishing corrections and errata, for passing accounts from editors who become non-responsive to other editors, etc. Not just dealing with readers or subscribers.

This comes up every time there's an article about academic publishing. Yes peer reviewers do the reviewing, but it's the long-term infrastructure and coordination that the journal provides.

AirBnB's content is generated by users, but AirBnB itself requires software development, legal, customer support, HR, program managers, quality control, etc. Same with publishers.

Note that this journal now has a publishing fee for authors to cover these costs, rather than a fee for the reader as before. The 2022 fee for each author is $1,705 according to the FAQ. So moving to open access it not about removing the costs (which many people on Hacker News seems to always assume), but changing who pays for it.

I would like to subscribe to your newsletter.

Agree with all your points. The author has been in academia all his life so after 20 years or so, I can see how he can feel frustrated by it.

I think the general summary of what you're pointing out is that in industry it's easy to do things that the company wants to do (make money, make your boss look good, make life easier for the execs), and hard to do things that you want to do (cool stuff, disseminate ideas, and explore).

When you start at a company, you're optimistic and campaign to do a balance of both, but over time it wears you out so almost every defaults to doing the company work and fades out of public view. And yes, MSR is generally the exception but even then notice how many researchers there now do something related to "optimizing productivity".

Do these status dashboards have any more credibility left? Always showing green when services are down.

I think I see the confusion between us. You don't see the difference between selling data to a company, and selling ad space where the advertiser can choose for what demographics it shows up for.

Let me try to make it more clear. Do you see the difference between "hey Chase Bank, do you want to buy this file containing data about grey-area's interests, age, political stance, credit score, purchasing habits, etc." versus "hey Chase Bank, do you want to put an ad on my website that is only shown to people with credit scores above 600 and are interested in savings accounts"?

If they both seem the same to you, then I don't think your perspective is one that a reasonable person would take. If you do see the difference, then Facebook is doing the latter, but the word "selling data" conjures the former, which you do recognize as a different matter.

I'm going to ignore the nonsensical definition you gave of selling data = selling movies, being the only other definition of selling you could imagine, in hopes that it was just an oversight.

They actually don't, unless you define selling as they allow advertisers to select what demographics/attributes their ads target. But the actual data stays on the Facebook servers. If you're referring to the apps having access to user data, that was not selling at all, but instead a permission originally granted by users by probably forgotten about. Basically, unless you contort the definition of selling to a very different meaning, that's simply not true.

And if you do use that definition of selling, then everyone is selling your data. All the politicians who decry tech companies are selling your data using the same definition. Every advertiser, retail store, bank, basically every large business offers other businesses a way to access a specific subset of their users.

Yet everyone is freaking out and moralizing about nonfinancial data voluntarily given to Facebook. If only the credit bureaus kept our financial and identity data as Facebook kept your list of favorite movies and your selfies.

You can't think of any reason? What about retaliation for being the less enthusiastic reviewer? What about the opposite, that you become known for being an easy reviewer who missed some obvious flaws?

Please do. Would love to hear success stories for data security.

Sure I guess I didn't make that very clear. Just going down the line:

1) stop releasing before the bugs are fixed. Software and games are rushed out. Companies need to take their time to fix the bugs so the users don't have to encounter them.

- This is a tradeoff between shipping time and bugs. You will never fix all bugs, so it's unrealistic. And it's a business decision in many cases. Even the definition of a bug is tricky, like is a usability issue a bug? So it's just a naive idea.

2) no more technical debt. Programmers are sloppy and introduce technical debt because they are not incentivized to do high quality programs.

- No one wants to create technical debt. Obviously it slows down development later on. Again this could be a business decision. Some programs like one-off data science scripts don't need to fix all their technical debt. Technical debt also accrues naturally (like just changing environment, platform, standards) so it's not possible to aim to not have debt in the very beginning. Hindsight is 20-20 and all that. Saying programmers are not incentivized to do high quality programs is just a blanket naive statement, and depends on the definition of high quality programs.

3) cap team sizes. Everyone knows that large teams fail more spectacularly. Gmail and Napster and the original version of Google were made by a group of 4 people. Software teams need to be 4-5 people max.

- Depends on the type of software. Can't just generalize given a few token examples. Expectations also change over the course of the product.

4) programmers must use a transparency scorecard. Software companies like Oracle and IBM charge ridiculous amounts for their work. They hide costs and cut corners. Programmers should be transparent about the work they are doing each day, what data they access, and which functions they are writing.

- This uses one subsection of the software economy to make a point (as a fallacy). But also some of these measures don't make sense, like some programmers read a lot of code or delete lines, and so the metrics are not generalizable.

In summary, these are ideas that someone who has not done long-term software development would say, or someone who has only had experience with one type of software or company would say. They're not well defined, not generalizable, and don't account for the complex and varied sociotechnical process that software development is.

It'd be a process: participant in a community in a subfield, listen to ideas that have been tried and outcomes, come up with an idea using those lessons learned and success metrics, build consensus around a new idea, test in a single-instance (like one conference, grant review panel, tenure committee at one university), share lessons to the field, do this for a couple of years to show clear success, expand to multiple events in the field, become an exemplar field for that idea and "infect" other fields

Sounds slow but there's thousands of such experiments happening simultaneously right now. This is how a long of major field-sized changes have happened, like the transition to conferences from journals (which had many initial problems like during tenure review or a lack of quality in reviews), etc. Ideas will lose traction at various stages (for example, there was a movement some time ago to use alpha=0.001 instead of alpha=0.05 for null hypothesis testing, which has been limited to that field or subfield).

Yeah I have no idea how it got to -2, but maybe it was my tone. I was even trying to not call out any specific post or debate any single issue.

The thing about academia is it's full of people who love talking about ways to make it better, and rotate into positions of power where they can change things after a few years.

The solutions are complex, require convincing many different stakeholders (even if they're amenable to the change), nailing lots of detail to make it work right. Because peoples lives and careers are on the line. Reputations of entire fields, the way medical discoveries happen, billions of dollars of taxpayer money, major institutions, etc. are not things you want to hack and discover that whoops, you just incentivized the wrong thing and set back cancer research for a decade.

This entire thread is really frustrating for me to read. A lot of these ideas are one-liners from people who have had very little experience with different parts of academia, except maybe being a student. Every single idea here are debated to death by academics already (not just university academics, but funding agencies, academic societies, award panels, journal editors and conference organizers), with changes happening all the time.

If it was this simple that a random person here could come up with how to "solve" academia, we'd have already done it decades ago. The ideas also lack nuance and when you get into the definitions of things (for example, p-hacking), then things become a lot more grey; are you allowed to look at a dataset that you spent 2 years collecting if your first hypothesis does not pan out? The clear cut cases are obvious to everyone, it's the grey area that takes 99% of the time to figure out.

Imagine reading a thread where everyone is proposing "solutions" to software development. It'd go something like "software development is a cesspool and 80% of it fails (see voting systems, MySpace, electronic health records, Theranos. Here's what software companies need to do:" [yes I'm being intentionally stupid to demonstrate how annoying this is]

1) stop releasing before the bugs are fixed. Software and games are rushed out. Companies need to take their time to fix the bugs so the users don't have to encounter them.

2) no more technical debt. Programmers are sloppy and introduce technical debt because they are not incentivized to do high quality programs. [yes, see how triggering that is]

3) cap team sizes. Everyone knows that large teams fail more spectacularly. Gmail and Napster and the original version of Google were made by a group of 4 people. Software teams need to be 4-5 people max.

4) programmers must use a transparency scorecard. Software companies like Oracle and IBM charge ridiculous amounts for their work. They hide costs and cut corners. Programmers should be transparent about the work they are doing each day, what data they access, and which functions they are writing.

These changes need to happen. derp derp

You make the assumption here that this hack halted all research. It's unlikely to have even affected 10% of ongoing research. Plenty of things stored on cloud storage, in email, in personal computers, on computers not locked out, etc.

I think we just keep going back to the same issue -- people agree that administration is too big. But if you break it down to actual positions or administrative units, it's not clear there is any agreement. There is no such thing as "right cuts" because that implies it's clear which parts of administration should be cut.

Same with government. Government waste is bad? Of course! Trim unnecessary government spending and pork barrel? Yes!! And which branches and positions should we cut? The Right Ones!! [cue nothing actually happens]

I don't see what assurances you have of this guarantee you give. As so far it seems like when funding is cut, faculty (teaching and research) and academic departments are cut and financial aid is reduced, rather than major cuts in administration. So it's worth discussing what administrative units are cuttable, or come to the realization there is no free cuts to make to administration without sacrifices to some existing principles.