"These results are from respected investigators, so it's inappropriate to question them."
FWIW, I wasn't really upset by that feedback. Senior spring; I don't care!
[1] http://static1.1.sqspcdn.com/static/f/1605966/27061956/14652...
> September 21: The Self (Prentice) > September 28: Social Cognition (Fiske) > October 5: Attitudes (Cooper) > October 19: Social Inference (Pronin) > November 2: Prejudice & Stigma (Shelton) > November 16: Dissonance (Cooper) > November 30: Close Relationships (Fiske) > December 7: Social Influence (Prentice)
It's quite okay. That's exactly the kind of attitude that's being scrutinized here ;)
Thanks.
The modern synthesis of contrasting styles of thought in “east” and “west” *Nisbett, R. E., Masuda, T. (2003). Culture and point of view. Proceedings of the National Academy of Sciences, 100, September 16, 2003, 11163-11170
I was like, "Um, the whole point of science is that it's knowledge in the public record. If you need to 'chill' with the original researcher to get critical details, that's not science anymore."
Remember, the studies might form the basis of an FDA approval. If only the original lab can produce the result, then is the FDA approval of the treatment going to be limited to only that lab as well? Kinda limits scalability...
I did a search and found the blog post that quoted the researcher [1] with this scary attitude:
>When researchers at Amgen, a pharmaceutical company in Thousand Oaks, California, failed to replicate many important studies in preclinical cancer research, they tried to contact the authors and exchange materials. They could confirm only 11% of the papers. I think that if more biotech companies had the patience to send someone to the original labs, perhaps the percentage of reproducibility would be much higher.
[1] http://andrewgelman.com/2013/12/17/replication-backlash/ (thanks Google)
Certainly there is always the possibility that there was some details misunderstood, something that needs to be clarified, a print error, etc. Your "that's not science anymore" statement seems highly exaggerated. People are not supposed to communicate only via papers.
But there's a difference between that, vs "we expect that someone will have to make personal contact with the original researchers in order to replicate it".
If you're explaining away replication failures by such non-contact (as the quote is), that's confirmation of a problem (in keeping with the standards of science), not a vindication of the results.
There's an additional danger of making it so that "you can't replicate until you have social contact with the original researchers". That way lies favoritism: it's harder to criticize someone as you get closer to them socially, and they can withhold the capability of criticism by not engaging the critics.
tl;dr for this wall of text: 1) Authors A describes algorithm, 2) Author B publishes counter-example to show where #1 fails, 3) Authors A say it wasn't wrong, but that the author of #2 'misunderstood', and B should have contacted them first; and in any case here are the missing details, 4) Author B points out that paper #1 should have said those details were missing. 5) Authors C point out that authors A misunderstood many things in their own publications; authors A can't complain about others not contacting them first when they don't do it themselves.
"Canonical Numbering and Constitutional Symmetry" (1977), DOI: 10.1021/ci60010a014 describes an algorithm.
"Erroneous Claims Concerning the Perception of Topological Symmetry" (1978), DOI: 10.1021/ci60014a015 points out examples where the algorithm from the first paper, and from another paper, don't work.
The authors of the first paper followup with "On the Misinterpretation of Our Algorithm for the Perception of Constitutional Symmetry" (1979), DOI: 10.1021/ci60017a012 .
> A recent paper in this journal contained critical comments on two methods for the perception of topological symmetry. Carhart’s claim that our algorithm does not correctly perceive topological symmetry and fails with certain structures is the result of a misinterpretation of our algorithm.
> Unfortunately, the author did not contact us directly to help him clarify his misunderstanding. This failure is unusual and difficult to understand. Thus, it was not until we received the recent issue of this journal that we learned of this misinterpretation.
> In our paper we were particularly aiming at catching the interest of the organic chemist for the problems of uniquely numbering the atoms of a molecule. Therefore, we put particular emphasis on the criteria for determining priorities among atoms to enable the chemist to manually number the atoms of molecules according to our procedure. We restrained from giving all small details of the algorithm to keep the paper concise, working under the assumption that persons interested in the details would contact us directly. It is astonishing that Carhart at the point where we did not fully elaborate on the details works with the premise that we misconceived the problem. Initially one should rather assume that other people, too, understand a problem. Only if explicit errors are found should one digress from this conviction.
Carhart followed up with a letter to the editor, "Perception of Topological Symmetry" (1979) DOI: 10.1021/ci60017a600 :
> I am delighted to see that my critique’ appearing in this Journal has encouraged C. Jochum and J. Gasteiger to present previously unreported steps in their algorithm for the canonical numbering of chemical graphs. They refer to these steps as “small details”, but in fact they are the very essence of any routine which reliably finds unique numberings for, ...
> However, I did not misunderstand their previous article (unless lack of clairvoyance can be classed as misunderstanding); I simply took it at face value. My critical comments, and the counterexamples I presented, were completely appropriate in the context of that article. In contrast with their latest offering, Jochum and Gasteiger’s previous paper did not present a sound and accurate definition of constitutional symmetry, nor did it indicate in any way that crucial steps had been omitted. I am sympathetic with the problems of describing a complex algorithm in the limited space of a journal article, but if space limits the development of a fundamental concept, it is the responsibility of the author to say so, and to indicate that a reader must obtain additional information before he tries to implement the described procedure.
It ended with a letter from still other people writing another letter to the editor, "Canonical Numbering" (1979), DOI: 10.1021/ci60019a600 :
> We have been following with some interest the controversy appearing in this Journal regarding canonical numbering and various types of The first article by Jochum and Gasteiger contains a number of incorrect and misleading statements about both their work and the work of those who preceded them. ...
> Jochum and Gasteiger also strongly implied that they had a “simple” algorithm which gave complete partitioning, eliminating the need for a comparison step. Carhart correctly pointed out that this was not the case. Subsequent publication of the details of Jochum and Gasteiger’s indicated that it does contain a comparison step ...
> On a more general level Jochum and Gasteiger complain that Carhart did not contact them “directly to help him clarify his misunderstanding”. Yet it is obvious from the large number of misinterpretations and/or misrepresentations which appear in their work that they made no attempt to clarify their misunderstandings by discussing such matters with the original authors. Publishing last on a particular subject accords one considerable power, power that carries with it the responsibility to treat the preceding work with fairness and objectivity.
A paper and its supplementary materials are supposed to be enough to reproduce the experiment. In practice, this often fails, but that is a fault in the scientific process. Science isn't just about empirical knowledge, it's about public and redundant empirical knowledge, as opposed to losing important knowledge of the natural world when the original investigator gets hit by a bus.
Then maybe the next generation of researchers documents their work better.
Or maybe the original researcher publishes a v2 edition of their paper.
That we use written communication that persists through generations is the basis of science and society in general. If we cannot communicate sufficiently via papers, we're in a world of trouble.
The core problem is that human communication is very difficult. It becomes even more difficult when we try to communicate ideas without interaction, as we do when writing a book and expect someone to read and understand it. If I read a paper and I can't understand a sentence, it might take me days to figure out what's going on by myself, whereas asking an expert might yield an answer in less than an hour (sometimes minutes). The difference is really orders of magnitude.
There are whole fields that have effectively died because no one works on them any more. That knowledge doesn't live in anyone's mind. All the literature is there, but actually acquiring that knowledge by reading the literature is incredibly challenging and time consuming.
I have come to believe that the main purpose of hiring scientists in academia is to keep knowledge alive and have it passed on to future generations. Advancing research is of secondary importance. In fact I would say that most new research I see probably has no intrinsic value. I include my own research in this category. We have researchers solving esoteric problems of no value to anyone besides their own personal entertainment. Except, working on such research keeps our neurons firing and keeps knowledge alive. It is a well known phenomenon that taking a break from research very quickly leads to a sort of decay of memory. Our learned ideas and the connections between them wither away without constant reinforcement. In order to keep knowledge alive we have to engage in research, even if it seems pointless.
Then these scientists should be devoted to producing textbooks and courses which can then be taught to non-research students. Yes, all knowledge about the scale of what a single individual knows (and keeps on their shelves, hard drives, etc) is embodied as communities and traditions, but we still get far greater redundancy of that knowledge from teaching it as undergraduate or master's-level coursework than from passing it down only via research mentoring.
If 25% of the population gets an undergraduate degree, 11% or so gets a postgraduate degree, and only about 1.7% get a PhD, then we need to be embodying society's knowledge among the larger cohorts for that knowledge to survive. We can't afford to live in a world where only 1.7% know how things work.
Textbooks and courses exist for everything but the most cutting edge stuff (which are still in flux anyway), but they are a very inefficient way of transferring knowledge. I would say they are practically useless without expert guidance. At the most basic level, there are so many of them that an expert has to tell you which ones are both good and relevant to what you want to learn. I've once seen a student waste months of his life studying a book he thought was relevant, only to discover that book wasn't building towards the sort of knowledge he needed in that subject. The book was about the correct subject, but was focused on somewhat different aspects than the ones he was interested in. There was no way for him to know this in advance without guidance.
So we don't know how to organize existing books. Also, even the books that exist are usually pretty bad at conveying knowledge. Or perhaps humans are just pretty bad at learning things from books. Either way, no one knows how to write textbooks and courses that are much better than what we have today. I really don't know of a better way to preserve knowledge than the current one. Perhaps technology can improve the situation by making access to knowledge more interactive. But I suspect this would require a real breakthrough.
> We can't afford to live in a world where only 1.7% know how things work.
Why not?
The tweaks I did to glue things together has no theoretical value and don't belong in the paper. As a practical matter, I can't fit a lot of source code into short paper format.
What do?
It almost never happens that a paper I read actually comes with usable source code.
Then leave out the results since they are just an anecdote. If you want to include experimental results then it has to be done in a scientific fashion.
So, the possibility of things like, say, a researcher not mentioning something that is standard practice in their lab that later is found to be a crucial part of the setup for an experiment seems high. But just like you don't want to provide a dictionary of standard terms with a paper you don't want to provide a list of the chemicals used to mop the floor, or a list of the lumen and colour temperature ratings of lights in the fume cupboards, or ...
IMO if a paper is not reproducible then yes it should be published but also the original team producing the paper should be challenged to reproduce the results. It's not a fight, we're all on the same team - work with them and try to find the reason for the lack of reproducibility.
I'd suggest a different formulation: "standard practice in their field"
Standard practice in general cooking? That's ok. Standard practice in my kitchen? That's a problem.
The research is IMO like a meal recipe a knowledgable chef should be able to reproduce.
Though it is understandable why one would forget to mention something. Especially if they thought it was general practice to do something their way.
There is a page limit to publications in high impact journals, and generally it's not great practice to utilize the limited space on the details of hurdles overcome.
I would argue that some of the most important papers in science don't really stand on their own... they need context and expertise that the paper can't and shouldn't cover.
> The gist of her article was that she feared too much of the criticism of past work had been ceded to social media, and that the criticism there is taking on an uncivil tone.
So it seems to me, her complaint was not that research shouldn't be criticized. It was more-so that research should be criticized well, and that social media is a poor place for it, because it favors personal attacks over thoughtful criticism. (I'm not saying I agree with her. Just trying to understand her argument in the best light.)
http://frog.gatech.edu/Pubs/How-to-Publish-a-Scientific-Comm...
"Eminent" specialists often argue in favor of eminence (journals, tenured investigators, blah blah) over evidence (tossing an analysis up on arXiv and posting a link to it). Young turks or iconoclasts often do the reverse, since there is a power structure in place via editors, reviewers, study sections, and the like. Orthodoxy often sticks even when the evidence supporting it is quite skimpy (or absent).
Funny thing is, when you pull the underlying data (in fields where this is possible), you routinely find that the reported conclusions are overblown. Not necessarily wrong, but routinely sold as more conclusive than they are (and sometimes they are in fact wrong, whether due to small sample sizes, "outlier removal", or outright fraud).
In no way, shape, or form are these bad habits limited to psychology. They happen in basic biology, they happen in medicine, they happen in clinical trial reporting. It's faster and easier to oversell shitty science than it is to do good, thorough science, so with the tight competition for grant funding, you can imagine what happens next.
Trust whoever you like, but verify results from everyone. There are good people out there who are fastidious; there are good people out there who are sloppy; and there are people who don't care one bit whether they're publishing absolute horse shit. The onus is on you, the reader or researcher, to do the requisite critical thinking (and, perhaps, a few analyses before you waste time running down a dead end).
Keep an open mind, but don't let your brain fall out. Also, many good scientists aren't very nice people, and some very pleasant people in science are shitty scientists. It's very hard to tell a priori which is which, so look at the evidence and decide for yourself.
It's quite unscientific to draw a blanket conclusion like that about a field based on the anecdotal evidence of one named person and some unnamed "critics".
Let's get some statistics before casting doubt on the entire field of psychology.
I think most people are trying to get by and to be good. My personal experience in academics is that the vast, vast majority of students and teachers just want to study their art or science and contribute to their community without having to spin or sell to get funding. Some of the people that you hear about in articles get attention because they like attention, and they will happily say opinionated, contentious and unscientific things because it gets talked about.
There are bad actors with high levels self interest in all fields of study and work, so really the only scientific conclusion you can come to is that not all humans are perfect.
How many psychologists have/will publicly denounced her? HN is ready to attack every police officer who hides and defends the actions of a few bad apples as bad apples themselves (rightly so), where's the equal treatment for psuedoscientists?
Well, no; even we accept the arguments that (1) professional organizations of psychologists are a good proxy for generalizations about psychologists, and (2) the head of an organization is a good proxy for the organization, and (3) that each of those relations are strong enough that treating the relationship as transitive such that the head of a professional organization of psychologists is a good proxy for psychologists in general, she still wouldn't be the first place we would look to represent psychologists as a whole.
Because she's a former head of such an organization, not the current head. Just as we wouldn't look to Jimmy Carter as our first choice to represent what Americans as a whole believe (or John McAfee as our first choice to understand what the business organization now known as "Intel Security Group" believes), we wouldn't look to the former head of an organization as the first choice for generalization about what the group the organization is supposed to represent believes.
Positions (1) (2) (3) are nearly axiomatically true -- if we are to generalize at all, and we must, we need to look for positions and people to generalize from. The very purpose of a professional organization is to aid us in doing this. Members of these organizations use it as a mental model to generalize "my professional community" and "the leaders of my professional community".
If you look at her resume, she is the former head of this and about 4 other high-prestige psychologists professional organizations, and a high ranking member of nearly every organization she is eligible to be a part of as a psychologist. She is the very definition as "well-respected representative of her field". She is a fine target for generalization. What you have not brought to the field are conflicting reports from other psychologists contradicting her or suggesting she is not an appropriate representative. Jimmy Carter and John McAfee are readily directly contradicted -- if you ask me to provide sources for numerous professionals in the field suggesting they are not the standard I can do so easily and so can you (though Jimmy Carter is a weird selection -- he is not a professional, and both are telling, since neither of them work in their field anymore)
I don't know Susan Fiske, and while she might deserve it, I won't be joining the crowd of pitchforks to judge and attack her based on a few forum comments. But, if you really want to excoriate someone who represents a lot of people poorly, it might be energy better spent and more timely to worry about Mike Pence who, I've heard directly from a handful of academics in Indiana, is doing very bad things for science and science funding.
is a proposition. He/she's not concluding psychology is not a proper social science; he/she's stating
> If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science.
But apparently we also have Susan Fiske. I think that the real difficulty with psychology is that people care too much. Its a little easier to be objective around the decay of atomic particles than about the intelligence of different genders and races.
Additionally, many psychologists are not comfortable with statistics. A large proportion went into psychology to avoid maths, and are somewhat disappointed to discover that this does not work.
So they follow guidelines, and paste tables from SPSS into MS Word (no automation for some reason), and stuff like this happens.
They don't use cross-validation and they are too much in love with their models, and the theory that said models facilitate.
I suppose this was a pretty long-winded way to say "Not all psychologists!"
Cohen, J: http://www.ics.uci.edu/~sternh/courses/210/cohen94_pval.pdf (The World is Round (p< 0.05))
http://andrewgelman.com/2016/09/21/what-has-happened-down-he...
Yup. There are powerful social forces which act to dissuade psychologists from publishing research which opposes to feelings of their peer group. One can't speak an inconvenient truth.
In my experience, such attitudes are typical. Here's a 2014 paper by a Harvard psychology professor who tries to argue that null replication results shouldn't be taken seriously because the failure to replicate ipso facto indicts the study's methods:
http://jasonmitchell.fas.harvard.edu/Papers/Mitchell_failed_...
Quote: "Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way ... The field of social psychology can be improved, but not by the publication of negative findings."
Wow... I really don't see it like that.
There was a story about two chemists, I might have some minor details wrong, but that shouldn't hurt this anecdote because something like this happened back when chemistry was still raw and new. The going wisdom was based on the assumption that the atoms in a molecule were the only thing that determined its properties. These two scientists were having trouble replicating each others results when working with a molecule of 2 Carbon, 2 Nitrogen, 2 Oxygen and 1 Mercury atoms.
One describe it as inert and the other as explosive. Neither bothered to include synthesis steps in their papers and each thought the other was wrong. Rather than insult they swapped more and more information until eventually each confirmed that we dealing with a molecule with identical parts, but different properties.
They had discovered that molecules have structure. One arrangement was Mercury Fulminate an unstable explosive used in primers and detonators today and the other was really inert and not of interest, to me at least.
Any scientist who argues with the concept of "disagreement based on evidence" is arguing a foundation of science. They should be looking for evidence of reasons for that disagreement instead of quibbling over personal attacks and defense.
The article went kinda off-topic on itself, by throwing in quotes about a completely unrelated incident, which on skimming may be read as referring to statcheck, despite having nothing whatsoever to do with it.
This is a generalization about psychologists from a sample size of one psychologist, to suggest that they don't understand science?
It would similarly be completely inappropriate to generalize the policy positions of Hillary Clinton to the Democratic party, yes?
Also to say i think it's a graded thing. Often such type of critique might be serious and well developed, often it might be serious but based on misunderstanding, and sometimes it might be an attempt at trolling, annoying or payback. I would guess your average peer reviewer engages more deeply with the paper than most readers and takes time to check their critique before sending it, and since there's an editor involved there might also be a better level of politeness. If now suddenly every reader is free to criticise (assuming they fall in category 1 & 2) that can end up in a barrage of critiques that an author might feel obliged to respond to in order to protect his/her reputation. And still those critiques can then be taken out of context, eg when someone in the media or from the competition tries to put you down a notch for holding the wrong views...
Think of anti-vaxers going around and criticising every paper on vaccines. You're always going to find something to criticise and once it's criticised you can point to SG authoritative 'the paper is criticised by other scientists for..'
Or another angle: do you want to feel obligated to defend every one of your papers for the rest of your life?
This can really feel like bullying even if it's not intended as such. That's in particular if you write about contentious issues or such with strong interest groups - think vaccines, abortion, gender, GMO, lobbying, financial regulation, inequality, race, ...
I think open 'peer review/critique' is a good idea, but it can take dark turns if there's no mechanism to prevent this.
Also, without meaning to defend Fiske, her comments here are taken out of context somewhat. Her reference to "methodological terrorism" (from what I have heard through the grapevine) is more, or at least in part, about the trend toward having scientific debates outside of the peer review process, in social media. So my guess is that she might say that part of what she objects to about Statcheck is that it crawls through the papers, labels an error, and then we end up discussing it on HN rather than through peer review. What if Statcheck made an error, which it does sometimes? I don't agree with it, but I think the position I'm describing (which I think is her point in part) isn't unreasonable either. That is, it's not the checking of stats, it's the chaos and disintegration of the peer review system, and "extrascientific" discourse that's happening in science today, if you define "extrascientific" as "outside of peer reviewed journals," where your critics are attacking you on twitter, forums, and facebook, more so than in professional published outlets in a sort of mob.
Again, I do not share her perspective at all (I'm in favor of a shift away from journals) but I do think here her original point was twisted a bit.
https://hardsci.wordpress.com/2009/03/14/making-progress-in-...
> What are the “hard” — as in difficult — problems in science? Hard problems in science are those that are embedded in complex systems; they are hard because to study something well you often need to isolate it from outside influences. Hard problems are those that vary by local conditions — science seeks to identify general laws, and when something is locally dependent, you need to sniff out the complex interactions that make it so. And hard problems are those that are difficult to quantify — science rests upon formalization and quantification, and you need to get traction at that initial step of quantification (i.e., measurement) before you can test theories. So… by these measures, if we are going to differentiate areas of science, the continuum of scientific problems should go from “hard” to “easy,” and psychology is clearly a science that deals with hard problems. Perhaps the hardest.
Reminds me of the counterintuitive about the two hard problems in computer science. Naming things, and deciding when data is obsolete, are easy mechanically. But because they have neither a deterministic nor correct solution, they end up being the hard problems you face as you build bigger projects. Or, to emphasize a line from Dr. Srivastava above:
> And hard problems are those that are difficult to quantify — science rests upon formalization and quantification, and you need to get traction at that initial step of quantification (i.e., measurement) before you can test theories.
It's possible for a science to be "hard" while having a lot of mediocre practitioners.
Is 'social science' even a 'proper science' ?
No, science is a means of generating (and justifying) beliefs, it is not knowledge itself.
> Susan Fiske, a former president of the Association for Psychological Science, alluded to Statcheck in an interview with Business Insider, calling it a “gotcha algorithm.”
> The draft of the article said these online critics were engaging in “methodological terrorism.”
If these are attitudes typical of psychology, then I cannot say I consider psychology to be a proper social science. There is a fundamental misunderstanding of how knowledge is created through the scientific process if the verification step is considered to be offensive or taboo. That anyone in the field of psychology would even be comfortable publically espousing a non-scientific worldview like that means that psychologists are not being properly educated in the scientific method and should not be in the business of producing research since they do not have a mature understanding of what "scientific" implies.