Good news from science

This is a really big deal. The NIH is now funding research into ways to enhance scientific rigor. This should be a game changer. I hope it's not too little or too late. Steve Novella at Neorologica writes:

This is a great idea, and in fact is long overdue. The NIH is awarding various grants to establish educational materials and centers to teach principles of scientific rigor to researchers. This may seem redundant, but it absolutely isn’t.

At present principles of research are taught in basic form during scientific courses, but advanced principles are largely left to individual mentorship. This creates a great deal of variability in how well researchers really understand the principles of scientific rigor. As a result, a lot of research falls short of scientific ideals. This creates a great deal of waste in the system. NIH, as a funding institution, has a great deal of incentive to reduce this waste.

The primary mechanism will be to create teaching modules that then can be made freely available to educational and research institutions. These modules would cover: 

“biases in research; logical fallacies around causality; how to develop hypotheses; designing literature searches; identifying experimental variables; and reducing confounding variables in research.”

Sounds like a good start. The “biases in research” is a broad category, so I’m not sure how thorough coverage will be. I would explicitly include as an area of education – how to avoid p-hacking. Perhaps this could be part of a broader category on how to properly use statistic in research, the limits of the p-value, and the importance of using other statistical methods like effect sizes and Bayesian analysis.  

Prior research has shown that when asked about their research behavior, about a third of researchers admit (anonymously) to bad behavior that amounts to p-hacking. This is likely mostly innocent and naive. I lecture about this topic all the time myself, and I find that many researchers are unfamiliar with the more nuanced aspects of scientific rigor.  

And of course, once the NIH requires certification, this will almost certainly make it uniform within academia, at least on the biomedical side. Then we need other research granting institutions to replicate this, also requiring certification. It basically should become impossible to have a career as a researcher in any field without some basic certification in the principles of research rigor.

OMG, someone outside Dissident Politics is actually taking logic fallacies seriously? I must have died and got reluctantly shoved up to heaven. Next after science, politics needs to tackle this same plague on democracy, humanity and civilization.

No, it is not the case that science and politics can be dealt with the same way. They are different. But it is the case that the data and reasoning behind politics can be subject to the same kind of rigor, if politics is to be based more on fact and sound reasoning than it is now. Opinions will still differ, but the extent of difference due to irrationally disputed facts, e.g., stolen election vs. not stolen, differences in opinions ought to be significantly reduced. Everyone doing politics firmly believes their politics is based on real facts and sound reasoning. A lot of research indicates that just is not true for most people, most of the time.

Politics is mostly sloppy, not rigorous.