Tilted Forum Project Discussion Community - View Single Post - Is there something wrong with the scientific method?

filtherton · 01-24-2011, 09:43 PM

A few thoughts. Maybe tangential.

One thing that is commonly glossed over in discussions about the nature of scientific certainty is the role that uncertainty plays. Uncertainty is everywhere. Typically, study designs seek to minimize the effects of likely sources of uncertainty and then statistical analysis is used post-data collection to gauge the extent of residual uncertainty and compensate for it.

This introduces at least two sets of problems. With respect to study design, investigators can only design to mitigate known sources of uncertainty and bias. There is no shortage of examples of clinical studies which seemed well designed, but ultimately failed because bias wasn't accounted for in the design of the study (at least my professors never seemed to run out of them) resulting in crippling levels of uncertainty. Obviously, if attempts to replicate flawed research themselves contain the same flaws, the newer results can either agree or disagree with the original results and still not accurately describe reality.

The second set of problems comes from the nature of statistical significance. Long story short, research generates data which is then analyzed using appropriate (hopefully) statistical methods. Each method has its own set of assumptions about the nature of the underlying data. Also, methods differ with respect to how accurate the results they generate are when their underlying assumptions are violated.

The basic strategy is this: gather data, look at it, determine appropriate statistical test, use test to generate appropriate test statistic (basically a number generated from the data via a test-specific method), compare this test statistic to what you'd expect it to be if your assumptions about the nature of the data are correct. If your test statistic is outside the range it should fall into 95% of the time, you say "Our results are significant (ie outside the 95% range) and they are ___________"

95% is arbitrary. Each time one of these tests is done, it's like someone is flipping a lopsided coin where 95% of the time heads comes up. Assuming the correct test is performed for each set of data, one should expect statistical significance to be erroneously found at most approximately 50 times for every 1000 significant results. I say at most, because many papers report a greater than 95% confidence level, say >99% or >99.9%. Even so, the sheer number of published results ensures that there will be many that find effects that aren't true.

The waters are further muddied by the fact that it is really easy to manipulate results using statistics. Your first analysis doesn't give you significant results? Try reformulating the age ranges in your analysis. Try limiting your analysis to a subset of your subjects. Repeat your analysis enough times and you're likely to stumble onto statistically significant results by sheer chance, never mind that they'll be illusory.

Further problems come from the fact that most consumers of scientific literature don't get beyond the press release or the abstract because they either don't have the time, don't want to pay to get past the pay wall or they lack the expertise to understand the paper.

None of this is to say that metaphysical alternatives are more compelling, or provide a more evidence-based foundation for understanding the world. However, I agree with roach that in certain types of discussions, the level of certainty generated by science is often given a level of reverence that is wholly unjustifiable in light of the amount of uncertainty inherent in actual research.

01-24-2011, 09:43 PM	#7 (permalink)
filtherton Junkie Location: In the land of ice and snow.	A few thoughts. Maybe tangential. One thing that is commonly glossed over in discussions about the nature of scientific certainty is the role that uncertainty plays. Uncertainty is everywhere. Typically, study designs seek to minimize the effects of likely sources of uncertainty and then statistical analysis is used post-data collection to gauge the extent of residual uncertainty and compensate for it. This introduces at least two sets of problems. With respect to study design, investigators can only design to mitigate known sources of uncertainty and bias. There is no shortage of examples of clinical studies which seemed well designed, but ultimately failed because bias wasn't accounted for in the design of the study (at least my professors never seemed to run out of them) resulting in crippling levels of uncertainty. Obviously, if attempts to replicate flawed research themselves contain the same flaws, the newer results can either agree or disagree with the original results and still not accurately describe reality. The second set of problems comes from the nature of statistical significance. Long story short, research generates data which is then analyzed using appropriate (hopefully) statistical methods. Each method has its own set of assumptions about the nature of the underlying data. Also, methods differ with respect to how accurate the results they generate are when their underlying assumptions are violated. The basic strategy is this: gather data, look at it, determine appropriate statistical test, use test to generate appropriate test statistic (basically a number generated from the data via a test-specific method), compare this test statistic to what you'd expect it to be if your assumptions about the nature of the data are correct. If your test statistic is outside the range it should fall into 95% of the time, you say "Our results are significant (ie outside the 95% range) and they are ___________" 95% is arbitrary. Each time one of these tests is done, it's like someone is flipping a lopsided coin where 95% of the time heads comes up. Assuming the correct test is performed for each set of data, one should expect statistical significance to be erroneously found at most approximately 50 times for every 1000 significant results. I say at most, because many papers report a greater than 95% confidence level, say >99% or >99.9%. Even so, the sheer number of published results ensures that there will be many that find effects that aren't true. The waters are further muddied by the fact that it is really easy to manipulate results using statistics. Your first analysis doesn't give you significant results? Try reformulating the age ranges in your analysis. Try limiting your analysis to a subset of your subjects. Repeat your analysis enough times and you're likely to stumble onto statistically significant results by sheer chance, never mind that they'll be illusory. Further problems come from the fact that most consumers of scientific literature don't get beyond the press release or the abstract because they either don't have the time, don't want to pay to get past the pay wall or they lack the expertise to understand the paper. None of this is to say that metaphysical alternatives are more compelling, or provide a more evidence-based foundation for understanding the world. However, I agree with roach that in certain types of discussions, the level of certainty generated by science is often given a level of reverence that is wholly unjustifiable in light of the amount of uncertainty inherent in actual research.