What makes "good science"?

One thing that science really taught me was how to evaluate something critically, rather than just accept what I'm told. Pseudoscience flourishes because so many people don't really know how to do that. That allows the kooks to get away with a lot. You're absolutely right: the media too frequently will just spit out information without really examining it.

Some branches of science lend themselves to this kind of stuff. The less hard evidence and the more speculation involved, the more trouble you're gonna get. Scientists are just as susceptible to ego as anyone else and they sometimes put their egos before the work.

My first line of defense: if it hasn't been peer-reviewed, it probably ain't very good. I know it's a bias, but if your results are good, you shouldn't have any trouble running them by a group of reputable scientists.

 

In the US, the population is generally scientifically illiterate. A glaring example is creationism. From a Gallup polling data set, in 2012 46% of respondents believed that God created humans "pretty much in their present form" in the past 10,000 years. In 2007, when asked to rate "Creationism, that is, the idea that God created human beings pretty much in their present form at one time within the last 10,000 years," 39% said that it's definitely true and 27% said that it's probably true. Not only is that a symptom of the problem, it's part of the cause. Indoctrinating children with science denialism from a young age perpetuates ignorance.

In general, I've gotten to the point that I tend to shrug off pseudoscience. I've given up on trying to talk sense into people about GMO foods, nobody among anti-GMO friends has any formal science education and I literally get shouted down when I try to present any evidence in favor of whatever memes are going around on Facebook. My mom in particular is bad about it, I get accusations of not being open minded (being closed-minded and demanding evidence are not the same thing,) and she reminds me of everything I've ever been wrong about and tells me to "be open-minded" any time I try to explain that there's no evidence for something. And you can't discount new ideas with nothing backing them or even say that you should wait for more evidence because she took folic acid and stopped taking Vitamin A when she was pregnant with me based on small pilot studies at the time that turned out to be right a decade before there was enough evidence for the medical community to form a consensus.

From a reporting standpoint, everything is sensationalized and any new study is treated as if it's the only research that matters rather than being evaluated on its merits and considered along with prior evidence. It's always "Scientists baffled by results of a new study ..." and "Was everything we thought we knew wrong?" because the people reporting it are unfamiliar with how the scientific method works.

 

And I certainly didn't mean to imply that. Critical thinking is a skill that everyone should have, regardless of their walk of life. It's just that science and pseudoscience appeared to be the point of this discussion thread.

 

The scientific method is good, and we should be looking for repeatable results. But, there is something missing when it comes to convincing other scientists or the general public that what you have tested is valid. Or that it is possible to test what you claim.

On the other side, 'new' scientific discoveries get over-hyped a lot of time for the competition or name recognition of being first. Also the media likes to report of stuff early, not once it is well tested. Communicating something with a person who knows nothing about the science in a few minutes is tougher to do than communicating it with another scientist.

 

It looks like the armchair scientists will have less input on science news.
Why We're Shutting Off Our Comments | Popular Science

I think it is for the best right now. Other people's opinions shouldn't be a influence on what is real and demonstrable, and internet comments about a story can polarize people and push the moderates to one side.

 

That is awesome! While I grew up (very young), lateral interpretations were not encouraged. We were all told to align(!) and be normal(!) in our thinking. I have some interesting examples in my memory even now.
--- merged: Sep 25, 2013 at 3:46 PM ---
To me Good Science is something I can either experiment myself or watch closely how someone else experiments it and follow (at least most of) it.
I am ok even if the experiment is done in a simulated environment as long as the simulation is built on facts or proven rules.

 

I look for honesty and math. I'm fine with a theory, as long as you present it as such. List your next steps and concerns, as well as details that you can't quite explain.

I expect rigorous and open math. List your sample size, collection methodology, and express your results in terms of standard deviation. Make it an appendix or link if it's unwieldy; but make it available.

I'm beyond skeptical, I assume folks are lying if they can't be bothered to demonstrate otherwise.

 

Without aggressive moderation, comments on anything other than a dedicated discussion space do more harm than good. It might be nice if people could ask honest questions, but it just goes to shit otherwise.

By "follow (at least most of) it," can I assume that you just mean you can see the process? I would agree with that, but I doubt that you or I could just throw on a labcoat and follow advanced research in most fields.

 

Related: James Randi, The Amazing Meeting, and the Bullshit Police - Newsweek and The Daily Beast

Snippet: TAM is organized by the James Randi Educational Foundation (JREF), a group devoted to a philosophy called skepticism: the debunking of psychics, mediums, pseudoscientists, faith-healers, homeopaths, and anyone else who makes claims that defy the known laws of science. Skepticism has a wide following—the Internet is littered with self-proclaimed skeptic blogs, podcasts, and forums—and JREF is widely acknowledged to be the movement’s hub. Over 1,000 people attended this year’s conference, which featured an array of panelists and speakers, from magician Penn Jillette to comedian Father Guido Sarducci to Steven Novella, a professor at the Yale School of Medicine. (And yes, it was ironic that this militantly rational group decided to hold its annual meeting in a casino.)
Ingrid Laas/Courtesy of James Randi Educational Foundation
James Randi speaks at The Amazing Meeting in 2013.
The activists of TAM see themselves as waging a broad, multifront battle to drag American culture, inch by inch, away from the nonscientific and the nonlogical. This turns out to be a surprisingly uphill struggle. Probably the majority of Americans believe in some degree of what JREF’s founder, James Randi, calls “woo-woo.” (“Please use woo-woo,” he instructs me. “I’m trying to get it into extensive use.”) In 2005, for instance, Gallup found that 73 percent of Americans subscribed to at least one paranormal belief. Television personalities like John Edward earn huge audiences by purporting to commune with the dead. Numerous Americans swear by homeopathy, ingest supplements with no proven medical benefit, or believe, against all available evidence, that genetically modified organisms might transform humans into tumor-covered golems.

 

The issue is, what do you believe when we haven't done the scientific studies or it is impracticable to do them? I've been saying that there needs to be some 'multi-variable' type of study procedure that also takes into account real world statistics for medical research. What works for one group, might not work for another due to something in their lifestyle or gender). But, they should also figure out how the drug works in the body and why it does what it does.

You also have to be a little unethical when it comes to some medical studies. Even if you find that 95% of the patients taking a drug were cured in 3 months, and 0% of the patients taking the placebo got better, you have to either repeat the study or let it go on to see if the results change with time.

 

Exactly! By follow I mean even if I am watching the experiment on TV or Tape, and able to rationally understand the step by step process, with no mere assumptions or beliefs, and understand the flow, then that is good science!

 

Umm, no, you don't have to be unethical to run clinical trials.
Data safety monitoring boards monitor trials and look for uneven outcomes for both positive endpoints and negative ones (side effects). They can and routinely do prematurely terminate trials that are having a negative impact on a subset of patients.
Also, a study with such a wide discrepancy of outcomes as you cite above would not be repeated.

 

I like science, and I love working in its bowels. Good science is like pornography, in that it can be tricky to define, but you can know it when you see it.

I've put in a fair amount of time in public health science. Like most professions, you have people who don't give a fuck who are just looking to make a name or get a paycheck. These people want to publish and don't care if their results are valid in any rigorous way. They are playing the academic funding/career advancement game. There are also a lot of people who passionate about doing things right. This latter group still has to play the game, but they try to do good science in the process.

I think the peer review process is deeply flawed. Bad research gets published all of the time and so bad ideas get undeserved credibility. Ideally, science should correct itself through repeated experiments, but, in reality, funds are limited and journals favor results that find associations rather than results that find nothing. Even if subsequent research explodes the findings of previous research, the media rarely focuses on these subsequent results, so you have the general public thinking things are true that aren't. There are plausible arguments that most published medical research is wrong.

Science as a brand is hot right now, as evidenced by the success of things like "I fucking love science". This means that there are a lot of science hipsters and poseurs. It's cool to be a nerd right now, but only if you're a sexed-up pretend nerd. Real nerds are still super uncool. Most people who "fucking love science" don't actually love science, they just like the way science's ass looks when its wrapped in a tight meme-generator-patterned dress. Science, by its very nature, is something that cannot be consumed on a meme-to-meme basis.

There's a lot of pseudoscience, and I don't really care about it all that much. People will always believe what they want to believe, and this will always be true in the context of science. People don't care about empirical evidence unless they're paid to, and most people aren't paid to care about empirical evidence. Shit. A lot of science becomes pseudoscience via how it's reported in the media. A lot of published research is pseudoscience when view through the right methodological perspective.