Friday, September 14, 2007

"Most Science Studies Appear to Be Tainted By Sloppy Analysis"

Wall Street Journal article here.

3 comments:

Unknown said...

I left the link to that paper on a previous thread over year ago. Those that haven't read it yet, should.

Anonymous said...

So, uh, what's the alternative?

Anonymous said...

Yes, that could be filed under the category of "seems surprising, but not completely unexpected", though not necessarily a desirable situation.

As a former researcher with high ideals about science, it took me a long time to come to some fundamental realizations about the contradictions inherent in publicly-funded research.
1-Anyone trying to make a career in science today has to publish a lot, because there is tremendous competition for funds. It also helps if your papers have high impact. The surest way to publish a few things with high impact is to have lots of good ideas and to publish them all--those with high impact will get noticed.
2-Furthermore, the public, quite rightly, expects results from public funding. The easiest way to measure results is by publication.
3-Any scientist who makes sure that every last error and contradiction is weeded out of a paper before submitting it for publication is going to have a very short career.
4-Scientists are just as human as the rest of us, and are quite capable of a little bit of self-delusion if it is in their own best interests. (That's not necessarily a good part of human nature, but that's the way we all are.)

The situation is really rather similar in other walks of life. For instance:
-Companies that create new consumer products in the hope of making money don't really have the time to do perfect product testing so that results are assured. Some other company will get a competing product out there if they take the time for perfection. They hope that they will, on the balance, put more successful products on the market than flops.
-Firms selling investment vehicles (mutual funds, venture capital funds, derivatives) don't have the time to do absolutely perfect research about the things they sell. They do the best research they can, given the time constraints they have. Wise investors know to take a sales pitch, even if dressed up with lots of "research", with a grain of salt.

In a similar vein, wise "consumers" of scientific literature know to take initial findings with a grain of salt. In science, replication of new findings by independent parties is paramount.

Again, I don't think the situation is necessarily a good thing, or the best way for science to operate, but it is not that big a surprise.