Top scientists who investigated a fraudulent article co-authored by a University of Pittsburgh biologist and South Korean cloning researchers are calling for tighter scrutiny over how high-profile research papers are selected for publication in scientific journals.Donald Kennedy's editorial "The Ivory-bill Returns" is available here.
...
Science editor-in-chief Donald Kennedy said the journal would consider the committee's recommendations, but didn't commit to any of them.
"We may now have to abandon the hope that the collective trust will keep on working," Kennedy said, referring to the trust journals place in the integrity of scientists.
Club of Rome: COP29 “No longer fit for purpose”
33 minutes ago
Posts
18 comments:
This Panel should re-read our dear Carpinterio's comments on this issue. "Tougher scrutiny"? How about just any peer review? Science didn't even do their basic due diligence on the CLO articles.
As the Carpinterio would say, "Sibley was your peer review, you dolts!"
"We may now have to abandon the hope that the collective trust will keep on working," Kennedy said
We may very well have to abandon trust in Donald Kennedy and other Science editors, but not in the competence and good will of scientists in general. Old-fashioned peer review works just fine when applied with integrity.
Kennedy lauds those who "assist Cornell with the Partners in Flight program." Uh, sorry, but PIF is not a Cornell program. Its this kind of sloppy confusion that got him in this whole mess.
Here we go casting more stones, this time at Donald Kennedy.
"Old-fashioned peer review works just fine when applied with integrity."
Of course it does--with integrity--but who is the judge of integrity? Old-fashioned peer review consists of the editor forwarding a manuscript to experts in the field, usually to two different experts (sometimes more or less, depending on the journal) who provide peer review. Normally the editor's decision is based on the recommendations of the reviewers, unless their is reason to believe one or more reviewers were unfair, in which case an additional reviewer or two is sought.
I recall reading an earlier comment from somebody that the ivory-bill rediscovery was a hoax because (among other reasons) Cornell's paper was not reviewed. Are any others of you claiming that the editor of Science failed to submit the Cornell paper to peer reviewers? If Donald Kennedy did something wrong with the Cornell paper, I'd like to know what the specific allegations are.
And if you ever review a paper, how many hours do you spend reviewing it? What percentage of scientists do you believe spend more than 3 hours reviewing a paper on average? Can you thoroughly investigate the possibility that fraud occurred within the typical hour, two or three it takes to review a paper?
Let's be realistic here: scientists are humans, and as long as scientists are going to publish their results, fraud is going to occur and cannot always be prevented no matter how astute the reviewers and editors may be. And when fraud occurs, who's to blame: the editor, the reviewers, or the scientists who committed fraud?
In my opinion, the ultimate judge of integrity is the scientific community. Fraud is unlikely to be uncovered in the first place if it isn't published and made available to public scrutiny.
The failure of Science editors is not limited to their handling of the initial paper and the CLO's response to Sibley. For example, they also accepted an absurd letter quantifying the probability of IBWO extinction, while presumably rejecting a large number of useful comments.
"when fraud occurs, who's to blame: the editor, the reviewers, or the scientists who committed fraud?"
All of the above! The editors' role is particular important as they select relatively few papers deemed worthy of review from very many worthy candidates and also select the reviewers. Any number of biologists, not limited to ornithologists or PhDs, could have provided a critical review had they been consulted and respected.
"Are any others of you claiming that the editor of Science failed to submit the Cornell paper to peer reviewers?"
The editor of Science evidently failed to submit the Cornell paper to SUFFICIENTLY CRITICAL peer reviewers or disregarded critical reviews that he may have received.
"Fraud is unlikely to be uncovered in the first place if it isn't published and made available to public scrutiny."
Shouldn't tantalizing but improbable results be published on personal webpages, not in Science?
For the record, you're the one casting larger stones by invoking "fraud" when I did not do so.
"...they also accepted an absurd letter quantifying the probability of IBWO extinction, while presumably rejecting a large number of useful comments."
Do you actually know anybody who submitted comments for publication that were rejected, or is this mere conjecture?
"The editor of Science evidently failed to submit the Cornell paper to SUFFICIENTLY CRITICAL peer reviewers or disregarded critical reviews that he may have received."
Strong allegations here! Do you have inside information on how many referees provided peer review and whether or not any were critical of the study and recommended rejection?
I have no personal knowledge about how the editorial process of Science works, but because there are so many disciplines covered, I'm quite certain that Donald Kennedy, whose position is listed as "Editor-in-chief", had absolutely nothing whatsoever to do with the review process for the Cornell paper. Kennedy obviously is not an ornithologist, so why would you expect him to know who the most suitable reviewers are? If you were the Editor-in-chief of Science, would you know who the most suitable reviewers are for a manuscript reporting a new planet within our solar system?
Such broadly disciplined journals normally have associate editors, each with expertise in a specific discipline, and the Cornell manuscript was almost certainly sent to one such editor who completely handled the review process.
If you're going to hold the Editor-in-chief responsible for allowing publication of one paper that you and others believe should never have been published, just remember that Science publishes many articles in an issue every week of the year. Now let's be realistic: do you really believe the Editor-in-chief of Science even has the time to read every one of those articles, let alone find suitable reviewers for each manuscript submitted, understand what each article is about, and independently judge its scientific merits?
If you're on a witch's hunt or have a personal vendetta, you're probably barking up the wrong tree. You might have have better luck searching for the associate editor.
Incidentally, I am not connected in any way to Science, the review process of the Cornell paper, or to any of the searchers involved in the Ivory-bill searches.
Inquiring minds might wish to check out the following additional links to discussions about peer review, including problems with the process and possible solutions:
Nature's peer review debate (22 papers)
The case against peer review
Could peer review be wrong?
Peer review works well enough in most instances. The problem with the Science paper is that is was an EXCEPTIONAL paper: a front page story with a lot of spin. It had to be one of the top-selling issues of Science in 2005, if not the best-selling issue (anyone got the numbers?).
"Kennedy obviously is not an ornithologist, so why would you expect him to know who the most suitable reviewers are?"
Look, you don't have to be an ornithologist to know when a photo or video of a bird is garbage.
Besides the paper's authors, the real stinker in my view has always been this Prum fellow who was allegedly skeptical at first but then claimed to have been convinced by the allegedly "distinctive" knocking sounds (which were recorded separately from the crap video, of course).
On its face -- absent ANY convincing data to support these knocks as "distinctive" of anything -- this claim amounted to pure unadultered easily detectable bull hockey and any self-professed ornithologist or birdwatcher who was taken in for one second by such a bizarre statement should be deeply deeply ashamed of his or herself.
"Knocking sounds" in the goddxmn swamp prove an extinct bird is alive??? "Knocking sounds" from a bird whose "knocking sounds" in the wild were never systematically recorded and studied when it was alive??? Give me a freaking break already.
I still can't believe that any human tried to pull that stunt off, much less SUCCEEDED!!!!! It's a travesty.
But it gets worse!!!! Prum then **retracts** his inept "distinctive knock" statements (first they were "clear and convincing" proof, now the evidence is "refutable") leaving nothing but the crap video, and the paper remains UNRETRACTED.
I suspect that if this sort of thing occurred in biotech, in the sandbox where speculators and big pharma play, Kennedy would be forced to have the paper retracted and the scientists and reviewers responsible for the mess would be roundly chastised and made examples of.
But Kennedy has waaaaaaaay too much crow to eat here. Read Kennedy's editorial that Tom linked to: I must add a note about the personal excitement and pleasure this discovery has brought me.... The pleasure came because the involvement of the Cornell laboratory closed a circle for me. As a boy in the 1930s, I was a faithful follower of National Geographic accounts of Cornell expeditions to Louisiana to record and photograph these magnificent birds. I even wrote a fan letter to the expedition's leader, the pioneer Cornell ornithologist Arthur Allen...
Bingo. Fitzcrow et al. exploited the lifelong dreams of a 74 year old man.
If you're going to hold the Editor-in-chief responsible for allowing publication of one paper that you and others believe should never have been published, just remember that Science publishes many articles in an issue every week of the year.
Nice try. There are 52 cover stories. You think Kennedy didn't have a major role in deciding whether the IBWO story got on the cover? From Kennedy's editorial:
I must add a note about the personal excitement and pleasure this discovery has brought me. The sense of excitement began about 2 months ago when I received a somewhat cryptic e-mail from John Fitzpatrick, the head of Cornell's Laboratory of Ornithology (located in a nice piece of deciduous forest called Sapsucker Woods). Fitzpatrick's message inquired as to whether Science would be interested in reviewing a report confirming the persistence of a bird (I believe he said "iconic" bird) long thought to have been extinct. That was not a difficult code to break, and I got back to him in a New York second!
Kennedy can't unpoop the bed but he could have washed the sheets. It's getting a bit late to do that now, though.
The ironic money quote from Kennedy's editorial:
It is fortunate for science that it attracts people who may lack special training or higher degrees but have found the knowledge and confidence to know that they can do real science.
Best to let that quote just flap in the breeze ...
It's a case of Observer Expectancy Bias. They saw what they wanted to see. They published what they wanted to believe. All without proof.
A classic case that will be studied as an example of how bias can trip up even first rate reputations.
But Hillcrow, what is the deal with him? A second rate reputation with nowhere to go but up who throws it all away on wishful thinking. What a dolt!
"...the real stinker in my view has always been this Prum fellow.."
Surprise, surprise! Skeptics turning on one another. Amy, do you think there are any ornithologists who aren't morons?
Kennedy is further quoted in 29 November's USA Today on the value of being published in Science: "... enhanced reputation, visibility, position or cash rewards is sufficiently high that some may not adhere to the usual scientific standards." That should apply only to articles on intelligent design.
Surprise, surprise! Skeptics turning on one another.
Oh, please. I'm glad that Prum ultimately came to his senses.
But his falling asleep at the wheel (to be kind -- it was more like drunk driving) appears to be a major contributing act leading to the Science paper debacle. He had a job to do, he effed up, bigtime, and his mea culpa was insufficient (though it's not entirely clear what could have sufficed one the pecker was out of the bag).
"Do you actually know anybody who submitted comments for publication that were rejected, or is this mere conjecture?"
I do not have any means to ascertain who submitted comments on the IBWO that were rejected by Science, but do you really think that Sibley's excellent reply and Robert's pointless letter were the only comments received?
"Strong allegations here! Do you have inside information on how many referees provided peer review and whether or not any were critical of the study and recommended rejection?"
I have no inside information and therefore cannot rule out the two alternatives I presented: 1) sufficiently critical reviews were not obtained because the editors chose the wrong reviewers or 2) at least one critical review was obtained but it was ignored. In either case the editors failed.
"I have no personal knowledge about how the editorial process of Science works"
Then why are you so opinionated?
"because there are so many disciplines covered, I'm quite certain that Donald Kennedy, whose position is listed as "Editor-in-chief", had absolutely nothing whatsoever to do with the review process for the Cornell paper."
I didn't mention Kennedy's role in
the CLO paper because this has been extremely well documented on this blog and elsewhere and should be very familiar by now to all who are well informed about the IBWO controversy.
"Kennedy obviously is not an ornithologist, so why would you expect him to know who the most suitable reviewers are? If you were the Editor-in-chief of Science, would you know who the most suitable reviewers are for a manuscript reporting a new planet within our solar system?"
Had you bothered to read my comments with care, you would have noticed that my critiscims were not limited to Kennedy. You are correct that other more specialized editors are to blame as well. A paper on the solar system would be taken seriously by a top journal and would receive proper review. Systematic biology seems to be a low priority for top journals and its papers, in my opinion, often receive inappropriate reviews. For example, papers on historical science are often reviewed by people trained in the more prestigious experimental sciences, who tend to advocate gratuitous and inappropriate statistical methods and use of models.
"Such broadly disciplined journals normally have associate editors, each with expertise in a specific discipline, and the Cornell manuscript was almost certainly sent to one such editor who completely handled the review process."
Kennedy's direct role with respect to the CLO paper has been very well documented. Please read up on this controversy before offering your opinions.
"If you're going to hold the Editor-in-chief responsible for allowing publication of one paper that you and others believe should never have been published, just remember that Science publishes many articles in an issue every week of the year."
So the CLO paper was just an average paper of average interest to the editor-in-chief? Again, this comment shows you have not been following this controversy.
"If you're on a witch's hunt or have a personal vendetta, you're probably barking up the wrong tree. You might have have better luck searching for the associate editor."
Its not personal at all. I'm not even an ornithologist. I do care about the integrity of birding and about the handling of papers related to systematic biology by top journals and the media. I am particularly annoyed by Kennedy's attempt to shift the blame for bogus papers promoted by himself and other Science editors to scientists and the review process in general.
"Incidentally, I am not connected in any way to Science, the review process of the Cornell paper, or to any of the searchers involved in the Ivory-bill searches."
And you are apparently not well connected to this blog as your comments show an ignorance of fundamental issues elaborated at great length in previous excellent posts.
"...do you really think that Sibley's excellent reply and Robert's pointless letter were the only comments received?"
I have no idea. But I agree that Sibley et al.'s reply was excellent. I don't agree that Robert's letter was pointless. I personally enjoy reading a diversity of opinions. I like reading the diversity of opinions on this blog, even though my opinions are repeatedly disparaged.
"Then why are you so opinionated?"
I'm NOT! I don't have a strong opinion on whether the Ivory-bill is extinct or extant. Or whether the Cornell paper should have been published. My point was that Kennedy probably had nothing to do with the review process. His comments obviously reveal that he viewed the paper favorably, but I doubt he would have had it not been recommended for publication by an associate editor.
"Systematic biology seems to be a low priority for top journals and its papers, in my opinion, often receive inappropriate reviews."
Fair enough. I agree systematic biology is a lower priority than it should be--but the Cornell paper had nothing to do with systematics. I'm not sure I would agree with the inappropriateness of reviews. Editors rely on the opinions of only two or three reviewers, and it's amazing how often the comments and conclusions conflict with each other. Obvious mistakes are easily missed when only a few reviewers assess manuscripts. Furthermore, some reviewers are less suitable than others. Some develop a reputation for being obviously biased, or are hypercritical, or are hyperuncritical, and once editors learn who these reviewers are they stop asking them to review manuscripts.
I heartily agree with the recommendations of the panel that more reviews, such as five or ten, should be obtained for the most important manuscripts. But if all scientific journals demanded so many reviews of a manuscript, the scientific process would nearly grind to a halt. Reviewers are busy scientists who seldom drop everything to provide a quick review. It's not easy to be an editor and it's not easy to review papers. If you've ever been an editor before, or spent much time reviewing manuscripts, you would know why. It's a fallible human process. And no editor alive could possibly please everybody.
I have no clue who the reviewers of the Cornell paper were, but I suspect they were well published experts on the biology of woodpeckers. Because the Cornell paper was more of an identification issue than an ecological, behavioral or systematic issue, I heartily agree with the readers of this forum who believe that the reviews should have been solicited from experts in bird identification. I would guess that the associate editor who selected reviewers had never handled a bird identification manuscript before, and simply didn't even think about this. But I'm speculating here because I have no inside information. How certain are we that bird identification experts like Pete Dunn didn't review the manuscript? Some journals encourage authors to submit names of potential reviewers, although personally I don't think this should be done as it introduces potential bias in the review process. I have no idea whether Science encourages this, but it's quite possible that Fitzpatrick et al. recommended certain individuals as reviewers.
"And you are apparently not well connected to this blog as your comments show an ignorance of fundamental issues elaborated at great length in previous excellent posts."
You are entitled to your opinion, although I suspect Tom is in a much better position to judge how often I have viewed and commented on this blog.
"I am particularly annoyed by Kennedy's attempt to shift the blame for bogus papers promoted by himself and other Science editors to scientists and the review process in general."
IMO it's the authors--not editors--who are responsible for the contents of what is published. Have there been any cases of a journal editor who went to jail or was fired from a job for publishing a fraudulent manuscript?
"IMO it's the authors--not editors--who are responsible for the contents of what is published."
But the authors are not responsible for which papers are chosen for the cover of Science.
Post a Comment