This July 2005 Memphis Flyer article has some details:
...Their paper announcing the find was completed and sent to Science April 6th.I just re-read the above article yet again. If you're interested in the Ivory-bill controversy, that particular article is just chock-full of interesting information.
...
Then, BAM-bam. On April 26th this year, with peer reviews of the paper still out, news of the bird began to hit the Internet. What was now called the Big Woods Conservation Partnership went into high gear -- biologists pulling an all-nighter editing the paper for Science, managers notifying agencies and others that a press conference they thought they had three weeks to plan for would be held April 28th in Washington, D.C., with Interior Secretary Gale Norton.
Science published the paper online the morning of the press conference...
Note: more discussion on peer review is here and here.
Posts
14 comments:
Couple of thoughts and some questions.
So Kennedy gets a cryptic email (on ??) and the paper is arrives april 6.
3 weeks later it appears online in connection with the announcement.
Do we know who reviewed the paper after it was submitted to science? Is it true that Jackson, the nations authority on IBWO was contacted on the same day this was announced and did not review the paper? If sibley heard a rumor from TNC MS how did Jackson not hear anything during that yearlong preparation for the announcement?
Publishing this in SCIENCE was clearly a PR tactic as much as anything else. If John Fitzpatrick had called called a press conference and announced it with the media present, that would have been fine enough to make headlines - but the decision to announce it with the authority of SCIENCE was a critical part of a campaign that the conservation framework indicated - it was the "pregnancy test" that showed that the team wasn't just "a little big pregnant".
How the journal SCIENCE behaves when confronted with an amazing "discovery" is an important part of the story. Major scientific journals do not have to "play ball" and unlike journalists they are not out trying to "break the big story" - for example when Stanley Pons and Martin Fleischmann -- announced they had made a breakthrough in the decades-long quest for cold fusion 1989, their claim was so "extra-ordinary" that the they had to withdraw a paper on their experiment that they had submitted to the British journal Nature because they were unwilling to respond to criticism by the journal's reviewers.
Jackson's comment, quoting Noss, that "reviewers have to rely on what the authors report". Isn't really clear to me (granted I haven't read Noss's paper). But clearly the roll of peer reviewers are supposed to evaluate the evidence. The key attribute of a "peer" is that they are an independent qualified expert who is who is researching and publishing work in the same field. The idea that a peer contains their evaluation to "what the authors report" isn't right because the peers are supposed to be experts and draw on their expertise. As the editor of SCIENCE Kennedy knows what "peer review" is and his main job at the Journal is to insure that it happens.
Here is Kennedy (see Kennedy Science 26 March 2004:Vol. 303. no. 5666, p. 1945) speaking broadly on his views of "how science should happen":
"A problem is defined so that the right measures for dealing with it can be selected; relevant information is gathered and then analyzed, so that meaningful relationships can be uncovered and understood; and tentative conclusions are reached and then tested against possible alternatives. Finally, perhaps most important, critical review is sought."
Kennedy defines "peer review" at Science, as a process where "qualifications and limitations on scientific conclusions are usually added to the text at the insistence of reviewers" .... In his editorial which which he is arguing that intelligence agencies impose a "reverse peer review" process on analyists, wherein they delete "qualifying language and caveats" so that conclusions were strengthened" and he calls this "arguing from the desired conclusion rather than from the data". (see Kennedy Science 26 March 2004:Vol. 303. no. 5666, p. 1945) http://www.sciencemag.org/cgi/content/summary/303/5666/1945? also cited here: http://www.senseaboutscience.org.uk/PDF/peerReview.pdf
Did Kennedy participate in a "reverse peer review process" in the case of the IBWO? Or did he "add qualifications and limitations to the paper at the insistence of his reviewers?"
How did Kennedy do his job - especially the part he describes as "the most important part"? This is not asking "does he beat his wife" - this is simply asking for details about how he did his job?
Here are links to the article by Kennedy with his views on the role of peer review ...
Intelligence Science: Reverse Peer Review?
Donald Kennedy
Editor-in-Chief
"While some believe passing the peer-review process is a certification of validity, those who study that process often hold a far more skeptical view. Drummond Rennie, deputy editor of Journal of the American Medical Association is an organizer of the International Congress on Peer Review and Biomedical Publication, which has been held every four years since 1986. [1]. We still don't know how well the peer-review process works, he says, although one thing is clear: "There seems to be no study too fragmented, no hypothesis too trivial, no literature too biased or too egotistical, no design too warped, no methodology too bungled, no presentation of results too inaccurate, too obscure, and too contradictory, no analysis too self-serving, no argument too circular, no conclusions too trifling or too unjustified, and no grammar and syntax too offensive for a paper to end up in print."
Peer review
From Wikipedia, the free encyclopedia.
The whole entry is very informative.
Another reading of this: "the reviews were still out," meaning the reviewers had not yet returned their comments to the editor. I can easily imagine an urgent phone call or e-mail exchange between the editor and the reviewers (who would certainly understand the need for urgency), and between the editor and the authors, to get the review comments in, the publish-or-not-decision made, and the last minute revisions of the manuscript done in 48 hours. I don't see anything that indicates the review process was circumvented, just hurried along.
There is nothing at all unusual about a paper being published in a major journal with conclusions that are controversial and not accepted by large numbers of readers. No single paper is the last word on anything. I do not understand why some of you are determined to identify and punish some imagined major failing in the process here. In fact, this is exactly how the process works. Now can you just settle down and read the articles coming up next week, which are the logical next step in the discussion?
You must be joking to suggest there is any relevence of the Kennedy editorial to the Ivorybill. Exactly who would have been responsible for the "reverse peer review" in which ordinary reservations and caveats were deleted at the insistence of external reviewers? The Secret Shadow Ornithology Underground? The National Avian Disinformation Council?
for a paper to end up in print."
right ... but NOT in the most respected journals ... those journals are HARD to get into. Those journals have TOUGH peer review.
Exactly who would have been responsible for the "reverse peer review"
I don't think there was a deliberate conspiracy ... there was a failure to due dilligence in a situation where an extra ordinary claim was being made at the same time that powerful and high profile events were taking place. The latter trumped the former ...
The real "peer-review process" begins AFTER publication, and always has.
well that isn't what Donald Kennedy says happens at SCIENCE, he says in his editorial that, peer review IS THE MOST IMPORTANT part of the process ... and he goes onto describe how this means adding caveats and limitations ... but then when the heat is on and the boys high above lake cayuga have the interior departments attention diverted from their busy day leasing oil and gas at bargain basement prices and revising the EIS on ANWR ... and Gail Nortons schedule can't be changed and the threat of hordes of birders crashing the big woods means that peer review can wait ... besides all the peers that were needed AUTHORED the paper and each one decided that 33.3 was a perched bird ... no reverse peer review here. These are not the droids you are looking for ...
Here is the other reason why Kennedy's editorial about "intelligence science" is so relevant - it isn't that he is saying that the anaylists are actually threatened or point blank "told" to take out qualifying or limiting information. What Kennedy is saying is that when science is pressed against power and policy - which it clearly was in this case - we know it was from the limited record that we have - people know what is being expected of them and they comply - it is easier to comply - their jobs are at stake - they have to play nice with others - they are part of a group and they have to behave in ways that keep them in that group or they risk getting isolated.
Kennedy is saying that at SCIENCE - this independent peer review PREVENTS this from happening. Only when the heat is on and important people are scheduling press conferences and "iconic birds" are in danger of going extinct ... again ... we can not act as we normally do. We must "push" the review - we must "speed" the process.
and that is how you get "reverse peer review" ...
I spoke from reality and experience. In the publication process, a paper is reviewed by 3-4 reviewers and the editor. After publication it is read by hundreds, thousands, or sometimes even more scientists and interested lay people, over many years. It is discussed in graduate seminars and in the pub over beers. It is argued about at meetings, in restaurants, and over the internet. Alternative interpretations, innovative ideas, contradictory data that the authors and reviewers did not yet know about, followup studies, all are brought together in one big noisy (and often rancorous) discussion. This is where new studies actually are put through their paces. And of course this is exactly what is happening with the Arkansas Ivorybill studies. Pre-publication peer-review is only the first step.
Hombre, you've published a paper in SCIENCE? Maybe you have. Take a blury video of it and post it here so I can read it.
Look I cede the letter of your point. But in this case the entire data in this paper is the video - this is no "new evidence" no additional data - we have figure S1 of 33.3 which is totally wrong, we have branch stubs etc ... in short we have "reverse peer review" - important people that decided that this was the science of the thing ... and now we are headed into the reality of the upcomming sibley paper.
I didn't write the Kennedy editorial about "the most important" part of the process at SCIENCE ... he did.
You can argue all you want about how crap gets published and then later in the pubs we get some real peer review. In this case all we need to do is get some dis-intersted people to watch the video in slow motion and we find that this paper wouldn't get published in a million years if it wasn't submitted by important powerful people who have important dealings to attend to ... so if you could just hurry up and ink your rubber stamp we'd be grateful.
It is a baroque excrescence founded on shakey evidence.
Not in Science but in other flagship journals of the one-notch-below Science, Nature, and JAMA tier.
Only problem is, plenty of competent people who have watched the video in slow motion do not agree ith your assessment. This may piss you off and frustrate you, but your opinion is just that, your opinion, not an Objective Truth that Everyone Would See if They Were Not All So Stupid.
For a bunch of people who are critical of others for believing "experts" just because they are "experts," you spend a lot of time quoting "experts" with the implication we should believe them because they are "experts." Why are "your experts" any different than "their experts?"
because I'm quoting the expert on what he thinks his job is an comparing that to what he actually does ... why is this hard to follow.
You are saying that I am employing a double standard by holding Donald Kennedy to account on Donald Kennedy's own views of peer review ... Is there a more authoritative source on what Donald Kennedy said than quoting Donald Kennedy?
Post a Comment