[e2e] was double blind, now reproduceable results
David P. Reed
dpreed at reed.com
Thu May 20 06:03:50 PDT 2004
At 07:56 PM 5/19/2004, Bob Braden wrote:
>This is often hard to do, and I cannot deny that in many cases we are
>mightily challenged to come up with an appropriate measure X. However,
>I would suggest that this is the ideal towards which we should strive.
Reproducing results is not always the appropriate goal. When it is, I
agree wholeheartedly. When it cannot be, there are other measures of
experimental and empirical quality. Sharing one's tools and techniques,
methods and materials, data collected, etc. are equally important.
>We should not just say, "It's too hard to do."
I have the impression you are trying to claim that there are only two
approaches, one hard and one lazy. Nothing I have said implies that we
should just say "it's too hard". My point is that the work of scientific
review is hard, and frivolous attempts to reduce it to a simplistic
cookbook rule such as "all experiments must be reproducible before we allow
them to be reported in print or in conferences" would have ruled out 90% of
the scientific breakthrough experiments I most admire. I would resist
such a rule, and though I share Jon's disappointment with the quality of
many papers in our field, this idea that there is a simple way to solve the
reviewer's task could be characterized as lazy.
The job of producing a refereed paper involves hard work on both
sides. When a paper does not provide enough information to judge its
contribution, that's the reviewer's job to point out. If there are good
reasons why the paper should be published, even if the results may not be
reproducible, the job of science is to come to terms with those reasons,
discuss better methods, suggest other tests, etc. But suppressing
publication based on mere matters of conformance to a mythical pure
Scientific Method seems absurd and extreme.
Science is a self-correcting process, and what we must call for is a much
harder standard than the trivial one, which is the standard of intellectual
integrity. That is my creed, and that is not laziness.
More information about the end2end-interest