[e2e] was double blind, now reproduceable results
mallman at icir.org
Wed May 19 09:19:46 PDT 2004
> Jon - there are a large class of experimental results that can never
> be reproduced again (measurements of network at a point in time).
Let me push back for a moment ...
+ One can argue that in the strictest sense Internet measurements can
never be reproduced. There are too many variables in play to ever
even hope to exactly gather the same data twice. (In labs one can
obtain more reproducability, of course. But, not live, production
+ That said, this argues for sharing of raw measurement results so
that various researchers can verify the analysis, at least. Or,
compare various schemes for measuring some piece of information on
the same dataset (which is not necessarily about reproducability,
but certainly within the realm of conducting good science).
+ And, just because we cannot reproduce numbers doesn't mean we cannot
gather a subsequent set of data and reproduce the insights gained.
I.e., "reproducable" can be defined in more than one way.
For instance, we have a few claimed invariants in networks (ideally,
we should have more), such as that FTP data transfers follow some
particular distribution. We have evidence that the distribution is
the same across networks, even if the mean of the distribution is
different. So, someone can reproduce the insight, but maybe not the
exact same exact numbers that were found at some other point in the
network (and in time). Or, alternatively, if someone found that the
distribution didn't hold then the claimed invariant wouldn't be.
That's valueable input, too.
So, I think that I agree with you that we cannot strictly reproduce
Internet measurements. However, I think there are things we can do to
cope and to improve our overall methodology to make it more scientific
(carefully not stringing the words "scientific method" together, you'll
Mark Allman -- ICIR -- http://www.icir.org/mallman/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 186 bytes
Desc: not available
Url : http://www.postel.org/pipermail/end2end-interest/attachments/20040519/72260abf/attachment.bin
More information about the end2end-interest