[e2e] was double blind, now reproduceable results

[Mark Allman]

> Jon - there are a large class of experimental results that can never
> be reproduced again (measurements of network at a point in time).

Let me push back for a moment ...

  + One can argue that in the strictest sense Internet measurements can
    never be reproduced.  There are too many variables in play to ever
    even hope to exactly gather the same data twice.  (In labs one can
    obtain more reproducability, of course.  But, not live, production
    networks.)

  + That said, this argues for sharing of raw measurement results so
    that various researchers can verify the analysis, at least.  Or,
    compare various schemes for measuring some piece of information on
    the same dataset (which is not necessarily about reproducability,
    but certainly within the realm of conducting good science).

  + And, just because we cannot reproduce numbers doesn't mean we cannot
    gather a subsequent set of data and reproduce the insights gained.
    I.e., "reproducable" can be defined in more than one way.  

    For instance, we have a few claimed invariants in networks (ideally,
    we should have more), such as that FTP data transfers follow some
    particular distribution.  We have evidence that the distribution is
    the same across networks, even if the mean of the distribution is
    different.  So, someone can reproduce the insight, but maybe not the
    exact same exact numbers that were found at some other point in the
    network (and in time).  Or, alternatively, if someone found that the
    distribution didn't hold then the claimed invariant wouldn't be.
    That's valueable input, too.

So, I think that I agree with you that we cannot strictly reproduce
Internet measurements.  However, I think there are things we can do to
cope and to improve our overall methodology to make it more scientific
(carefully not stringing the words "scientific method" together, you'll
note :-).

allman


--
Mark Allman -- ICIR -- http://www.icir.org/mallman/



-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 186 bytes
Desc: not available
Url : http://www.postel.org/pipermail/end2end-interest/attachments/20040519/72260abf/attachment.bin