[e2e] was double blind, now reproduceable results
jonathan at dsg.stanford.edu
Thu May 20 13:26:24 PDT 2004
In message <40ACD7E8.3060004 at isi.edu>Joe Touch writes
>>>I don't disagree with making all the data available, and that is almost
>>>always possible, even where reproducibility is hard.
>> With the notable exception of whole-payload packet traces. If you can
>> get permission to collect them at all, its well-nigh impossible to
>> redistribute them. For obvious and excellent reasons.
>It ought to be possible in those cases - if not required - to provide
>reasonable access to run second-party tests to confirm results.
Oh, absolutely. I've been on both sides of that fence, and that sounds
reasonable to me. Much of the data I published was from researchers
(such as Vern) who kindly ran my tools ``under escrow'', against their
own traces. I never saw the raw traces. The tools were validated
against synthetic test data, and against my own traces.
But getting permission to collect whole-packet traces may well impose
conditions more restrictive than you, or I, or Jon might wish from
solely a resarch perspective.
Stanford stipulated I could not disclose *anything* about my
whole-packet traces, without first clearing it with the appropriate
campus authorities. For example, they got veto rights over early
drafts of my papers, to ensure I didnt disclose anything untoward.
That same veto would apply if I were to run third-party tools over
those traces. Getting permission to run second-party tests might
be a difficult, time-consuming task.
Traces collected from ISPs may (if the ISP is also a telco) also have
to pass the `statistical measurement only' regulatory requirements for
telcos; but I have no idea what those are, these days.
More information about the end2end-interest