[e2e] Is sanity in NS2?
floyd at icir.org
Fri Sep 16 10:05:59 PDT 2005
>Unfortunately, submitted papers that both use unvalidated models
>and use them in an unvalidated scenario are all too common.
The newly-created Transport Modeling Research Group (TMRG) is working
towards creating Best Current Practice sets of simulation and testbed
scenarios for the evaluation of congestion control mechanisms.
These are not intended to be complete benchmarks, but just "you
should have tested with these scenarios also". Towards that end,
two documents have been written so far, "Metrics for the Evaluation
of Congestion Control Mechanisms" and "Tools for the Evaluation of
Simulation and Testbed Scenarios".
Any feedback and contributions would be welcome, on the tmrg mailing
list. Abstracts are below, for those who are interested.
Metrics for the Evaluation of Congestion Control Mechanisms:
This document discusses the metrics to be considered in an
evaluation of new or modified congestion control mechanisms for the
Internet. This document is intended to be the first in a series of
documents aimed at improving the models that we use in the
evaluation of transport protocols.
Tools for the Evaluation of Simulation and Testbed Scenarios:
This document describes tools for the evaluation of simulation and
testbed scenarios used in research on Internet congestion control
mechanisms. We believe that research in congestion control
mechanisms has been seriously hampered by the lack of good models
underpinning analysis, simulation, and testbed experiments, and that
tools for the evaluation of simulation and testbed scenarios can
help in the construction of better scenarios, based on better
underlying models. One use of the tools described in this document
is in comparing key characteristics of test scenarios with known
characteristics from the diverse and ever-changing real world.
Tools characterizing the aggregate traffic on a link include the
distribution of per-packet round-trip times, the distribution of
packet sequence numbers, and the like. Tools characterizing end-to-
end paths include drop rates as a function of packet size and of
burst size, the synchronization ratio between two end-to-end TCP
flows, and the like. For each characteristic, we describe what
aspects of the scenario determine this characteristic, how the
characteristic can affect the results of simulations and experiments
for the evaluation of congestion control mechanisms, and what is
known about this characteristic in the real world. We also explain
why the use of such tools can add considerable power to our
understanding and evaluation of simulation and testbed scenarios.
More information about the end2end-interest