[e2e] Is sanity in NS2?
erik.nordstrom at it.uu.se
Thu Sep 15 04:47:32 PDT 2005
Having been an e2e reader for some time (but infrequent poster) I think
I can contribute something to this thread.
Rik Wade wrote:
>On Thu, September 15, 2005 02:59, Detlef Bosau wrote:
>>The best, and basically the only _real_, validation is to compare
>>simulation results with reality.
I totally agree with the statement above. Comparison to reality is the
only way to do validation.
>And we all know how accurate and RFC-compliant real-world protocol
>implementations are. I've worked in a few large ISP/telcos and cannot
Reality consists of the world around us, and as far as I know it is not
defined by protocol implementations that run in operating systems.
Reality is what simulators should simulate, for example radio
propagation, not some invented artifact.
Having some experience from implementing two ad hoc routing protocols, I
can say that my primary usage of simulators such as ns is for
prototyping and testing. It is very convenient to test the logic of your
protocol implementation in a simulated environment because running
around, for example, in a corridor trying to set up multi-hop routes
with laptops is very tedious. I think we should realise that this is one
of the main applications of tools such as simulators. We should stop
using ns-2 for generating last minute results for those mandatory (but
seldom useful) graphs at the end of our research paper.
The other thing you can use simulators for is to achieve scalability.
Tests that are simply too large and complex to conduct in reality are a
clear application for simulation. However, if an experiment can be run
on a smaller scale on real hardware, I see no excuse to use a simulator
other than simply as
a tool for debugging and development.
That brings me to another point. Why do people abstract away more than
necessary in simulators? Why can't ns for example use real packet
headers and treat packets in the way a real OS does it? If people claim
the reason is for efficiency I can point out that the current way ns
deals with packets and headers is not exactly optimal. If ns would
implement or mimic real API:s (e.g., berkeley sockets) to the extent
possible, without sacrificing too much performance and scalability, it
would be much easier to develop protocol code that can run both in
reality and simulation. That way validation with reality is so much
easier and trust in the code (that it does the right thing) would
For my own implementations, I have gone to great lengths to make sure
that my code runs both in reality and in ns-2 (without emulation layers
or similar). This has made it possible to deploy the same code in real
testbeds and has allowed us to run real experiments and simulations in
parallel. This way we have discovered many false assumptions made in
other research that rely solely on simulations. It is also possible to
do trace based simulation instead of using the de-facto standard ns-2
randomly generated scenarios that have little resemblance to reality.
I think simulators are heavily over-utilized in our reasearch field,
simply because we are geeks that are too lazy to do real experiments. I
think we should put our efforts in developing good methodologies for
conducting repeatable real world experiments instead. Simulation should
only be used where and when it can be a complement.
More information about the end2end-interest