[e2e] Is sanity in NS2?
detlef.bosau at web.de
Thu Sep 15 08:29:54 PDT 2005
RJ Atkinson wrote:
> On Sep 14, 2005, at 15:58, Mark Allman wrote:
>> Nobody said it was perfect. But, to assert that there is nothing done
>> in this area of validation or regression or quality control or whatever
>> you want to call it is just wrong, IMHO. That is all I was trying to
> I think a large part of the issue is that many of the "externally
> contributed" NS models have had little or no credible validation,
> unlike the solid models done by some others (e.g. Floyd). This is
This what I wanted to call "core" yesterday: The stable part (perhaps
the _major_ part) in the NS in contrast to many hacks which are not
validated and even not known to the public. E.g. my PTE stuff. Nobody in
the community ever read it. No one can say whether results using this
code are certain or not.
And one remark to the validation runs in the NS. As far as I see, their
purpose ist not primarily to validate the modules but the compilation.
It is validated whether the compiled and installed NS yields the
expected results for a huge number of tests.
> then compounded by those ns users who don't validate their particular
> simulation scenario before believing its results.
> Unfortunately, submitted papers that both use unvalidated models
> and use them in an unvalidated scenario are all too common. Even
And, again refering to my own work, it´s hardly possible to do this in
some other way.
Of course we could discuss whether I first write my code, then ask
someone (btw: whom?) whether I may contribute the code, then wait for
some people to approve it and then, when I get older, loosing my hair,
many years from know (The Beatles) I perhaps get the chance to have a
> more scary to me, some authors who submit papers to IEEE periodicals
> don't seem to understand what validation means when the reviewer
> asks them to edit the paper to discuss how the simulation model
> and scenario were validated so the reader can have some basis
> for believing the paper.
And this is something you have to learn and where you need assistance.
That´s why I complain that nobody teached me there. I know this is
annoying to many of you. However, it is somewhat unfortunate to do this
all from scratch and the only feedback you have is the one rejected
paper every year. Personally, I have only a rough idea, how I could
validate my own code. I can describe the PTE algorithm. I can run a
number of test scenarios. I can compare the output in the logfile with
the intended behaviour of the algorithm. So, I could say: Every scenario
yields the correct result. But is this sufficient?
Unfortunately, the typical attitude is that: If the paper is accepted,
your teacher was correct in his orders he has given to you and you
followed them corectly. If the paper was rejected, your teacher was
.....to you and you did not follow them correctly.
I have no teacher, so the former does not apply.
Hoewever: I still do not fully undertand the standards, by which a paper
is accepted or rejected.
>> Researchers are responsible for ensuring that their experiments
>> are sound. Doesn't matter if you're using opnet or ns2 or two
>> Solaris boxes and a Cisco router or a pencil and paper.
> Quite so. Unfortunately, this reviewer sees a lot more poorly
> designed simulations being written up and submitted to periodicals
> than poorly designed laboratory experiments (or, for that matter,
> papers with conclusions based on incorrect maths).
> I think the root problem here is that many universities
> are encouraging students to run simulations without teaching
> students about how to use simulation properly and without
> sufficient supervision that the simulation was used properly.
> This is unfortunate.
And this is exactly what I complained about.
Moreover, it´s an inacceptalbe waste of time and money. How long does it
take for a gifted student to get _really_ familiar with the NS and the
basics of simulations? At the risk that I have to ask my provider for
additional disk space for my mailbox and that I will have learned at
least two hundered new swearwords by the end of this day: I don´t
believe that even an excellent student will really _understand_ what he
is doning in less than one year. Assumed, the student is engaged in the
NS _full_ _time_. I do know some PhD students engaged in "projects"
(this are these funny things to make your department "visible", in real
life, it´s called "advertising") the major part of their time. And even
with the best effort, they don´t achieve useful simulations. They
achieve pure botchery. And they can´t even help it because the day has
only 24 hours.
Yesterday we talked about Tcl here. I did not fully understand the
arguments in favour of Tcl yet. However, I´m not fully convinced that
the mixture of Tcl, OTcl, Tclcl and C++ is really that good idea here.
Even this mixture is extremely error prone (I once asked a colleague who
was thought to be a brillant expert for the simple reason, why a
Trace/Deque is Trace/Deque and he smattered a lot of bullshit but _NO_
_ONE_ could explain me what I already guessed that time, that for the
split object the initialization code for a Trace object is executed only
for the reason of the notation "Trace/Deque". This appears plausible.
But I have looked for this in numerous documents for nearly a year to
make it sure, not full time of course, otherwise I could have read the
whole Tcl interpreter). It is extremely difficult to understand.
And it´s a nuisance to debug.
I spent many hours in the debugger to get problems in the C code fixed
and it´s always nasty when you have to step over a "tcl.evaf()" and have
to treat it as a black box. Particularly when you run in problems with
dangling pointers or null pointer exceptions or something similar.
Perhpas, I don´t have the right intuition for the big picture here, but
for me the NS is a toolbox, a framework. And if I had a faster computer
and not have to wait hours for the link run, I would appreciate doing
the whole thing in C++. Perhaps, I´m not an experienced programmer, but
I´m not convinced of having the need for an embedded language. (And I´m
old fashioned as well, I strongly prefer C++ over Java, but this is my
personal attitude. One single reason: Java does not have pointers. And I
don´t know a language, where a similar depth of knowledge and experience
with pointers is required as in Java, only to give yourself the illusion
you would not use pointers. In consequence, I´ve never seen that much
null pointer exceptions and pointer errors as in Java elsewhere. We
should not avoid pointers, if we can´t, but learn to use them.)
When I said, it takes a year to understand the ns, at least nine month
of this is spent for understanding the mixed language stuff. And the
bindings. And the slit objects. And what is initialized where and when.
Other people make a baby in this time.
So, finally a student often simply wastes a year only to understand the
ns. And this should be drastically shortened because no one has the time
and the money for that.
The ns itself is basically a framework and other frameworks can be
understood in a shorter time too.
The important difference to other frameworks is that in the ns, you are
often required to work at the framework itself. It is not really
possible to take the C++ framework as a black box and do some Tcl
scripting. In fact, I would prefer to do that little Tcl sripting I have
done in C++ as well.
Who complained recently, his students did forget the binding?
God in Heaven! If we did not do this mixed language stuff, we would not
have these pitfalls!
I think, this is simply cumbersome and nasty.
> I'm not sure what is available in the line of standard textbooks,
> but if one doesn't yet exist maybe there would be room for someone
> to write up a textbook on when and how to simulate communications
> systems -- or maybe even a text on good experimental design
> more generally.
Mail: detlef.bosau at web.de
Mobile: +49 172 681 9937
More information about the end2end-interest