[e2e] Is sanity in NS2?
detlef.bosau at web.de
Wed Sep 14 06:52:13 PDT 2005
May I first, once again, refer to the quote on Wesley Eddy´s Homepage:
"Mathematicians stand on each other's shoulders while computer
scientists stand on each other's toes." -- R. W. Hamming
Jon Crowcroft wrote:
> I think the open source/commons implication is part of the problem, but also
> the success disasster that NS became is partly because they made it too easy
> (analogous to socket programming)
> so anyone who could wield a bit of tcl or modify an existing bit of C++,
> thought they could build an interesting new simulation, and didnt have to acquire
> any disciple to do so - some people (typically hard put upon PhD students either
> strongly self motivated and sef-disciplined) managed to do some useful things, but
> an awful lot are those hotmail/yahoo email folks you allude to,
> (I have no idea where they are really from - are they using such
> addresses because they are afraid their university will catch them plagiarising,
> or are they blocked in china?) and they do more harm than good.
It´s a difficult issue.
First of all: I don´t think that something must be complicated in order
to be good. I don´t believe in the "Quality by obscuracy" principle.
(I must admit: I never submitted something to the IOCCC.)
I´m a strong believer of clarity.
Unfortunately, the clarity of the NS code is somewhat different in
different parts of the NS.
Concerning my own NS code: This mostly consists of evil hacks, as I´m a
bad programer. If someone wants to have a look at it (at his own risk of
course and I take no responsiblity for any consequences such as brain
damage, loss of any kind of programming expertise etc.) he can have a
look at my homepage: http://www.detlef-bosau.de/ns-2.26-cvs.2005.03.15.tgz
It´s not documented in any way. Don´t look for PTE. It´s not that I´m
not interested in PTE anly longer. It´s because much of the ideas came
to me when my chair got broken and so, I had to go to the IKEA furniture
shop (is this correct? I did not find it in my dictionary.) to buy a new
one. Therefore, it´s called "ikea". However, I could not use this name
for copyright reasons, so I renamed my approach to "Path Tail Emulation".
Basically, open source programs may work fine. Take Linux as an example.
However, contributing to a large program requires knowledge and self
disciipline. (I lack both of it....) Particularly for the NS, it
requires a very deep and very broad knowledge of programming techniques.
And I´ld better not talk about my first steps into the NS2 world. You
would roll on the floor laughing.
The problem was: There was no one to teach me.
So, basically I believe into open software and open source programs.
However, this requires two necessary things:
1.: An excellent maintenance of the program. I think, Linux Torvalds,
Lars Wirzenius etc. spent at least that much time for maintenance and
code integration than all contributors together spend in programming.
(It´s a guess, I did not talk to one of them two.)
2.: An excellent programming education. That´s a matter of course for
computer scientists. And if there is any school with a different
attitude, it´s clear what must change there. A computer scientist is
bascially an engineer, the computer is his engine and programming is his
hammer and his screwdriver.
Of course, some programmers are more gifted than others. However, a
computer scientist is an _excellent_ programmer anyway - or he is no
> anecdote - about 6-7 years ago we went through a lot of papers on tcp/aqm that
> had used ns (on the order of 100 papers) and tried to find the ns code to see if
> we could reproduce the results - around 50% of the papers appeared to be either bogus (when you
> got th code it didnt work with the alleged version of ns) or appeard to have graphs generated
> from a single run of ns (thre first one). quite a few were based o na (short lived) bug in the
> congestion control code in ns, (EPFL and others confirmed this bug - i cannot remember the exact detail,
> but it meant that the otcl and c++ variables weren't bound, so if you varied one, the other one didnt -
> this meant that the results for cwnd were basically random - papers were publised on this too).
The _serious_ issue behind this anecdote is: Who has ever validated NS2
code? How much papers are based upon "irreproducible simulations"?
In consequence, I think: Simulation proves anything and nothing. It´s
basically what Keshav mentions on his homepage. It´s the _rationale_
that must be convincing. Not one simulation.
Simulation results may result from a "lucky choice" in your scenario or
from sum "lucky bugs".
You might kill me for that example. But when I think of the TCP-AP
approach, Mobihoc 2005, ElRakabawy, Klemm, Lindemann, I simply don´t buy
it. I contacted the authors and asked for the rationale of this
apporach. I don´t got one. The authors refered to the simulations.
First: I don´t have the source.
Second: I don´t have the time to analyze each simulation artefact.
This paper "proves" its approach with "a pletora" (i.e. two) scenarios
and no rationale. I don´t understand, why this paper was accepted. It´s
an embarrassment for the whole community.
I conducted a similar approach years ago - and I had to learn it was
wrong. It was a bitter lesson. You can imagine my aggravation.
> before ns, there was XSim, and real and the other descendants of the MIT simulator, which had a similar series of
> problems, although were sufficiently hard to use that most people ended up doing pretty much clean 100%
> rewrites of the relevant part of the code for their thesis work, and ended up
> i) understanding it
> ii) validating it
> iii) getting lots of meaningful results out
> iv) abandoning it completely upon graduating...
The pity is iv.
That´s why I wrote the qote above.
It´s hardly possible to reinvent the wheel each time you do a research.
And it´s a validation matter.
Why did I ask for AWND and flow control?
Because I want to compare my own PTE approach with a "dumb spoofing
approach" (look at Mark Allman´s work on that one).
It´s basically no problem to introduce a "dynamic AWND" into the NS2.
It´s the problem to have the TCP sender react properly particularly on
zero windows. In addition, a split gateway or a PEP using PTE has to
obey some rules for the proper definition of the AWND.
It´s not that I don´t want to do this myself (although I would be glad
to use existing code here) or that I was not able to do this myself. Of
couse I´m able to.
It´s a matter of validation.
And I´m afraid that a submitted paper would be rejected, because the
reveiwer simply does not believe me or the scenario is not "complex"
enough or whatever excuse is used.
To be honest: I´m still not sure, wheter I will use a simulation in a paper.
> Your comments about opnet also apply to matlab and other propietary and quite good (or very good, respectively)
> tools, that are supported so long as the relevant supervisrs ask for funds - they are relatively expensive for
The problem with proprietary code is that I cannot validate it. I cannot
read the code. I have to "believe" it.
> UK university budgets so are typically default-off - which is ludicrous really given the time it can save a student
> and supervisor and quality improvement in results....
> we attempted to do an NS re-write in java (jns and jvs) which worked pretty well and several folks picked up on it,
> but exactly the same thing started to happen to us (at UCL) so I abandoned the program of work (though others
> picked up on it and it is on sourceforge i believe (not due to us) but i dont know how active it is -
> one specific thing that attempted to do was to make it "proper programming" to use the simulator,
> so the tecchnical bar was set a bit higher than just throwing some modified tcl at something and hoping...
I basically don´t believe that making thinks more complicated would be
In my opinion, the main issue is discipline and proper maintenance.
Particularly, it´s a bad idea to ad arbitrary "contributors" on the NS2
homepage without any consideration if
1. the contributed code is _relevant_,
2. the contributed code is _correct_,
3. the contributed code conforms to certain standards which consider the
quality of code as well as some style rules.
AFAIK, in commercial companies you have programmes who _intergrate_
contributed code. Who rework the programming style, who care for proper
> so trying to counter the point i made above by setting an implict "qualifying exam" to drive jns -
> but i think its evidence that at least in the case of educational software, open source may not be a good idea...
> supporting your point again.
At least for me, I´m an unemployed person, there is no alternative to
open source software.
> agree with your points on support mail lists etc too
> in fact i think this is the nearly first time i agree 100% with what you wrote!
We all are happy for both of you :-)
Refering to Hamming once again: Feel perfectly free to continue to do so :-)
(And yes, someone told me that a book on the NS2 would be helpful....
However, at the moment I don´t dare to start a project like this. First,
I don´t know wheter I´m able to do so. Second, there are other things
more important, e.g. to find a job in Germany at the age of 42. Perhaps,
I should apply as a test person for coffins. However, this is not
adequate for a person interested in packet switching, as undertakers
usually prefer to do line switching.)
(And please, don´t sent me comments on my programming style. I know, if
I ever wrote a book on the NS2, I would look for a co-author who is a
Mail: detlef.bosau at web.de
Mobile: +49 172 681 9937
More information about the end2end-interest