From detlef.bosau at web.de Tue Jul 16 07:13:43 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Tue, 16 Jul 2013 16:13:43 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? Message-ID: <51E55517.9080303@web.de> I'm still to understand this decision. The more I think about it, the more I fear, that although the decision to abandon hop by hop flow control (which is still in Vint Cerf's catenet paper but is abandoned in RFC 791) may be the most seminal generator for PhD theses in the history of science, on the one hand, however I'm not quite sure whether this was one of the largest fallacies ever. Actually, the VJCC kludge turned to be quite successful up to now, however VJ himself admits some basic limitations in his well known talk "A New Way to look at Networking". Does anyone happen to know, whether this was decision for a concrete reason and the rational behind it? Or did this "simply happen"? -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From jnc at mercury.lcs.mit.edu Tue Jul 16 08:14:37 2013 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Tue, 16 Jul 2013 11:14:37 -0400 (EDT) Subject: [e2e] Why was hop by hop flow control eventually abandonded? Message-ID: <20130716151437.CB9E618C0CF@mercury.lcs.mit.edu> > From: Detlef Bosau > the decision to abandon hop by hop flow control > ... > Does anyone happen to know, whether this was decision for a concrete > reason and the rational behind it? Or did this "simply happen"? Probably the internet-history list is a better place to ask this? I don't know for sure, but having arrived on the scene shortly thereafter, and knowing intimately what packet switches were like then, my _guess_ is that it had to do with state. It seems to me that to be able to do hop-by-hop flow control, you have to have some state in the switches, yes? (I can't see a way to do it without.) And in a 'smallish' network, like the ARPANET, it's reasonable to have that state. But when you're talking about (potentially) a much bigger network, as the Internet even then was planned to be, the amount of state potentially needed would quite likely have been too much for the switches of the day. Noel From jnc at mercury.lcs.mit.edu Tue Jul 16 08:29:17 2013 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Tue, 16 Jul 2013 11:29:17 -0400 (EDT) Subject: [e2e] Why was hop by hop flow control eventually abandonded? Message-ID: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> PS: It might also have been the extension of more general design principle to all areas - i.e. getting rid of state in the switches (be it for reliability, flow control, or anything else), and moving it to the ends; that way, everything (reliability, flow control, etc, etc) was done on an end-end basis. And maybe it was both? All just speculation on my part, though, I'm afraid.. Noel From faber at isi.edu Tue Jul 16 10:43:40 2013 From: faber at isi.edu (Ted Faber) Date: Tue, 16 Jul 2013 10:43:40 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <20130716151437.CB9E618C0CF@mercury.lcs.mit.edu> References: <20130716151437.CB9E618C0CF@mercury.lcs.mit.edu> Message-ID: <51E5864C.2040403@isi.edu> On 07/16/2013 08:14, Noel Chiappa wrote: > > From: Detlef Bosau > > > the decision to abandon hop by hop flow control > > ... > > Does anyone happen to know, whether this was decision for a concrete > > reason and the rational behind it? Or did this "simply happen"? > > Probably the internet-history list is a better place to ask this? > > I don't know for sure, but having arrived on the scene shortly thereafter, > and knowing intimately what packet switches were like then, my _guess_ > is that it had to do with state. > > It seems to me that to be able to do hop-by-hop flow control, you have to > have some state in the switches, yes? (I can't see a way to do it without.) One of the many interesting ideas in Dina Katabi's XCP work is that she distributes the per-flow/per-switch data into the packets of the flow. It's not an obvious idea (in my dumb opinion) and reading about XCP with that in mind is worthwhile. I think at its core congestion control is an end-to-end problem, not so much because of state in elements (though that does matter) but diversity of elements. End-to-end congestion control makes TCP over avian carriers possible. There are many elements in a network path that can become a flow's bottleneck. There's no way to guarantee that the thing that's bottlenecked is also smart enough (or honest enough) to participate in hop-by-hop congestion control. An end-to-end system can assess the path and make decisions without cooperation of internal elements (though an end-to-end system can make use of cooperation that it can get). No matter what's slowing your pigeons down, an end-to-end congestion control will react to it. Eventually. :-) -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 261 bytes Desc: OpenPGP digital signature Url : http://mailman.postel.org/pipermail/end2end-interest/attachments/20130716/2963a6af/signature.bin From touch at isi.edu Tue Jul 16 11:32:33 2013 From: touch at isi.edu (Joe Touch) Date: Tue, 16 Jul 2013 11:32:33 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> Message-ID: <51E591C1.9010407@isi.edu> On 7/16/2013 8:29 AM, Noel Chiappa wrote: > PS: It might also have been the extension of more general design principle to > all areas - i.e. getting rid of state in the switches (be it for reliability, > flow control, or anything else), and moving it to the ends; that way, > everything (reliability, flow control, etc, etc) was done on an end-end > basis. > > And maybe it was both? All just speculation on my part, though, I'm afraid. Maybe also the E2E principle - i.e., that even HBH flow doesn't relieve the endpoints from participating in E2E flow. As Noel noted, though, Internet-history may be a useful place to ask this. Joe From detlef.bosau at web.de Tue Jul 16 12:35:38 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Tue, 16 Jul 2013 21:35:38 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <51E5864C.2040403@isi.edu> References: <20130716151437.CB9E618C0CF@mercury.lcs.mit.edu> <51E5864C.2040403@isi.edu> Message-ID: <51E5A08A.3080304@web.de> Am 16.07.2013 19:43, schrieb Ted Faber: > > One of the many interesting ideas in Dina Katabi's XCP work is that she > distributes the per-flow/per-switch data into the packets of the flow. > It's not an obvious idea (in my dumb opinion) and reading about XCP with > that in mind is worthwhile. > > I think at its core congestion control is an end-to-end problem, not so > much because of state in elements (though that does matter) but > diversity of elements. End-to-end congestion control makes TCP over > avian carriers possible. Hm. This exhibits an interesting twist of the loss differentiation problem ;-) However, I doubt TCP over Avian Carriers would allow a "coddle" implementation which would be acceptable for PETA ;-) > > There are many elements in a network path that can become a flow's > bottleneck. http://www.youtube.com/watch?v=yhuMLpdnOjY *SCNR* > There's no way to guarantee that the thing that's > bottlenecked is also smart enough (or honest enough) to participate in > hop-by-hop congestion control. An end-to-end system can assess the path > and make decisions without cooperation of internal elements (though an > end-to-end system can make use of cooperation that it can get). No > matter what's slowing your pigeons down, an end-to-end congestion > control will react to it. Eventually. :-) Oh yeah.... Hopefully, no PETA members become aware of this discussion.... > > -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From jnc at mercury.lcs.mit.edu Tue Jul 16 13:18:29 2013 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Tue, 16 Jul 2013 16:18:29 -0400 (EDT) Subject: [e2e] Why was hop by hop flow control eventually abandonded? Message-ID: <20130716201829.EC56018C0CB@mercury.lcs.mit.edu> > From: Ted Faber > One of the many interesting ideas in Dina Katabi's XCP work is that she > distributes the per-flow/per-switch data into the packets of the flow. Goodness, I'd forgotten all about that - been many years since I looked at it! Clearly I need to re-read it, to swap the ideas back in... > It's not an obvious idea (in my dumb opinion) and reading about XCP > with that in mind is worthwhile. Concur. I was definitely awed when I read about it - 'very clever' is a significant understatement. Noel From braden at isi.edu Tue Jul 16 14:24:18 2013 From: braden at isi.edu (Bob Braden) Date: Tue, 16 Jul 2013 14:24:18 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <51E591C1.9010407@isi.edu> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> Message-ID: <51E5BA02.9030501@isi.edu> I believe that hop-by-hop flow control only works with per-flow state in the routers (see X.25 for example). Once the decision was made that the routers should be stateless, end-to-end flow control was the only option. That is what the end-to-end principle is all about. Bob Braden From m.handley at cs.ucl.ac.uk Tue Jul 16 15:47:53 2013 From: m.handley at cs.ucl.ac.uk (Mark Handley) Date: Tue, 16 Jul 2013 15:47:53 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <51E5BA02.9030501@isi.edu> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> Message-ID: <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> It's before my time, but I'd always assumed it was also influenced by the NVP work, which would not have wanted hop-by-hop flow control. Mark On Tue, Jul 16, 2013, at 02:24 PM, Bob Braden wrote: > I believe that hop-by-hop flow control only works with per-flow state in > the routers (see X.25 for example). Once the decision was made that the > routers should be stateless, end-to-end flow control was the only > option. That is what the end-to-end principle is all about. > > Bob Braden > From Jon.Crowcroft at cl.cam.ac.uk Wed Jul 17 01:36:08 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Wed, 17 Jul 2013 09:36:08 +0100 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> Message-ID: so most systems in the world do hop by hop as well as end to end e.g. transportation systems (traffic lights, stacks of planes, semaphores to control trains entry/exit from track sections etc etc) power systems (elec ant gas) water systems (valves etc) eco-systems (food chains/feast/famine etc) political systems (you switch from feudal to democract, btut you still have periodic elections - you switch from city states to countries, but still have border and immigration/emigration controls...:) so why do we think comms should be different? in fact, I suggest we do some flow control inside nets - its called traffic engineering and operates on aggregates - when we do multipath routing, we also select a modest number of routes (obviously more than 1 but a lot less than actually would give connectivity or even some additional capacity).... so i think the design decision to throw out all hop by hop flow control was probably an error (not a disastrous one: as many people have pointed out, it simplified early router design a lot to be completely stateless - but you don't need to keep per-5-tuple based e2e state to do hop by hop flow control if its on aggregates, right?) In missive <1374014873.23736.140661256474709.49B0CF90 at webmail.messagingengine.c om>, Mark Handley typed: >>It's before my time, but I'd always assumed it was also influenced by >>the NVP work, which would not have wanted hop-by-hop flow control. >> >>Mark >> >>On Tue, Jul 16, 2013, at 02:24 PM, Bob Braden wrote: >>> I believe that hop-by-hop flow control only works with per-flow state in >>> the routers (see X.25 for example). Once the decision was made that the >>> routers should be stateless, end-to-end flow control was the only >>> option. That is what the end-to-end principle is all about. >>> >>> Bob Braden >>> cheers jon From michawe at ifi.uio.no Wed Jul 17 02:13:26 2013 From: michawe at ifi.uio.no (Michael Welzl) Date: Wed, 17 Jul 2013 11:13:26 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> Message-ID: <1D9ADEAB-16E0-444A-8C97-3D1262F350E7@ifi.uio.no> i agree like i haven't agreed to anything in a long time! in fact i agree with every single word jon wrote here Sent from my iPhone On 17. juli 2013, at 10:36, Jon Crowcroft wrote: > so most systems in the world do hop by hop as well as end to end > e.g. > transportation systems (traffic lights, stacks of planes, semaphores to > control trains entry/exit from track sections etc etc) > power systems (elec ant gas) > water systems (valves etc) > eco-systems (food chains/feast/famine etc) > political systems (you switch from feudal to democract, btut you still > have periodic elections - you switch from city states to countries, > but still have border and immigration/emigration controls...:) > > so why do we think comms should be different? > > in fact, I suggest we do some flow control inside nets - its called > traffic engineering and operates on aggregates - when we do > multipath routing, we also select a modest number of routes (obviously > more than 1 but a lot less than actually would give connectivity or > even some additional capacity).... > > so i think the design decision to throw out all hop by hop flow > control was probably an error (not a disastrous one: as many people > have pointed out, it simplified early router design a lot to be > completely stateless - but you don't need to keep per-5-tuple based > e2e state to do hop by hop flow control if its on aggregates, right?) > > In missive <1374014873.23736.140661256474709.49B0CF90 at webmail.messagingengine.c > om>, Mark Handley typed: > >>> It's before my time, but I'd always assumed it was also influenced by >>> the NVP work, which would not have wanted hop-by-hop flow control. >>> >>> Mark >>> >>> On Tue, Jul 16, 2013, at 02:24 PM, Bob Braden wrote: >>>> I believe that hop-by-hop flow control only works with per-flow state in >>>> the routers (see X.25 for example). Once the decision was made that the >>>> routers should be stateless, end-to-end flow control was the only >>>> option. That is what the end-to-end principle is all about. >>>> >>>> Bob Braden >>>> > > cheers > > jon > From Jon.Crowcroft at cl.cam.ac.uk Wed Jul 17 03:07:34 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Wed, 17 Jul 2013 11:07:34 +0100 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <1D9ADEAB-16E0-444A-8C97-3D1262F350E7@ifi.uio.no> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> <1D9ADEAB-16E0-444A-8C97-3D1262F350E7@ifi.uio.no> Message-ID: I think the notion of "smearing out traffic" over multiple alternate routes (ie. resource pooling) at a slightly faster timescale than traffic engineering, is actually part of Davies' original vision for packet switching (see isarythmic flow control).... for me, since it isn't end2end, it is sort of hop by hop....but it isn't "per e2e flow" either I don't see a problem with it - in fact re-ecn/congestion exposure is surely a new variant on that old old old idea...? In missive <1D9ADEAB-16E0-444A-8C97-3D1262F350E7 at ifi.uio.no>, Michael Welzl typ ed: >>i agree like i haven't agreed to anything in a long time! in fact i agree with every single word jon wrote here >> >>Sent from my iPhone >> >>On 17. juli 2013, at 10:36, Jon Crowcroft wrote: >> >>> so most systems in the world do hop by hop as well as end to end >>> e.g. >>> transportation systems (traffic lights, stacks of planes, semaphores to >>> control trains entry/exit from track sections etc etc) >>> power systems (elec ant gas) >>> water systems (valves etc) >>> eco-systems (food chains/feast/famine etc) >>> political systems (you switch from feudal to democract, btut you still >>> have periodic elections - you switch from city states to countries, >>> but still have border and immigration/emigration controls...:) >>> >>> so why do we think comms should be different? >>> >>> in fact, I suggest we do some flow control inside nets - its called >>> traffic engineering and operates on aggregates - when we do >>> multipath routing, we also select a modest number of routes (obviously >>> more than 1 but a lot less than actually would give connectivity or >>> even some additional capacity).... >>> >>> so i think the design decision to throw out all hop by hop flow >>> control was probably an error (not a disastrous one: as many people >>> have pointed out, it simplified early router design a lot to be >>> completely stateless - but you don't need to keep per-5-tuple based >>> e2e state to do hop by hop flow control if its on aggregates, right?) >>> >>> In missive <1374014873.23736.140661256474709.49B0CF90 at webmail.messagingengine.c >>> om>, Mark Handley typed: >>> >>>>> It's before my time, but I'd always assumed it was also influenced by >>>>> the NVP work, which would not have wanted hop-by-hop flow control. >>>>> >>>>> Mark >>>>> >>>>> On Tue, Jul 16, 2013, at 02:24 PM, Bob Braden wrote: >>>>>> I believe that hop-by-hop flow control only works with per-flow state in >>>>>> the routers (see X.25 for example). Once the decision was made that the >>>>>> routers should be stateless, end-to-end flow control was the only >>>>>> option. That is what the end-to-end principle is all about. >>>>>> >>>>>> Bob Braden >>>>>> >>> >>> cheers >>> >>> jon >>> >> cheers jon From loa at pi.nu Wed Jul 17 03:24:28 2013 From: loa at pi.nu (Loa Andersson) Date: Wed, 17 Jul 2013 12:24:28 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <1D9ADEAB-16E0-444A-8C97-3D1262F350E7@ifi.uio.no> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> <1D9ADEAB-16E0-444A-8C97-3D1262F350E7@ifi.uio.no> Message-ID: <51E670DC.4030705@pi.nu> Michael and Jon, While I agree that a combination of end to end and hop by hop is a good idea for communication systems, I do don't buy the idea that it must be so *just* because it has been done so in other system. You may get good ideas from how it is done elsewhere, but it is not always the right solution /Loa On 2013-07-17 11:13, Michael Welzl wrote: > i agree like i haven't agreed to anything in a long time! in fact i agree with every single word jon wrote here > > Sent from my iPhone > > On 17. juli 2013, at 10:36, Jon Crowcroft wrote: > >> so most systems in the world do hop by hop as well as end to end >> e.g. >> transportation systems (traffic lights, stacks of planes, semaphores to >> control trains entry/exit from track sections etc etc) >> power systems (elec ant gas) >> water systems (valves etc) >> eco-systems (food chains/feast/famine etc) >> political systems (you switch from feudal to democract, btut you still >> have periodic elections - you switch from city states to countries, >> but still have border and immigration/emigration controls...:) >> >> so why do we think comms should be different? >> >> in fact, I suggest we do some flow control inside nets - its called >> traffic engineering and operates on aggregates - when we do >> multipath routing, we also select a modest number of routes (obviously >> more than 1 but a lot less than actually would give connectivity or >> even some additional capacity).... >> >> so i think the design decision to throw out all hop by hop flow >> control was probably an error (not a disastrous one: as many people >> have pointed out, it simplified early router design a lot to be >> completely stateless - but you don't need to keep per-5-tuple based >> e2e state to do hop by hop flow control if its on aggregates, right?) >> >> In missive <1374014873.23736.140661256474709.49B0CF90 at webmail.messagingengine.c >> om>, Mark Handley typed: >> >>>> It's before my time, but I'd always assumed it was also influenced by >>>> the NVP work, which would not have wanted hop-by-hop flow control. >>>> >>>> Mark >>>> >>>> On Tue, Jul 16, 2013, at 02:24 PM, Bob Braden wrote: >>>>> I believe that hop-by-hop flow control only works with per-flow state in >>>>> the routers (see X.25 for example). Once the decision was made that the >>>>> routers should be stateless, end-to-end flow control was the only >>>>> option. That is what the end-to-end principle is all about. >>>>> >>>>> Bob Braden >>>>> >> >> cheers >> >> jon >> > -- Loa Andersson email: loa at mail01.huawei.com Senior MPLS Expert loa at pi.nu Huawei Technologies (consultant) phone: +46 739 81 21 64 From detlef.bosau at web.de Wed Jul 17 03:52:57 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Wed, 17 Jul 2013 12:52:57 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <51E5BA02.9030501@isi.edu> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> Message-ID: <51E67789.40003@web.de> Am 16.07.2013 23:24, schrieb Bob Braden: > I believe that hop-by-hop flow control only works with per-flow state > in the routers (see X.25 for example). Once the decision was made that > the routers should be stateless, end-to-end flow control was the only > option. That is what the end-to-end principle is all about. > > Bob Braden > Is this really the "end-to-end principle"? To my understanding, Jerry Salzer indicated that it is an important discussion whether a certain mechanism is placed on the communication end points or somewhere along the path. The Salzer paper is not primarily a "principle", it is more a "culture of thorough thinking". -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From m.handley at cs.ucl.ac.uk Wed Jul 17 03:52:56 2013 From: m.handley at cs.ucl.ac.uk (Mark Handley) Date: Wed, 17 Jul 2013 03:52:56 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> Message-ID: <1374058376.18708.140661256658777.03B34654@webmail.messagingengine.com> On Wed, Jul 17, 2013, at 01:36 AM, Jon Crowcroft wrote: > so most systems in the world do hop by hop as well as end to end > e.g. > transportation systems (traffic lights, stacks of planes, semaphores to > control trains entry/exit from track sections etc etc) > power systems (elec ant gas) > water systems (valves etc) > eco-systems (food chains/feast/famine etc) > political systems (you switch from feudal to democract, btut you still > have periodic elections - you switch from city states to countries, > but still have border and immigration/emigration controls...:) > > so why do we think comms should be different? Because it's not politically acceptable to drop planes when air traffic control gets congested? Dropping a packet that has not yet consumed any resource at the bottleneck does not cost anything. In fact it does cost latency, but it seems to me that keeping latency low is the only strong reason for wanting any form of in-network flow control. - Mark From jnc at mercury.lcs.mit.edu Wed Jul 17 05:36:58 2013 From: jnc at mercury.lcs.mit.edu (Noel Chiappa) Date: Wed, 17 Jul 2013 08:36:58 -0400 (EDT) Subject: [e2e] Why was hop by hop flow control eventually abandonded? Message-ID: <20130717123658.5AA2018C0D5@mercury.lcs.mit.edu> > From: Jon Crowcroft > so i think the design decision to throw out all hop by hop flow control > was probably an error I'm not sure I agree with that.. but i) I need to ponder for a while, and ii) I want to make sure I understand what you mean by 'hop by hop flow control', because I'm not sure we're all using the same meaning. Everyone seems to be assuming there are two choices: hop-by-hop and end-end. However, depending on _exactly what one means by those terms_, I think there may be four, or so, choices on how to signal congestion. First, end-end - which I assume means 'the destination tells the source things that the source uses to do rate control'. One would tend to assume that this is the oldest meaning of the TCP window - i.e. information about the amount of buffering available at the destination. But does this also include information about the state of the network?/ Next, at the other end of the spectrum, hop-by-hop. When _I_ hear this term, it means to me something like X.25/X.75 congestion control: e.g. you have a channel across 5 networks A,B..E, and if D starts to congest, it tells that to C, and then C tells that to B, and eventually word gets back to the source host. Then, there is something in the middle, which I will call 'direct', which is 'the place which is experiencing congestion tells the source about that congestion directly'. There are two sub-flavours of this: 'direct-direct', in which the congestion location does communicate directly with the source (think ICMP Source Quench). The second is 'piggy-backed direct', in which, to avoid sending a separate control packet, the information is added to packets heading to the destination, where it is turned around and sent back to the source. (I guess a better short term for this is 'echoed-direct'. Now, perhaps people are including that one in 'end-end' when they think of 'only two', but I think it's clear that in some sense it's fundamentally different from 'true' end-end: the _source_ of the congestion indication is _in the network_, not at the end - and the only reason it comes from the destination host is that that's the most efficient way to get it back. With that taxonomy in hand, we can first examine the history of congestion control in the Internet. Initially, we tried to use a mixture of end-end (for buffering in the destination), and direct-direct (ICMP SQ). The latter part didn't work so well (for reasons I suspect we still don't understand very well - and I think it might work better in today's networks than it did across the ARPANet - I would _love_ to see someone experiment with it, and try and see if it could in fact work). So then we moved onto pure end-end. That didn't work so well either. So then Van added some echoed-direct, using _only the information already available to him_ - i.e. information about dropped packets (the simplest congestion signal). Then later on, we started tinkering with more sophisticated echoed-direct, first with ECN, and then with XCP (which turns out, now that I refresh my memory, to also be in this class - it's just that the information sent back is much richer, semantically, than just 'yes/no'). And now back to your original point - leaving out hop-by-hop was bad. If you really meant HbH as defined above, I don't agree. The reason is that I suspect it's slower (not to mention more complex) than simply sending a signal straight back to the original source: either direct-direct, or echoed-direct. I suspect that even the latter is faster, in real time, than having network D congest, and tell network C about it, etc, etc. Reaction? Noel From barney at databus.com Wed Jul 17 08:04:16 2013 From: barney at databus.com (Barney Wolff) Date: Wed, 17 Jul 2013 11:04:16 -0400 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> Message-ID: <20130717150416.GA85043@pit.databus.com> What are we talking about when we say "hop-by-hop" - the classic "don't send me anything on this link because I've got no buffer to receive it" or "don't send me anything on this flow because something downstream is pushing back"? I thought these days bufferbloat is the problem, not buffer depletion. What's an aggregate? On Wed, Jul 17, 2013 at 09:36:08AM +0100, Jon Crowcroft wrote: > so i think the design decision to throw out all hop by hop flow > control was probably an error (not a disastrous one: as many people > have pointed out, it simplified early router design a lot to be > completely stateless - but you don't need to keep per-5-tuple based > e2e state to do hop by hop flow control if its on aggregates, right?) From michawe at ifi.uio.no Wed Jul 17 09:25:49 2013 From: michawe at ifi.uio.no (Michael Welzl) Date: Wed, 17 Jul 2013 18:25:49 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <20130717150416.GA85043@pit.databus.com> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> <20130717150416.GA85043@pit.databus.com> Message-ID: i'd say bufferbloat is only one part of the problem. the other is and for a long time has been underutilization. it's long been the problem of congestion control - empty networks! but we're all waiting for stuff to happen as fast as possible... meaningful aggregates are, for example, several flows that share the same bottleneck along their individual e2e path - they could be treated "together", rather than each probing the network on its own in one way or another On Jul 17, 2013, at 5:04 PM, Barney Wolff wrote: > What are we talking about when we say "hop-by-hop" - the classic "don't > send me anything on this link because I've got no buffer to receive it" > or "don't send me anything on this flow because something downstream is > pushing back"? I thought these days bufferbloat is the problem, not buffer > depletion. What's an aggregate? > > On Wed, Jul 17, 2013 at 09:36:08AM +0100, Jon Crowcroft wrote: >> so i think the design decision to throw out all hop by hop flow >> control was probably an error (not a disastrous one: as many people >> have pointed out, it simplified early router design a lot to be >> completely stateless - but you don't need to keep per-5-tuple based >> e2e state to do hop by hop flow control if its on aggregates, right?) From detlef.bosau at web.de Wed Jul 17 11:25:29 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Wed, 17 Jul 2013 20:25:29 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <1374065440.897723396@apps.rackspace.com> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> <1374065440.897723396@apps.rackspace.com> Message-ID: <51E6E199.3090204@web.de> Am 17.07.2013 14:50, schrieb dpreed at reed.com: > The end-to-end argument (which I developed and Jerry decided to write down) is quite specific: if you can't do the function entirely in the network, focus on how to do it at the end-to-end level. "The network" sounds a bit confusing to me: "The network" sounds like a more or less closed "block". And particularly VJCC has a typical "black box approach" to the network. > > And the *congestion control* function is one that cannot achieve entirely in the network. The reason is simple: congestion management requires the endpoints to change their behavior in one or more ways. Absolutely. Particularly the reason for congestion is offering more load to the network than the network can carry. And this misbehaviour is to be changed. When I take a white box view on networks, I may understand congestion control as a sequence, or mesh, of concatenated flow controls. (To me, flow control makes sure that not more data is sent than the receiver can accept, congestion control makes sure that no more data is sent than links and nodes along the path can store.) Perhaps, Jon is a bit on the white box side here? > > By conflating "flow control" (which has to do with orderly sequencing and local error management) and congestion control (which has to do with *resource allocation* among unanticipatable bursty exogenous demand) you don't further understanding of the problem. > > Air traffic control involves both congestion and flow control. Congestion control is a negotiation solely between endpoints of "trips", while flow control is about keeping airplanes separated in the air so they don't collide or get forced to land. > > The original separation of these concepts was the design choice in TCP that made the system scalable and robust with respect to topology and dynamics of the underlying network. I'm not quite sure whether flow and congestion control are intentionally separated in TCP. To me, VJCC appears as a (actually quite reasonable) kludge to solve the problem with an extreme black box view of the network. Today, however, we see the limits of this network view on the one hand - and have the technical possibilities for a white box view on the other. > > To do congestion control, one needs to have inputs. > > Perhaps the controversial thing in TCP you might want to focus on is not the falsely confused congestion and flow control functions, but the idea that packet drops are integral to providing input to the distributed negotiation among endpoints. They are an important means for this purpose. However, others exist. (I was thinking about IP over Avian Carriers quite some time yesterday. And I had a first glance at the XCP work as well. And my impression was that VJCC is neither a good idea for congestion control in carrier pigeons nor in LFN networks. It works quite reasonable for a number of technologies and a black box approach, however, for particular technologies there may be smarter approaches.) > It need not have been so - a simple alternative would have been to record in each packet how much excess queueing delay it encountered during its end-to-end path, and develop a distributed control algorithm that min-maxed a figure of merit consisting of global end-to-end queuing delay "excess". Sounds like some kind of banker's algorithm ;-) > > Strategies such as source routing (for path selection), dynamic route-table optimization (for traffic engineering dynamically), codec changes (to change the underlying needed bitrate, or to take advantage of FEC to recover lost packets), low-rate or rateless erasure coding, and even "network coding" all need *end-to-end* congestion control. > And why shouldn't these strategies take advantage from local information on intermediate network nodes? -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From touch at isi.edu Wed Jul 17 13:14:42 2013 From: touch at isi.edu (Joe Touch) Date: Wed, 17 Jul 2013 13:14:42 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> Message-ID: <51E6FB32.4060101@isi.edu> On 7/17/2013 1:36 AM, Jon Crowcroft wrote: > so most systems in the world do hop by hop as well as end to end > e.g. > transportation systems (traffic lights, stacks of planes, semaphores to > control trains entry/exit from track sections etc etc) > power systems (elec ant gas) > water systems (valves etc) > eco-systems (food chains/feast/famine etc) > political systems (you switch from feudal to democract, btut you still > have periodic elections - you switch from city states to countries, > but still have border and immigration/emigration controls...:) > > so why do we think comms should be different? Router-router links can also have flow control, e.g., on ethernet using pause frames. However, not all links have that sort of back-pressure capability; many are treated as fixed-capacity simplex channels. A lot of the examples above have that sort of backpressure built-in, or we have engineered it because we don't want (or can't afford) to lose the 'message' (as was noted, dropping a car or a plane would be bad). IP networks don't require that sort of engineering - which makes them simpler to implement, but also means they have in-network drops in response to congestion. Joe From touch at isi.edu Wed Jul 17 13:16:44 2013 From: touch at isi.edu (Joe Touch) Date: Wed, 17 Jul 2013 13:16:44 -0700 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <51E5BA02.9030501@isi.edu> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> Message-ID: <51E6FBAC.5020907@isi.edu> On 7/16/2013 2:24 PM, Bob Braden wrote: > I believe that hop-by-hop flow control only works with per-flow state in > the routers (see X.25 for example). Once the decision was made that the > routers should be stateless, end-to-end flow control was the only > option. That is what the end-to-end principle is all about. > > Bob Braden IMO, E2E says that E2E can't replace HBH, but not that HBH can't or shouldn't exist. HBH can be important to optimizing the E2E, so long as it operates over the correct relative timescales (HBH being faster than E2E, so you don't have the two fighting each other). Joe From michawe at ifi.uio.no Wed Jul 17 15:10:47 2013 From: michawe at ifi.uio.no (Michael Welzl) Date: Thu, 18 Jul 2013 00:10:47 +0200 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <1374095819.748619071@apps.rackspace.com> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> <20130717150416.GA85043@pit.databus.com> <1374095819.748619071@apps.rackspace.com> Message-ID: <02324060-5418-40B3-BA42-4D6DA8339C2C@ifi.uio.no> On Jul 17, 2013, at 11:16 PM, dpreed at reed.com wrote: > I would dispute underutilization being a problem. Most engineers confronted with an extremely bursty load (where the burstiness must be supported to get the kind of interaction responses desired - i.e. very low latency on complex web requests, etc.) would understand and *accept* the need for the network to be *on the average* very lightly loaded. > > That is, even on the creaky old Bell System digital phone network, the actual capacity was over provisioned by a large factor - a factor of 10 or so - because of *Mother's Day* phenomena. "even on" => but this was connection oriented. "especially on" seems more likely to me. Mother's Day phenomena won't be so bad when things are more dynamic, that's the whole point. FWIW you could fill a net with LEDBAT-like traffic all the time? this might not be "green", but it won't cause a problem on Mother's Day. > > If you aim for full utilization, even if you are a government sanctioned *monopoly*, you will serve your customers badly if you try to fill up your entire fixed capacity to > 50% on the average. > > Of course, in the minds of academics, this is intolerable! But academics rarely bother to think about the real world of engineering, preferring to invent problems (like using all available capacity *on the average*) that give them well-formed mathematical optimization problems. Yes, I said that I consider underutilization a problem, but I didn't say what you say here. > > Are you serving your students well? Are you serving your research community well? Not bloody likely. I beg to differ :-) Academics should be forward-looking, no? Cheers, Michael From yangbaohua at gmail.com Wed Jul 17 18:14:34 2013 From: yangbaohua at gmail.com (Baohua Yang) Date: Thu, 18 Jul 2013 09:14:34 +0800 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <20130716151437.CB9E618C0CF@mercury.lcs.mit.edu> References: <20130716151437.CB9E618C0CF@mercury.lcs.mit.edu> Message-ID: agree. switch is hard to support compliacted states and logics, due to the performance requirement and the hardware limitation. however, it would be a worthy idea in SDN, where more flexibility can be provided by the stateful controller. On 7/16/13, Noel Chiappa wrote: > > From: Detlef Bosau > > > the decision to abandon hop by hop flow control > > ... > > Does anyone happen to know, whether this was decision for a concrete > > reason and the rational behind it? Or did this "simply happen"? > > Probably the internet-history list is a better place to ask this? > > I don't know for sure, but having arrived on the scene shortly thereafter, > and knowing intimately what packet switches were like then, my _guess_ > is that it had to do with state. > > It seems to me that to be able to do hop-by-hop flow control, you have to > have some state in the switches, yes? (I can't see a way to do it without.) > > And in a 'smallish' network, like the ARPANET, it's reasonable to have that > state. But when you're talking about (potentially) a much bigger network, > as > the Internet even then was planned to be, the amount of state potentially > needed would quite likely have been too much for the switches of the day. > > Noel > -- Sent from my mobile device Best wishes! Baohua From ggm at apnic.net Wed Jul 17 19:09:08 2013 From: ggm at apnic.net (George Michaelson) Date: Thu, 18 Jul 2013 12:09:08 +1000 Subject: [e2e] Why was hop by hop flow control eventually abandonded? In-Reply-To: <8847d71d2dcc4406a1b581c85d028ad5@IAMDA2.org.apnic.net> References: <20130716152917.4387D18C0CF@mercury.lcs.mit.edu> <51E591C1.9010407@isi.edu> <51E5BA02.9030501@isi.edu> <1374014873.23736.140661256474709.49B0CF90@webmail.messagingengine.com> <20130717150416.GA85043@pit.databus.com> <1374095819.748619071@apps.rackspace.com> <8847d71d2dcc4406a1b581c85d028ad5@IAMDA2.org.apnic.net> Message-ID: There is something slightly awry for me seeing a bunch of people discuss this like we're history of science students reviewing the royal society in 1688. Did I drop into an alternative universe? Donald Davies aside, the players aren't all "dropped off the twig" yet. I personally think their input has some primacy for me in the "why" side of things, looking backwards. Maybe I'm missing the posts which say "I thought that..." or "I decided that..." Speaking for myself, as a aficionado of history of science, The post-hoc rationalisations over why things happen often don't really reflect what was going on at the time. People say "because of " but it isn't always that simple. Sometimes you're just playing and your idea gets traction. Sometimes you're exploring and you luck into something and it becomes bigger than you thought. (mostly, I think I find myself down rat holes which are irrelevant to the mainstream. Like my time coding ISO Transport Classes 0-3 and misunderstanding that Transport Class 4 was TCP by another name) I think that hbh happens right now. As Jon said, its on aggregates. But it undoubtedly (oh, don't you love it when people make assertions which invite the 'hang on, I doubt it' response.. so this is in my opinion..) informs e2e by the outcomes on delay and drop. Two systems operating on the same outcome fully decoupled invites interactions which are chaotic. They might couple, or they might not. For other unrelated reasons I found myself reviewing the TCP over X.25 back history and a purdue [1] study noted that a 2 packet window in X.25 SVC and PVC had a really bad effect on TCP. As did establishment cost on e2e delay for those crucial first few packets. If you pace your behaviour based on what happens at the start, this intermixing of dumb and smart can play bad. [1] http://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1574&context=cstech On Thu, Jul 18, 2013 at 8:10 AM, Michael Welzl wrote: > > On Jul 17, 2013, at 11:16 PM, dpreed at reed.com wrote: > > > I would dispute underutilization being a problem. Most engineers > confronted with an extremely bursty load (where the burstiness must be > supported to get the kind of interaction responses desired - i.e. very low > latency on complex web requests, etc.) would understand and *accept* the > need for the network to be *on the average* very lightly loaded. > > > > That is, even on the creaky old Bell System digital phone network, the > actual capacity was over provisioned by a large factor - a factor of 10 or > so - because of *Mother's Day* phenomena. > > "even on" => but this was connection oriented. "especially on" seems more > likely to me. Mother's Day phenomena won't be so bad when things are more > dynamic, that's the whole point. FWIW you could fill a net with LEDBAT-like > traffic all the time? this might not be "green", but it won't cause a > problem on Mother's Day. > > > > > If you aim for full utilization, even if you are a government sanctioned > *monopoly*, you will serve your customers badly if you try to fill up your > entire fixed capacity to > 50% on the average. > > > > Of course, in the minds of academics, this is intolerable! But > academics rarely bother to think about the real world of engineering, > preferring to invent problems (like using all available capacity *on the > average*) that give them well-formed mathematical optimization problems. > > Yes, I said that I consider underutilization a problem, but I didn't say > what you say here. > > > > > Are you serving your students well? Are you serving your research > community well? Not bloody likely. > > I beg to differ :-) Academics should be forward-looking, no? > > Cheers, > Michael > > > From keithw at mit.edu Fri Jul 19 09:36:53 2013 From: keithw at mit.edu (Keith Winstein) Date: Fri, 19 Jul 2013 12:36:53 -0400 Subject: [e2e] TCP ex Machina Message-ID: Hello, I'd be grateful for the list's feedback re: our new paper on computer-generated end-to-end congestion-control schemes. The paper, FAQ, code, and steps for reproducing the results are up at: http://web.mit.edu/remy We find, essentially, that computers can create better congestion-control protocols than the best human-designed schemes to date -- if given a clear statement of the prior assumptions that the protocol is entitled to have about the network going in, and a well-stated objective. (Which could be throughput fairness, throughput over delay, average flow completion time...) Our program, Remy, has generated end-to-end congestion-control schemes that get more throughput, less queueing delay, and more fairness than TCP Cubic, Compound TCP, TCP Vegas, etc. Relevant to this list, we were surprised to find that these computer-generated schemes (running over "dumb" DropTail gateways in simulation) can even outperform schemes like XCP and Cubic-over-sfqCoDel, which depend on intelligence inside the network, and in XCP's case depend on prior knowledge of the exact wire speed at the gateway. It seems like an unexpected victory for end-to-end algorithms that we're working to explain. To be clear, the work is academic research, in ns-2, and has a long ways to go (including a real implementation, and understanding HOW these crazy computer-generated schemes work) before we would think anybody would want to deploy it in a datacenter under the control of one entity, much less over the Internet But in our view, making the transport layer be simply a function of (a) what the lower layers are doing and (b) what the users and applications want is a way to get a more evolvable Internet, and avoid the problem where network infrastructure does unfortunate things to try to mask loss and out-of-order delivery from TCP. We are presenting this next month at SIGCOMM so will be grateful for any early feedback the list can supply. Regards, and thank you, Keith From wes at mti-systems.com Sun Jul 21 06:53:59 2013 From: wes at mti-systems.com (Wesley Eddy) Date: Sun, 21 Jul 2013 09:53:59 -0400 Subject: [e2e] forming an IETF AQM working group Message-ID: <51EBE7F7.2020701@mti-systems.com> We are trying to start a working group in the IETF to focus on Active Queue Management (AQM) and packet scheduling or fair queuing algorithms. There is a mailing list setup at: https://www.ietf.org/mailman/listinfo/aqm Anyone interested is free and welcomed to join the discussion. We especially want to have people with different perspectives participating, including hardware and software developers for different types of systems, network operators, academics, etc. There will also be a Birds of a Feather (BoF) meeting at the IETF meeting in Berlin (http://www.ietf.org/meeting/87) later this month, which we wanted to make people aware of. The goal of the meeting is to form a working group following the meeting. A draft charter that is being discussed can be found here: http://www.ietf.org/mail-archive/web/aqm/current/msg00165.html -- Wes Eddy MTI Systems From detlef.bosau at web.de Sun Jul 21 13:28:24 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Sun, 21 Jul 2013 22:28:24 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: Message-ID: <51EC4468.1010105@web.de> To my understanding, you write down the whole communication (who wants to sent what to whom) and afterwards you calculate an optimized schedule. Reminds me of undergraduate homework in operating systems, gantt diagrams and that funny stuff. You cannot predict your link's properties (e.g. in the case of wireless links), you cannot predict your user's behaviour, so you conjecture a lot from presumptions which hardly ever will hold. Frankly spoken: This route leads to nowhere. -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From jon.crowcroft at cl.cam.ac.uk Sun Jul 21 14:14:46 2013 From: jon.crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Sun, 21 Jul 2013 22:14:46 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: <51EC4468.1010105@web.de> References: <51EC4468.1010105@web.de> Message-ID: it is a tiny bit cleverer than that - the work is the moral equivalent of the Axelrod experiment in emergent cooperation, but neater because it is quantitative rather than just qualitative selection of strategies - what is important (imho) is that they use many many simulation runs to evaluate a "fitness" of a given protocol...this is heavy lifting, but pays off - so it will be nice to see empirical follow up work but this isn't some naive "overfitting" undergrad work - it is rather different and requires a considered response On Sun, Jul 21, 2013 at 9:28 PM, Detlef Bosau wrote: > To my understanding, you write down the whole communication (who wants to > sent what to whom) and afterwards you calculate an optimized schedule. > > Reminds me of undergraduate homework in operating systems, gantt diagrams > and that funny stuff. > > You cannot predict your link's properties (e.g. in the case of wireless > links), you cannot predict your user's behaviour, so you conjecture a lot > from presumptions which hardly ever will hold. > > Frankly spoken: This route leads to nowhere. > > > -- > ------------------------------**------------------------------**------ > Detlef Bosau > Galileistra?e 30 > 70565 Stuttgart Tel.: +49 711 5208031 > mobile: +49 172 6819937 > skype: detlef.bosau > ICQ: 566129673 > detlef.bosau at web.de http://www.detlef-bosau.de > > From jtw at isi.edu Sun Jul 21 17:09:53 2013 From: jtw at isi.edu (John Wroclawski) Date: Sun, 21 Jul 2013 17:09:53 -0700 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> Message-ID: <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> Well, it seems to me that the key question is what's being traded off. Think of it this way: Imagine hypothetically for the moment that TCP is "optimal", ore at least effective, over some very wide and somewhat unspecified range of dynamic operating conditions, in the sense that _if you require that full range_ then TCP does about the best you can do. Now, here we have some work that basically says "if I narrow the range of conditions, I can do better within that range". (Figure 11 in the paper ..) So the first level interesting result is essentially pragmatic: that use of the ML algorithm appears to let them, plus or minus, choose any particular manifold within the overall "desired range of conditions" space, and then finds a CC algorithm that works "optimally", or at least "really well", within that manifold. That in and of itself is a neat result. But it seems to me that a fairly fundamental question is "is the performance improvement intrinsically coupled to narrowing the range, or did it come from the ML approach actually finding a "better" algorithm overall?" That is 1) hypothetically, you can imagine an "operating-range/performance product", which says that if I accept a narrower range of possible operating conditions I can do better within that range, and this is basically a conservation law; 2) and if that's true, then hypothetically - some portion of the performance gain shown by Remy is because they narrowed the range, and thus rode the R/P product, while - some other portion of the performance gain is because the ML is producing "better" algorithms overall, that is, more optimal than TCP given the same range objective for operating conditions. However it doesn't seem clear at the moment that we have any idea what the relative portions are, or even whether the overall hypothesis (the nature & existence of a relatively tractable R/P product in this space) is right. But in some sense this is the crucial question for a couple of reasons - both a) because it gets to the question of whether Remy is actually creating "better" algorithms, or just ones with narrower scope, and b), crucially IMO, because it gets to the question of whether the algorithms it creates are "fragile" and prone to rare-event catastrophic failure and "works today but not tomorrow" syndrome. (To be clear, the authors have made this observation too, as expressed in the discussion at the end of pg. 11 of the paper) Another way to say this is that if you think of Remy as a pragmatic tool that offers a way to design algorithms that operate "closer to the limits" than current TCP - if only because it lets you pick your limits and then tunes for what you picked - then it becomes even more motivating than today to back it up with a fundamental understanding of complex system CC dynamics and fragility. This for the same basic reason that once people started to try and "optimize" bridge design using finite element analysis, rather than just intuitive and empirical overkill, it became really important to be good at it. (and there was that troubling intermediate step in the middle, when bridges kept falling down..). In fact one might argue that TCP is as robust as it is today precisely _because_ no-one actually tried a priori to define some specific "operating condition boundaries" and push right out to those limits, and to the extent that that's what Remy is doing it raises some pretty interesting big-scale questions about how you keep complex systems like the Internet robust in the face of ever-increasing optimization. If, on the other hand, a Remy-like approach could produce higher-performance algorithms over the _same_ range of operating conditions as current TCP, but with that range more explicitly specified, that'd be even more interesting.. cheers, john On Jul 21, 2013, at 2:14 PM, Jon Crowcroft wrote: > it is a tiny bit cleverer than that - the work is the moral equivalent of > the Axelrod experiment in emergent cooperation, but neater because it is > quantitative rather than just qualitative selection of strategies - what is > important (imho) is that they use many many simulation runs to evaluate a > "fitness" of a given protocol...this is heavy lifting, but pays off - so it > will be nice to see empirical follow up work but this isn't some naive > "overfitting" undergrad work - it is rather different and requires a > considered response > > > On Sun, Jul 21, 2013 at 9:28 PM, Detlef Bosau wrote: > >> To my understanding, you write down the whole communication (who wants to >> sent what to whom) and afterwards you calculate an optimized schedule. >> >> Reminds me of undergraduate homework in operating systems, gantt diagrams >> and that funny stuff. >> >> You cannot predict your link's properties (e.g. in the case of wireless >> links), you cannot predict your user's behaviour, so you conjecture a lot >> from presumptions which hardly ever will hold. >> >> Frankly spoken: This route leads to nowhere. >> >> >> -- >> ------------------------------**------------------------------**------ >> Detlef Bosau >> Galileistra?e 30 >> 70565 Stuttgart Tel.: +49 711 5208031 >> mobile: +49 172 6819937 >> skype: detlef.bosau >> ICQ: 566129673 >> detlef.bosau at web.de http://www.detlef-bosau.de >> >> > From keithw at mit.edu Sun Jul 21 17:14:41 2013 From: keithw at mit.edu (Keith Winstein) Date: Sun, 21 Jul 2013 20:14:41 -0400 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> Message-ID: Thanks, Jon, I think the analogy to the Axelrod experiment is quite apt! Wish I had thought of that myself. :-) Similar to Axelrod, my dream is to have a "kaizen for congestion" where anybody could contribute a new algorithm and we would evaluate it vs. the existing schemes and see how it performs on different kinds of benchmark networks, and then add it to these same plots. Detlef, I'm afraid I don't think your email quite summarized our approach accurately. We do not give the optimizer advance information about who wants to send what to whom and we don't calculate an optimized "schedule." Remy develops rules for a TCP sender; e.g. when to increase the window and by how much, when/how to decrease the window, when to enforce a minimum interval between outgoing packets (a pacer) and what that interval should be. It tries to find the best rules for a given set of assumptions specified explicitly -- e.g., what are the range of possible networks the protocol is intended for, and what is the goal. We model the arrival and departure of flows as drawn from some stochastic process, e.g., flows are "on" for some amount of time or some number of bytes, drawn from an exponential or Pareto distribution or from an empirical CDF of flow lengths on the Internet. The traffic model given to Remy at design time usually is not the same as the case we then evaluate in ns-2 when comparing against the other schemes. Regarding wireless links, you might be interested in some of our prior work (http://alfalfa.mit.edu) that shows one can achieve considerable gains by modeling the link speed variation of cellular networks as a simple stochastic process, then making conservative predictions about future link speeds at the endpoints in order to compromise between throughput and delay. Best regards, Keith On Sun, Jul 21, 2013 at 5:14 PM, Jon Crowcroft wrote: > it is a tiny bit cleverer than that - the work is the moral equivalent of > the Axelrod experiment in emergent cooperation, but neater because it is > quantitative rather than just qualitative selection of strategies - what is > important (imho) is that they use many many simulation runs to evaluate a > "fitness" of a given protocol...this is heavy lifting, but pays off - so it > will be nice to see empirical follow up work but this isn't some naive > "overfitting" undergrad work - it is rather different and requires a > considered response > > > On Sun, Jul 21, 2013 at 9:28 PM, Detlef Bosau wrote: > > > To my understanding, you write down the whole communication (who wants to > > sent what to whom) and afterwards you calculate an optimized schedule. > > > > Reminds me of undergraduate homework in operating systems, gantt diagrams > > and that funny stuff. > > > > You cannot predict your link's properties (e.g. in the case of wireless > > links), you cannot predict your user's behaviour, so you conjecture a lot > > from presumptions which hardly ever will hold. > > > > Frankly spoken: This route leads to nowhere. > > > > > > -- > > ------------------------------**------------------------------**------ > > Detlef Bosau > > Galileistra?e 30 > > 70565 Stuttgart Tel.: +49 711 5208031 > > mobile: +49 172 6819937 > > skype: detlef.bosau > > ICQ: 566129673 > > detlef.bosau at web.de http://www.detlef-bosau.de > > > > > From fred at cisco.com Sun Jul 21 23:35:24 2013 From: fred at cisco.com (Fred Baker (fred)) Date: Mon, 22 Jul 2013 06:35:24 +0000 Subject: [e2e] TCP ex Machina In-Reply-To: <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> Message-ID: <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> I'm not sure folks are really concerned about operating boundaries as much as outcomes. In general, and in specific in the words of Nandita Dukkipati and folks like her, I think folks would like to maximize throughput and minimize the wall clock time it takes to deliver an item of data. Most of our algorithms confuse that with maximizing the amount of data in the network at any given time; I'll argue that this maximizes both throughput and the probability of loss, which increases the amount of time it takes to deliver something. Stated in terms of operating boundaries, I think the optimal TCP-or-equivalent probably maximizes throughput while minimizing its effective window - it keeps just enough outstanding to maximize throughput, and as a result minimizes its own loss rate and the loss rates of sessions it interacts with. Literally doing that, as CalTech FAST does, runs the risk of being too gentlemanly, but CAIA CDG, which will detect when it is competing with a sub-optimal transport like CUBIC, will in effect switch to a loss-based algorithm under duress. But I think maximizing throughput while minimizing the probability of loss is probably the holy grail. From jon.crowcroft at cl.cam.ac.uk Mon Jul 22 00:04:30 2013 From: jon.crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Mon, 22 Jul 2013 08:04:30 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> Message-ID: Richard Clegg at UCL presented a nice study recently where he has being trying to fit TCP equations to lots if big packet traces...he found precious few loss events, and the best explanation appears to be that the world is made of many zebras and mice....zebras are flows that are application rate limited...e.g video streaming...mice are flows that just don't make it out if slow start before they are done...eg web transactions, social media chat etc...it appears TCP CC is rarely invoked these days...... The other comments I'd make about the Remy thing is that it may be overfitting (like latterday netflix prize entries)..oh and they need to do more work on fairness I suppose... But I still like it On Jul 22, 2013 7:35 AM, "Fred Baker (fred)" wrote: > I'm not sure folks are really concerned about operating boundaries as much > as outcomes. In general, and in specific in the words of Nandita Dukkipati > and folks like her, I think folks would like to maximize throughput and > minimize the wall clock time it takes to deliver an item of data. Most of > our algorithms confuse that with maximizing the amount of data in the > network at any given time; I'll argue that this maximizes both throughput > and the probability of loss, which increases the amount of time it takes to > deliver something. Stated in terms of operating boundaries, I think the > optimal TCP-or-equivalent probably maximizes throughput while minimizing > its effective window - it keeps just enough outstanding to maximize > throughput, and as a result minimizes its own loss rate and the loss rates > of sessions it interacts with. Literally doing that, as CalTech FAST does, > runs the risk of being too gentlemanly, but CAIA CDG, which will detect > when it is competing with a sub-optimal transport like CUBIC, will in > effect switch to a loss-based algorithm under duress. But I think > maximizing throughput while minimizing the probability of loss is probably > the holy grail. From detlef.bosau at web.de Mon Jul 22 03:48:29 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 22 Jul 2013 12:48:29 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> Message-ID: <51ED0DFD.4020206@web.de> Am 22.07.2013 09:04, schrieb Jon Crowcroft: > Richard Clegg at UCL presented a nice study recently where he has being > trying to fit TCP equations to lots if big packet traces...h TCP is not about equations. TCP is about data transport. And ONLY about data transport. Some days ago, I once again read Metcalfe's Quote, networking would be inter process communication. No, and once again no. Communication is always a bilateral thing which requires basic agreements. I, myself, am not a process and when I now and then receive messages about penis enlargement pills and the like, this is neither bilateral not based upon basic agreements, it is simply annoying. Packet switched networking is bringing datagrams from a sender to somewhere, hopefully a receiver. (However, next time when a weird printer box causes a broadcast storm and renders a whole company network useless, I will say: "the printer box does inter process communication".) When Fred Baker points to the amount of data being in flight, he points to our worship of work conservation. (At least in part an outcome of Calvinistic ethics. Idleness is a sin. And punished by god.) (However, work conservation is well reasonable when idleness causes cost and should be avoided to spare these.) The more I think about it, the more I tend to distinguish two perspectives. The packet perspective: A packet wants to travel a network and for this purpose it wants to travel a) in direction of its final goal, b) to a free place in the network. Admittedly, for the purpose of a) a packet relies upon routing tables and the like because a packet has no idea about a network's topology. The global perspective: Some global "God", "Spirit", (Adam Smith would call it "the invisible hand") should mange the routing tables that way, that packets will reach their goal reasonable fast and that individual parts of the network must not become overcrowded. The latter is called congestion control. And the more I think about it, the less I'm convinced that our "amount of data in flight" probing, including any forms of "sophisticated dropping of packets" really follows this goal. An individual always travels to a free place which is as close as possible to its final destination. An individual has no idea whether the so found route is optimal (following some criteria) or even makes sense, it could be a dead end. Consequently, my question is whether congestion control should be primarily a matter of route planing. And not a mixture of self scheduling and a struggle for resources. (Although the local perspective, the packet's perspective, is necessarily a struggle for resources.) Perhaps this question was asked many times ago. However I simply ask it once again. From detlef.bosau at web.de Mon Jul 22 06:01:18 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 22 Jul 2013 15:01:18 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> Message-ID: <51ED2D1E.3050303@web.de> Am 22.07.2013 02:14, schrieb Keith Winstein: > Detlef, I'm afraid I don't think your email quite summarized our approach > accurately. We do not give the optimizer advance information about who > wants to send what to whom and we don't calculate an optimized "schedule." > Remy develops rules for a TCP sender; e.g. when to increase the window and > by how much, when/how to decrease the window, when to enforce a minimum > interval between outgoing packets (a pacer) and what that interval should > be. And I'm absolutely not sure whether it makes sense to increase or to decrease windows. Van Jacobson himself complains about packets lying in the network quite clumsy and not being "evenly distributed" along the path. Stated in an extremely harsh way, this would mean that the concept of "self scheduling", to which the original self clocking is extended by VJCC and the like, simply does not hold. So, we can improve purely endpoint based congestion control algorithms as much as we want - and perhaps increase the PhD outcome to 100 doctoral hats per year on congestion control, we will meet the same problems again and again, as we do for about 20 years now. > It tries to find the best rules for a given set of assumptions > specified explicitly -- e.g., what are the range of possible networks the > protocol is intended for, and what is the goal. > > We model the arrival and departure of flows as drawn from some stochastic > process, e.g., flows are "on" for some amount of time or some number of > bytes, drawn from an exponential or Pareto distribution or from an > empirical CDF of flow lengths on the Internet. The traffic model given to > Remy at design time usually is not the same as the case we then evaluate in > ns-2 when comparing against the other schemes. > > Regarding wireless links, you might be interested in some of our prior work > (http://alfalfa.mit.edu) that shows one can achieve considerable gains by > modeling the link speed variation of cellular networks as a simple > stochastic process, then making conservative predictions about future link > speeds at the endpoints in order to compromise between throughput and delay. > > Best regards, > Keith > > On Sun, Jul 21, 2013 at 5:14 PM, Jon Crowcroft > wrote: > >> it is a tiny bit cleverer than that - the work is the moral equivalent of >> the Axelrod experiment in emergent cooperation, but neater because it is >> quantitative rather than just qualitative selection of strategies - what is >> important (imho) is that they use many many simulation runs to evaluate a >> "fitness" of a given protocol...this is heavy lifting, but pays off - so it >> will be nice to see empirical follow up work but this isn't some naive >> "overfitting" undergrad work - it is rather different and requires a >> considered response >> >> >> On Sun, Jul 21, 2013 at 9:28 PM, Detlef Bosau wrote: >> >>> To my understanding, you write down the whole communication (who wants to >>> sent what to whom) and afterwards you calculate an optimized schedule. >>> >>> Reminds me of undergraduate homework in operating systems, gantt diagrams >>> and that funny stuff. >>> >>> You cannot predict your link's properties (e.g. in the case of wireless >>> links), you cannot predict your user's behaviour, so you conjecture a lot >>> from presumptions which hardly ever will hold. >>> >>> Frankly spoken: This route leads to nowhere. >>> >>> >>> -- >>> ------------------------------**------------------------------**------ >>> Detlef Bosau >>> Galileistra?e 30 >>> 70565 Stuttgart Tel.: +49 711 5208031 >>> mobile: +49 172 6819937 >>> skype: detlef.bosau >>> ICQ: 566129673 >>> detlef.bosau at web.de http://www.detlef-bosau.de >>> >>> -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From detlef.bosau at web.de Mon Jul 22 13:23:10 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 22 Jul 2013 22:23:10 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> Message-ID: <51ED94AE.4080204@web.de> Am 22.07.2013 08:35, schrieb Fred Baker (fred): > I'm not sure folks are really concerned about operating boundaries as much as outcomes. In general, and in specific in the words of Nandita Dukkipati and folks like her, I think folks would like to maximize throughput and minimize the And I'm not sure, that we are maximizing or minimizing anything. The more complex the formulae will be, the more difficult it will be to make reality obey them. Sometimes the belief in an optimum window in TCP resembles the belief in an optimum amount of money by the monetarism guys. - We have to share ressources: No problem! Use the optimum window! - We want to maximize throughput? " " " - We want to minimze latency? " " " Perhaps it's a personal attitude of mine at the moment, perhaps I'm simply tired, but it doesn't matter how complex the question is: The simple answer is: Use the correct CWND. (When I was a student I was acquainted to some guys of Campus Crusade for Christ. They shared the "Four Spiritual Laws" as the only effective solution for each and anything. Unfortunately, although this has been more then twenty years ago, I'm still looking for the problem fitting this solution.) However, it may be my mood in the moment. E.g., VJ told us in his famous talk from 2006, that TCP does not work well in wireless environments when we're moving around. O.k., this is only seven years ago and my Linux now include an experimental BIC (I don't know for sure, I never looked particularly for this), and this doesn't help in wireless environments when I'm moving around. Did anything else change? Or do we, to a certain degree, repeat the same things we repeat for 20 years now? In the talk from 2006, VJ was afraid that we are stuck in some kind of deadlock. This is basically a similar view to that taken by Ford and Iyengar in 2009. Is there some truth in it? Detlef -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From detlef.bosau at web.de Mon Jul 22 15:39:58 2013 From: detlef.bosau at web.de (Detlef Bosau) Date: Tue, 23 Jul 2013 00:39:58 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> Message-ID: <51EDB4BE.1070602@web.de> Just for the record, you think of Robert Axelrod and his game theoretic experiments? Am 21.07.2013 23:14, schrieb Jon Crowcroft: > it is a tiny bit cleverer than that - the work is the moral equivalent of > the Axelrod experiment in emergent cooperation, but neater because it is > quantitative rather than just qualitative selection of strategies - what is > important (imho) is that they use many many simulation runs to evaluate a > "fitness" of a given protocol...this is heavy lifting, but pays off - so it > will be nice to see empirical follow up work but this isn't some naive > "overfitting" undergrad work - it is rather different and requires a > considered response > > > On Sun, Jul 21, 2013 at 9:28 PM, Detlef Bosau wrote: > >> To my understanding, you write down the whole communication (who wants to >> sent what to whom) and afterwards you calculate an optimized schedule. >> >> Reminds me of undergraduate homework in operating systems, gantt diagrams >> and that funny stuff. >> >> You cannot predict your link's properties (e.g. in the case of wireless >> links), you cannot predict your user's behaviour, so you conjecture a lot >> from presumptions which hardly ever will hold. >> >> Frankly spoken: This route leads to nowhere. >> >> >> -- >> ------------------------------**------------------------------**------ >> Detlef Bosau >> Galileistra?e 30 >> 70565 Stuttgart Tel.: +49 711 5208031 >> mobile: +49 172 6819937 >> skype: detlef.bosau >> ICQ: 566129673 >> detlef.bosau at web.de http://www.detlef-bosau.de >> >> -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From jon.crowcroft at cl.cam.ac.uk Wed Jul 24 01:29:37 2013 From: jon.crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Wed, 24 Jul 2013 10:29:37 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: <51ED0DFD.4020206@web.de> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51ED0DFD.4020206@web.de> Message-ID: the work i was referring to uses the Padhye equation which is not claiming that TCP is "about equations", but has a multi-regime model of how TCP CC appears to work (based on code - the difference between C code and equations is of course a purely academic distinction anyhow). so taking _actual_ packet traces (lots of them) you can see how many packet losses and timeouts trigger retransmits, and you can see how the TCP send windows evolve over time, and correlate these with the distribution of mean and mean square difference of RTT over large numbers of flows... this correlation is done by curve/parameter fitting, so you need an equation with parameters (the padhye one is possibly the best of breed) and the result was as I said..... the rest of your response doesn't seem to be about end-to-end research, so I'll leave that to a more philosophical venue for discussion by people qualified to do so (not me) j. On Mon, Jul 22, 2013 at 12:48 PM, Detlef Bosau wrote: > Am 22.07.2013 09:04, schrieb Jon Crowcroft: > > Richard Clegg at UCL presented a nice study recently where he has being >> trying to fit TCP equations to lots if big packet traces...h >> > TCP is not about equations. > > TCP is about data transport. > > And ONLY about data transport. > > Some days ago, I once again read Metcalfe's Quote, networking would be > inter process communication. > > No, and once again no. > > Communication is always a bilateral thing which requires basic agreements. > > I, myself, am not a process and when I now and then receive messages about > penis enlargement pills and the like, this is neither bilateral not based > upon basic agreements, it is simply annoying. > > > Packet switched networking is bringing datagrams from a sender to > somewhere, hopefully a receiver. > > (However, next time when a weird printer box causes a broadcast storm and > renders a whole company network useless, I will say: "the printer box does > inter process communication".) > > When Fred Baker points to the amount of data being in flight, he points to > our worship of work conservation. > (At least in part an outcome of Calvinistic ethics. Idleness is a sin. And > punished by god.) > (However, work conservation is well reasonable when idleness causes cost > and should be avoided to spare these.) > > The more I think about it, the more I tend to distinguish two perspectives. > > The packet perspective: A packet wants to travel a network and for this > purpose it wants to travel a) in direction of its final goal, b) to a free > place in the network. Admittedly, for the purpose of a) a packet relies > upon routing tables and the like because a packet has no idea about a > network's topology. > > The global perspective: Some global "God", "Spirit", (Adam Smith would > call it "the invisible hand") should mange the routing tables that way, > that packets will reach their goal reasonable fast and that individual > parts of the network must not become overcrowded. > > The latter is called congestion control. And the more I think about it, > the less I'm convinced that our "amount of data in flight" probing, > including any forms of "sophisticated dropping of packets" really follows > this goal. > An individual always travels to a free place which is as close as possible > to its final destination. An individual has no idea whether the so found > route is optimal (following some criteria) or even makes sense, it could be > a dead end. > > Consequently, my question is whether congestion control should be > primarily a matter of route planing. And not a mixture of self scheduling > and a struggle for resources. (Although the local perspective, the packet's > perspective, is necessarily a struggle for resources.) > > Perhaps this question was asked many times ago. However I simply ask it > once again. > > From touch at isi.edu Thu Jul 25 14:00:00 2013 From: touch at isi.edu (Joe Touch) Date: Thu, 25 Jul 2013 14:00:00 -0700 Subject: [e2e] TCP ex Machina In-Reply-To: <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> Message-ID: <51F191D0.6030807@isi.edu> On 7/21/2013 11:35 PM, Fred Baker (fred) wrote: > But I think maximizing throughput while minimizing the probability of loss is probably the holy grail. It also depends on whether you measure that as an individual or as a group. I tend to think of TCP as supporting the most participants having reliable communication as a primary goal, and a secondary goal to maximize (though not optimize) the aggregate data per unit time -- subject to the primary goal. I.e., it's not about max BW for a given connection, and it also almost never means minimizing the transfer time for a single connection. Joe From touch at isi.edu Thu Jul 25 14:22:16 2013 From: touch at isi.edu (Joe Touch) Date: Thu, 25 Jul 2013 14:22:16 -0700 Subject: [e2e] TCP ex Machina In-Reply-To: <51F191D0.6030807@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> Message-ID: <51F19708.3050206@isi.edu> FWIW, I've been concerned about a number of proposed "extensions" to TCP, either to optimize it for DNS (TCP-CT) or for Google (various). My concern is because these optimizations sidestep my view of the primary goal of TCP as described below. Joe On 7/25/2013 2:00 PM, Joe Touch wrote: > > > On 7/21/2013 11:35 PM, Fred Baker (fred) wrote: >> But I think maximizing throughput while minimizing the probability of >> loss is probably the holy grail. > > It also depends on whether you measure that as an individual or as a group. > > I tend to think of TCP as supporting the most participants having > reliable communication as a primary goal, and a secondary goal to > maximize (though not optimize) the aggregate data per unit time -- > subject to the primary goal. > > I.e., it's not about max BW for a given connection, and it also almost > never means minimizing the transfer time for a single connection. > > Joe From Jon.Crowcroft at cl.cam.ac.uk Thu Jul 25 21:53:43 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Fri, 26 Jul 2013 05:53:43 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: <51F19708.3050206@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> Message-ID: I'm sorry, but this smacks of "orthodoxy" which is anathema to good science. the reality is that the nature of most of the "optimization" work that makes it into good conferences (nsdi, icnp, sigcomm, pick your fave) are in the nature of the word you use - extensions they do not "sidestep" anything - they change the behaviour in the situation where the extesion applies and not otherwise, and are typically "backwards compatible" as much as possible, as the one true orthodoxy that is around is that any decent paper evaluates the tcp-atlantis or tcp-erewhon both in the new scenario, and in a mix with tcp-vanilla or tcp-realworld, to see what the impact of each on the other is, because thats what scientists do when they do their evaluation properly - they want to know what happens in the real world, not just in some bubble world however, it is also legitimate science to ask what happens in some brave new world, if one has the capability to change everything - some of these brave new worlds aren't small - they are places like all cell phones, or in a data center with a few million cores, where one might actually have the capability to have a flag day (ship the tcp-flagrante to all devices and enable)... that new world might not share the same primary goal - for example, in some data center it might be perfectly reasonable to sidestep your goal as the modest owner of the data center might actually want to maximise total crosssectional throughput for some odd reason... I just dont buy the homogenous trope any more - what is more, we have dynamic systems that rebuild, relink, reload, reboot - we can decide which is the appropriate behaviour - indeed, many of these systems do this anyhow at some level (when switching from wifi to 3g/4g, when switching from hadoop using tcp to do reduce/shuffle phase, to video streaming over tcp to a customer - in lots of other cases) for these people, what is wrong with experimenting and publishing papers on the results of the experiments? it would be wrong not to publically propose and expose evaluations of these extensions or modifications. in any case, TCP and IP were not perfect at birth through some sort of intelligent design which magically foresaw all these new scenarios- they were prototypes that expressed some very cool ideas, but missed some details and I believe we learned some new stuff since anyhow... so of course there might be extensions which would kick in, in the wrong circumstances and do harm - so do you believe any OS vendor or community would ship those? do you believe that unless the papers about them showed a net win, anyone would use them? do you think we shouldn't propose ideas, that occasionally turn out to be wrong? how else will things move on? in any case, the TCP ex Machina paper isn't this at all so on the topic of evolving (and evaluating in a complex rich way) new variants of CC, I think one way you could think about this paper is the following instead of waiting a duty cycle of 1 grad-student-year for a next TCP congestion control idea, you use a big cluster of machines and evolutionary computing to run at the rate of around 100 graduate students-per-week and pick the best according to some metric - you could, if you like, pick a different metric - Fred's proposed one - that's fine - the paper just illustrates that we have a capability to do some thing which was probably rather difficult to consider in 1976 or 1988 (pick your preferred incept date here)... I'd quite like DNS lookup and search to go faster anyhow- but i dont think the people working on that want the download times of the PDFs of their papers to slow down or be unrelable as a result... Do you have specific examples of work that would do that? cheers jon In missive <51F19708.3050206 at isi.edu>, Joe Touch typed: >>FWIW, I've been concerned about a number of proposed "extensions" to >>TCP, either to optimize it for DNS (TCP-CT) or for Google (various). >> >>My concern is because these optimizations sidestep my view of the >>primary goal of TCP as described below. >> >>Joe >> >>On 7/25/2013 2:00 PM, Joe Touch wrote: >>> >>> >>> On 7/21/2013 11:35 PM, Fred Baker (fred) wrote: >>>> But I think maximizing throughput while minimizing the probability of >>>> loss is probably the holy grail. >>> >>> It also depends on whether you measure that as an individual or as a group. >>> >>> I tend to think of TCP as supporting the most participants having >>> reliable communication as a primary goal, and a secondary goal to >>> maximize (though not optimize) the aggregate data per unit time -- >>> subject to the primary goal. >>> >>> I.e., it's not about max BW for a given connection, and it also almost >>> never means minimizing the transfer time for a single connection. >>> >>> Joe cheers jon From touch at isi.edu Fri Jul 26 10:44:32 2013 From: touch at isi.edu (Joe Touch) Date: Fri, 26 Jul 2013 10:44:32 -0700 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> Message-ID: <51F2B580.9040703@isi.edu> Hi, Jon, On 7/25/2013 9:53 PM, Jon Crowcroft wrote: > I'm sorry, but this smacks of "orthodoxy" > which is anathema to good science. I'm not referring to science, or research. I'm referring to changes to TCP that are being pushed for deployment or already deployed. E.g., TCP-CT is not an approved IETF standard. It was not taken to through the IETF process. But it is *deployed* in Linux. As an example of bad behavior, TCP-CT uses a TCP option codepoint that is reserved for experiments that are not widely deployed. > the reality is that the nature of > most of the "optimization" work that makes it into good conferences > (nsdi, icnp, sigcomm, pick your fave) > are in the nature of the word you use - extensions There are different kinds of TCP extensions, falling into at least two broad categories: A- extensions to TCP itself A1- some intended to make TCP more robust or address a corner case A2- some intended to correct a problem in TCP's design A3- some intended to support a specific use case B- using the TCP protocol ID and coarsely conforming to TCP's connection establishment I'm not referring to A1 or A2, which includes some published research. I'm concerned more about B, and somewhat about A3. B-style extensions are really new transport protocols. That's a valid area of research, but the deployment reality is that no such new transports can traverse widely-deployed non-standard network elements (e.g., NATs). So B is really a new transport masquerading as TCP. A3 are extensions to optimize specific use cases. The trouble with such optimizations is that they can easily undermine a key feature of TCP - that it falls back to working nearly all the time it possibly can. I don't consider breaking that key property of TCP a good thing. Experimentation is fine, as is research. But the medical community doesn't start playing with new meds by handing them out on street corners like candy. ... > so of course there might be extensions which would kick in, > in the wrong circumstances and do harm - > > so do you believe any OS vendor or > community would ship those? Linux. Cisco. Alcatel-Lucent. This is a partial list of vendors who have deployed code that uses *the same* experimental codepoint. > do you believe that unless the papers about them showed a net win, > anyone would use them? Yes. Linux has done this repeatedly. ... > instead of waiting a duty cycle of 1 grad-student-year for a next > TCP congestion control idea, > you use a big cluster of machines and evolutionary computing > to run at the rate of around 100 graduate students-per-week > and pick the best according to some metric - > > you could, if you like, pick a different metric I did, and you took issue with it. Joe From Jon.Crowcroft at cl.cam.ac.uk Fri Jul 26 21:48:53 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Sat, 27 Jul 2013 05:48:53 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: <51F2B580.9040703@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> Message-ID: while linux (pick your flavour) cubic isn't vanilla, neither is microsoft's compound tcp - the latter might have seen a bit more eval in the literature but the former has seen a lot more big iron deployment and doesn't appear to have broken the internet yet (although there are rumours and reports of corner case problems) but i dont think either of these are "non tcp" - they are variants on CC behaviour.... but this isn't relevant to the _starting point_ of this discussion which was the annoucement of an interesting, useful, and peer-reviewed paper on an experiment on accelating TCP cc research the folks from MIT just posted a perfectly reasonable message about this valuble contribution, and some people choose to link their work with a line of argument, which is unfair, and unjustly including them (by at the least not changing the subject line). also - the ability to do any deployment testing of a new tcp in anger _requires you_ to be wireline compatible with TCP because of the "non stadard" but ubiquitous NATs and other middleboxes (as reported in several papers - e.g. UCL fine wotk on deploying MPTCP and the difficulties testing in the real world which doesn;t confoirm to the IETF model much at all any more) so the gold standard you quite reasonably want to hold people to, to show their work doesn't do harm in the wild, requires them to do "harm" by making their new variant TCP appear chameleon like, vanilla TCP, so they can get results (this, as well as simulation of jain-fairness beahvous or Remy v. reno, is something I hope the MIT folks do next) In missive <51F2B580.9040703 at isi.edu>, Joe Touch typed: >>Hi, Jon, >> >>On 7/25/2013 9:53 PM, Jon Crowcroft wrote: >>> I'm sorry, but this smacks of "orthodoxy" >>> which is anathema to good science. >> >>I'm not referring to science, or research. I'm referring to changes to >>TCP that are being pushed for deployment or already deployed. >> >>E.g., TCP-CT is not an approved IETF standard. It was not taken to >>through the IETF process. But it is *deployed* in Linux. As an example >>of bad behavior, TCP-CT uses a TCP option codepoint that is reserved for >>experiments that are not widely deployed. >> >>> the reality is that the nature of >>> most of the "optimization" work that makes it into good conferences >>> (nsdi, icnp, sigcomm, pick your fave) >>> are in the nature of the word you use - extensions >> >>There are different kinds of TCP extensions, falling into at least two >>broad categories: >> >> A- extensions to TCP itself >> A1- some intended to make TCP more robust or address >> a corner case >> A2- some intended to correct a problem in TCP's design >> A3- some intended to support a specific use case >> >> B- using the TCP protocol ID and coarsely conforming to TCP's >> connection establishment >> >>I'm not referring to A1 or A2, which includes some published research. >> >>I'm concerned more about B, and somewhat about A3. >> >>B-style extensions are really new transport protocols. That's a valid >>area of research, but the deployment reality is that no such new >>transports can traverse widely-deployed non-standard network elements >>(e.g., NATs). So B is really a new transport masquerading as TCP. >> >>A3 are extensions to optimize specific use cases. The trouble with such >>optimizations is that they can easily undermine a key feature of TCP - >>that it falls back to working nearly all the time it possibly can. >> >>I don't consider breaking that key property of TCP a good thing. >> >>Experimentation is fine, as is research. But the medical community >>doesn't start playing with new meds by handing them out on street >>corners like candy. >> >>... >>> so of course there might be extensions which would kick in, >>> in the wrong circumstances and do harm - >>> >>> so do you believe any OS vendor or >>> community would ship those? >> >>Linux. Cisco. Alcatel-Lucent. This is a partial list of vendors who have >>deployed code that uses *the same* experimental codepoint. >> >>> do you believe that unless the papers about them showed a net win, >>> anyone would use them? >> >>Yes. Linux has done this repeatedly. >> >>... >>> instead of waiting a duty cycle of 1 grad-student-year for a next >>> TCP congestion control idea, >>> you use a big cluster of machines and evolutionary computing >>> to run at the rate of around 100 graduate students-per-week >>> and pick the best according to some metric - >>> >>> you could, if you like, pick a different metric >> >>I did, and you took issue with it. >> >>Joe >> cheers jon From touch at isi.edu Sat Jul 27 09:24:47 2013 From: touch at isi.edu (Joe Touch) Date: Sat, 27 Jul 2013 09:24:47 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> Message-ID: <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> On Jul 26, 2013, at 9:48 PM, Jon Crowcroft wrote: > while linux (pick your flavour) cubic isn't vanilla, neither is > microsoft's compound tcp - the latter might have seen a bit more > eval in the literature but the former has seen a lot more big iron > deployment and doesn't appear to have broken the internet yet How would we know? They're not instrumented. These are not experiments; they're deployments. Even Schr?dinger's cat eventually sees the light of day (as much as there is a cat in the first place). > (although there are rumours and reports of corner case problems) > but i dont think either of these are "non tcp" - they are variants > on CC behaviour.... Which is a specified standard, which these mechanisms violate. You do bring up a valid point about the subject line, so I've changed it for this thread going forward. > also - the ability to do any deployment testing of a new tcp in > anger _requires you_ to be wireline compatible with TCP because of > the "non stadard" but ubiquitous NATs and other middleboxes The environment doesn't support safe experiments, but that is not a valid excuse for unsafe ones. ... > so the gold standard you quite reasonably want to hold people to, > to show their work doesn't do harm in the wild, > requires them to do "harm" by making > their new variant TCP appear chameleon like, > vanilla TCP, so they can get results So they do harm to avoid doing harm? They have failed because of their first step. Joe From jeanjour at comcast.net Sat Jul 27 11:36:30 2013 From: jeanjour at comcast.net (John Day) Date: Sat, 27 Jul 2013 14:36:30 -0400 Subject: [e2e] TCP "experiments" In-Reply-To: <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: As those with experience in the product world or operational world will know, the distinction here is between research/development and product/operations. One never does experiments with a production network. There was a time when there was little choice. The cost of building a network of any size was astronomical, or at least very high. So one had to do experiments on live networks and be as careful as you could. Today that is not the case. An arbitrary network of several hundred nodes or even a few thousand is not that big a deal. Joe has a point. An production network is no place to be doing experiments. In fact, quite the opposite. Experiments should be done on experimental networks. (to state the opposite.) Take care, At 9:24 AM -0700 7/27/13, Joe Touch wrote: >On Jul 26, 2013, at 9:48 PM, Jon Crowcroft wrote: > >> while linux (pick your flavour) cubic isn't vanilla, neither is >> microsoft's compound tcp - the latter might have seen a bit more >> eval in the literature but the former has seen a lot more big iron >> deployment and doesn't appear to have broken the internet yet > >How would we know? They're not instrumented. >These are not experiments; they're deployments. > >Even Schr?dinger's cat eventually sees the light >of day (as much as there is a cat in the first >place). > >> (although there are rumours and reports of corner case problems) >> but i dont think either of these are "non tcp" - they are variants >> on CC behaviour.... > >Which is a specified standard, which these mechanisms violate. > >You do bring up a valid point about the subject >line, so I've changed it for this thread going >forward. > >> also - the ability to do any deployment testing of a new tcp in >> anger _requires you_ to be wireline compatible with TCP because of >> the "non stadard" but ubiquitous NATs and other middleboxes > >The environment doesn't support safe >experiments, but that is not a valid excuse for >unsafe ones. >... >> so the gold standard you quite reasonably want to hold people to, >> to show their work doesn't do harm in the wild, >> requires them to do "harm" by making >> their new variant TCP appear chameleon like, >> vanilla TCP, so they can get results > >So they do harm to avoid doing harm? > >They have failed because of their first step. > >Joe From lachlan.andrew at gmail.com Sat Jul 27 17:33:57 2013 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Sun, 28 Jul 2013 10:33:57 +1000 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: Greetings John, On 28 July 2013 04:36, John Day wrote: > One never does experiments with a production network. > > An arbitrary network of several hundred nodes > or even a few thousand is not that big a deal. Greetings John, You are absolutely right that testbed experiments should be performed before "live" experiments. However, it is not so much the size of the network as the mix of applications running on it that makes the test representative. It is still very difficult to perform a test with a few thousand human users all doing their thing. That means that live experiments still have a place. Of course, that doesn't excuse un-monitored deployments as occurred when Linux started using BIC as the default. To my mind, the solution would be for the IETF to provide more practical guidance on how to perform limited-scale, monitored tests on the real Internet. The process of getting a protocol "approved", even as an experimental RFC, is far too cumbersome for most researchers, especially since there is no way to police the use of non-approved protocols. The IETF will be most relevant if its processes reflect its power. We (or at least I) want the Internet to be inherited by those who try to play by the rules rather than those who flaunt them, but the if the only way to make timely progress is by breaking the rules then we won't achieve that (as we saw with CUBIC and NATs). Getting the balance right is difficult, but important. $0.02, Lachlan -- Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) Swinburne University of Technology, Melbourne, Australia Ph +61 3 9214 4837 From jeanjour at comcast.net Sat Jul 27 18:25:54 2013 From: jeanjour at comcast.net (John Day) Date: Sat, 27 Jul 2013 21:25:54 -0400 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: Lachlan, At 10:33 AM +1000 7/28/13, Lachlan Andrew wrote: >Greetings John, > >On 28 July 2013 04:36, John Day wrote: >> One never does experiments with a production network. >> >> An arbitrary network of several hundred nodes >> or even a few thousand is not that big a deal. > >Greetings John, > >You are absolutely right that testbed experiments should be performed >before "live" experiments. However, it is not so much the size of the >network as the mix of applications running on it that makes the test >representative. It is still very difficult to perform a test with a >few thousand human users all doing their thing. That means that live >experiments still have a place. Are you telling me that we don't have good statistical models for the behavior of large numbers of users? I would suggest that this is just laziness, irresponsible behavior or both. > >Of course, that doesn't excuse un-monitored deployments as occurred >when Linux started using BIC as the default. To my mind, the solution >would be for the IETF to provide more practical guidance on how to >perform limited-scale, monitored tests on the real Internet. That is not the job of an organization dedicated to the maintenance of a production network. >The >process of getting a protocol "approved", even as an experimental RFC, >is far too cumbersome for most researchers, especially since there is >no way to police the use of non-approved protocols. Have you ever seen what researchers in other fields do to ensure that they get accurate and reliable results? Triple distillations, putting precisely the same amount in 2000 test tubes, making your own reagents? O, these poor CS researchers might have to actually do some real science. I would question whether it is the job of *researchers* to get a protocol approved by the IETF for use in product. O, sorry, I forgot. University researchers don't do research anymore, they are just cheap developers for big corporations, or wimps using tax dollars for angel funding from the safety of their tenure. > The IETF will be >most relevant if its processes reflect its power. We (or at least I) >want the Internet to be inherited by those who try to play by the >rules rather than those who flaunt them, but the if the only way to >make timely progress is by breaking the rules then we won't achieve >that (as we saw with CUBIC and NATs). Getting the balance right is >difficult, but important. But you do have a good point. I have some interesting ideas for more efficient use of the power grid, I think I will deploy them to see how they affect the balance of the grid. Gee, what could go wrong! Take care, John >$0.02, >Lachlan > >-- >Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) >Swinburne University of Technology, Melbourne, Australia > >Ph +61 3 9214 4837 From mattmathis at google.com Sat Jul 27 19:20:37 2013 From: mattmathis at google.com (Matt Mathis) Date: Sat, 27 Jul 2013 19:20:37 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: The real issue is the diversity of implementations in the Internet that allege to be standard IP and TCP NAT, but contain undocumented "features". No level of simulation has any hope of predicting how well a new protocol feature or congestion control algorithm will actually function in the real Internet - you have to measure it. Furthermore: given that Google gets most of its revenue from clicks, how much might it cost us to "deploy" a protocol feature that caused 0.01% failure rate? If you were Google management, how large of sample size would you want to have before you might be willing actually deploy something globally? Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. On Sat, Jul 27, 2013 at 5:33 PM, Lachlan Andrew wrote: > Greetings John, > > On 28 July 2013 04:36, John Day wrote: > > One never does experiments with a production network. > > > > An arbitrary network of several hundred nodes > > or even a few thousand is not that big a deal. > > Greetings John, > > You are absolutely right that testbed experiments should be performed > before "live" experiments. However, it is not so much the size of the > network as the mix of applications running on it that makes the test > representative. It is still very difficult to perform a test with a > few thousand human users all doing their thing. That means that live > experiments still have a place. > > Of course, that doesn't excuse un-monitored deployments as occurred > when Linux started using BIC as the default. To my mind, the solution > would be for the IETF to provide more practical guidance on how to > perform limited-scale, monitored tests on the real Internet. The > process of getting a protocol "approved", even as an experimental RFC, > is far too cumbersome for most researchers, especially since there is > no way to police the use of non-approved protocols. The IETF will be > most relevant if its processes reflect its power. We (or at least I) > want the Internet to be inherited by those who try to play by the > rules rather than those who flaunt them, but the if the only way to > make timely progress is by breaking the rules then we won't achieve > that (as we saw with CUBIC and NATs). Getting the balance right is > difficult, but important. > > $0.02, > Lachlan > > -- > Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) > Swinburne University of Technology, Melbourne, Australia > > Ph +61 3 9214 4837 > From jeanjour at comcast.net Sat Jul 27 20:18:05 2013 From: jeanjour at comcast.net (John Day) Date: Sat, 27 Jul 2013 23:18:05 -0400 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: I believe what you are pointing out is known as "the horse is already out of the barn." ;-) Take care, John At 7:20 PM -0700 7/27/13, Matt Mathis wrote: >The real issue is the diversity of implementations in the Internet >that allege to be standard IP and TCP NAT, but contain undocumented >"features". No level of simulation has any hope of predicting how >well a new protocol feature or congestion control algorithm will >actually function in the real Internet - you have to measure it. > >Furthermore: given that Google gets most of its revenue from clicks, >how much might it cost us to "deploy" a protocol feature that caused >0.01% failure rate? If you were Google management, how large of >sample size would you want to have before you might be willing >actually deploy something globally? > > >Thanks, >--MM-- >The best way to predict the future is to create it. - Alan Kay > >Privacy matters! We know from recent events that people are using >our services to speak in defiance of unjust governments. We treat >privacy and security as matters of life and death, because for some >users, they are. > > >On Sat, Jul 27, 2013 at 5:33 PM, Lachlan Andrew ><lachlan.andrew at gmail.com> wrote: > >Greetings John, > > >On 28 July 2013 04:36, John Day ><jeanjour at comcast.net> wrote: >> One never does experiments with a production network. >> > > > An arbitrary network of several hundred nodes >> or even a few thousand is not that big a deal. > >Greetings John, > >You are absolutely right that testbed experiments should be performed >before "live" experiments. However, it is not so much the size of the >network as the mix of applications running on it that makes the test >representative. It is still very difficult to perform a test with a >few thousand human users all doing their thing. That means that live >experiments still have a place. > >Of course, that doesn't excuse un-monitored deployments as occurred >when Linux started using BIC as the default. To my mind, the solution >would be for the IETF to provide more practical guidance on how to >perform limited-scale, monitored tests on the real Internet. The >process of getting a protocol "approved", even as an experimental RFC, >is far too cumbersome for most researchers, especially since there is >no way to police the use of non-approved protocols. The IETF will be >most relevant if its processes reflect its power. We (or at least I) >want the Internet to be inherited by those who try to play by the >rules rather than those who flaunt them, but the if the only way to >make timely progress is by breaking the rules then we won't achieve >that (as we saw with CUBIC and NATs). Getting the balance right is >difficult, but important. > >$0.02, >Lachlan > >-- >Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) >Swinburne University of Technology, Melbourne, Australia ><http://caia.swin.edu.au/cv/landrew> >Ph +61 3 9214 4837 From lachlan.andrew at gmail.com Sat Jul 27 21:18:10 2013 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Sun, 28 Jul 2013 14:18:10 +1000 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: Greetings John, On 28 July 2013 11:25, John Day wrote: > At 10:33 AM +1000 7/28/13, Lachlan Andrew wrote: >> >> You are absolutely right that testbed experiments should be performed >> before "live" experiments. However, it is not so much the size of the >> network as the mix of applications running on it that makes the test >> representative. > > Are you telling me that we don't have good statistical models for the > behavior of large numbers of users? I would suggest that this is just > laziness, irresponsible behavior or both. It depends on what you mean by "good". We have lots of models of lots of aspects of their behaviour, but cannot possibly hope to model all of them. If you go to TCPM and say "Here is an enhancement of TCP. I know it works because I have a statistical model of it", do you think they will be happy, however "good" your model is? I don't. There is a need for testing at many levels. We need to test with simulations / models, with testbeds with synthetic users, with testbeds with real users, and with the real internet. Otherwise, we don't understand how things will really behave. >> Of course, that doesn't excuse un-monitored deployments as occurred >> when Linux started using BIC as the default. To my mind, the solution >> would be for the IETF to provide more practical guidance on how to >> perform limited-scale, monitored tests on the real Internet. > > That is not the job of an organization dedicated to the maintenance of a > production network. True. That is why I don't suggest that NANOG provide such a list. I see the role of the IETF as to promote and guide innovation in the Internet, rather than to maintain it. > Have you ever seen what researchers in other fields do to ensure that they > get accurate and reliable results? Triple distillations, putting precisely > the same amount in 2000 test tubes, making your own reagents? In physics and chemistry, experiments are done on the real physical and chemical world. In medicine, they run clinical trials on real people. As I said, I'm not proposing that people should run tests on the public internet *instead* of doing testbed studies. I'm saying that there comes a time when things need to be tested in reality. The IETF has an opportunity to guide these tests. If it chooses not to guide them, then they will be carried out without its guidance. > I would question whether it is the job of *researchers* to get a protocol > approved by the IETF for use in product. I thought that was the role of experimental RFCs. Who will do the experiments, if not researchers? Also, I was explicitly not talking about putting things in products. I was talking about doing experiments on the Internet. I believe that Cubic should not have been deployed as the Linux default until it had been tested properly. I maintain that "proper testing" includes properly monitored experiments on the Internet. The issue of what "experiments" are allowed on the Internet goes to the heart of things like the ACM's IMC. Should CAIDA not be allowed to experiment on the Internet because it is a production network? Should companies like Akamai be allowed to repeat the (sometimes quite burdensome) measurements on a regular basis? How often? The IETF is welcome not to care about such things, but I was suggested that they may have useful things to say. If they don't care about such things, why should they care about other experiments? Since so few TCP flows are affected by the congestion control algorithm, why is it so sacrosanct? >> The IETF will be >> most relevant if its processes reflect its power. > > But you do have a good point. I have some interesting ideas for more > efficient use of the power grid, I think I will deploy them to see how they > affect the balance of the grid. Gee, what could go wrong! The makers of electric vehicles are doing exactly that. Of course, EVs have to be approved for electrical safety, but that doesn't do anything to show what effect they have on the balance of the grid. EVs are a nightmare. (As an aside, last week, at last week's IEEE Power and Energy Society General Meeting we were told that in Alberta, the network operator has no say in what new generation is placed where. They just have to deal with what happens. The situation is not ideal, but is not unique to the Internet. I assume you were being sarcastic, but if you really do have some ideas for more efficient energy use, I certainly hope that you pursue them. That is much more likely to help the world than any development in cyberspace ever could.) Cheers, Lachlan -- Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) Swinburne University of Technology, Melbourne, Australia Ph +61 3 9214 4837 From randy at psg.com Sat Jul 27 21:35:00 2013 From: randy at psg.com (Randy Bush) Date: Sun, 28 Jul 2013 06:35:00 +0200 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: > One never does experiments with a production network. some of us do. many things are impossible to see in vitro. randy From Jon.Crowcroft at cl.cam.ac.uk Sat Jul 27 22:26:29 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Sun, 28 Jul 2013 06:26:29 +0100 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: at some point you need to do live testing - this is done in pharmaceuticals (c.f.clinical trials), food (gm crops), power systems, transportation, pretty much every sector (hey even politics) - it is n't avbout academic v. commerical - and there's a non-zero risk things wont work out the point is that there are a number of more important players than the IETF who (also) have an interest in not breaking the internet, who do these experiments - and they monitor stuff and they (usually) report on it - and they operate within real world constraints but need those constraints (as everyone's said, the internet doesn't have nice theoertical traffic that stays still while you do your experiment, nor does it just have vanilla routers that forard any and all standards-compliant packets) to tell if the experimental results are going to be useful or not the internet isn't just one production network - its an ecosystem with lots of components - just oone R&E network I'm connected to specifically allows for experiments (with support from its operational team) - its a modest 10-100Gbps backbone with around 7M hosts.... it also runs production services and has AUPs etc we've always done experiments on "production" networks - every day is an experiment on planet earth by the human race... hmm - maybe that's one place we should be a bit more careful - is it good or bad that it isn't run by the IETF:) In missive , John Day typed: >>As those with experience in the product world or >>operational world will know, the distinction here >>is between research/development and >>product/operations. One never does experiments >>with a production network. >> >>There was a time when there was little choice. >>The cost of building a network of any size was >>astronomical, or at least very high. So one had >>to do experiments on live networks and be as >>careful as you could. >> >>Today that is not the case. An arbitrary network >>of several hundred nodes or even a few thousand >>is not that big a deal. >> >>Joe has a point. An production network is no >>place to be doing experiments. In fact, quite >>the opposite. Experiments should be done on >>experimental networks. (to state the opposite.) >> >>Take care, >> >> >> >>At 9:24 AM -0700 7/27/13, Joe Touch wrote: >>>On Jul 26, 2013, at 9:48 PM, Jon Crowcroft wro= >>te: >>> >>>> while linux (pick your flavour) cubic isn't vanilla, neither is >>>> microsoft's compound tcp - the latter might have seen a bit more >>>> eval in the literature but the former has seen a lot more big iron >>>> deployment and doesn't appear to have broken the internet yet >>> >>>How would we know? They're not instrumented. >>>These are not experiments; they're deployments. >>> >>>Even Schr=F6dinger's cat eventually sees the light >>>of day (as much as there is a cat in the first >>>place). >>> >>>> (although there are rumours and reports of corner case problems) >>>> but i dont think either of these are "non tcp" - they are variants >>>> on CC behaviour.... >>> >>>Which is a specified standard, which these mechanisms violate. >>> >>>You do bring up a valid point about the subject >>>line, so I've changed it for this thread going >>>forward. >>> >>>> also - the ability to do any deployment testing of a new tcp in >>>> anger _requires you_ to be wireline compatible with TCP because of >>>> the "non stadard" but ubiquitous NATs and other middleboxes >>> >>>The environment doesn't support safe >>>experiments, but that is not a valid excuse for >>>unsafe ones. >>>... >>>> so the gold standard you quite reasonably want to hold people to, >>>> to show their work doesn't do harm in the wild, >>>> requires them to do "harm" by making >>>> their new variant TCP appear chameleon like, >>>> vanilla TCP, so they can get results >>> >>>So they do harm to avoid doing harm? >>> >>>They have failed because of their first step. >>> >>>Joe >> cheers jon From jon.crowcroft at cl.cam.ac.uk Sun Jul 28 01:09:19 2013 From: jon.crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Sun, 28 Jul 2013 09:09:19 +0100 Subject: [e2e] TCP "experiments" In-Reply-To: <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: we've been using overlays to run experimental protocols on the production internet for two decades - your (minor) correctness point about abuse of code points is unfortunately almost unavoidable because the internet in the wild doesn't let stuff through using new code points in enough places to get deployment - this goes back a long time (e.g. when micsosoft shipped a perfectly ok ECN enabled TCP stack only to find a bunch of their servers werent reachable due to incorrect dropping and re-writing (normalization) of various tcp options - it is extremely well illustrated in the work I mentioned (but here's the reference from NSDI 2012) on MPTCP - a perfectly respectable TCP extension which has seen a lot of careful work and refinement, and simulation, and testbed eval, only to hit barreiers to the next step, the deployment trials - see https://www.usenix.org/system/files/conference/nsdi12/nsdi12-final125.pdf I remember having discussions in both the IETF//IAB (e.g. when i was on it:) and the more "academic" research community (e.g. SIGCOMM when sally floyd and i were co chairs) 15+ years ago - back then, it was conceivable someone would build a TCP with a hidden flaw as yet undiscovvered and get the code shipped widely, then discover in the wild that very bad things happen now, we have a) lots of players who can try things in "their" part of the net, and switch back if things don't all pan out b) lots of mechanisms to roll out new code (online s/w update in all major OS, including ove the air for mobile devices) c) as I said before, many correctly aligned incentives to not break stuff gratuitously the worries about arbitrary changes to TCP breaking the whole internet the environment lets you do statistically safe experiments, and many people do - also, we are not yet quite as bad as the pharmaceutical industry where negative results from clinical trials are suppressed so we don't get to see a balanced picture (see Ben GOldacre's book, or summary of at http://en.wikipedia.org/wiki/Bad_Pharma for that story - although they are getting a lot better recently) , we tend to try to report stuff, or at least try to.... I'd note that if you were to try some really disruptive thing using some widespread deployment, you'd find yourself kicked off the net pretty fast - planetlab has for example, had its share of such stories but also a very large number of successes - the additional joy of planetlab actually being connected to the real world rather than to some academic idealised notion of what the internet might have been like, is that a LOT Of really useful lessons in scale were learned and in the process a very large number of people trained in what you can and can't do (as well as a few quite succesful startups emerging from some of the work).... http://www.planet-lab.org/ I do think you also need the testbeds and simulations (of course), but you need to go that last step to validate the work and there will always be an element of risk about it (but also we have several aforementioned mechanisms to mitigate that risk).... it has also been said that certain very large network and OS vendor companies have used the installed customer base as their test engineers, which kind of implies that what we discuss here is pretty irrelevant in practice anyhow, so I think i'll go back to my reviewing ttfn jon On Sat, Jul 27, 2013 at 5:24 PM, Joe Touch wrote: > > > On Jul 26, 2013, at 9:48 PM, Jon Crowcroft > wrote: > > > while linux (pick your flavour) cubic isn't vanilla, neither is > > microsoft's compound tcp - the latter might have seen a bit more > > eval in the literature but the former has seen a lot more big iron > > deployment and doesn't appear to have broken the internet yet > > How would we know? They're not instrumented. These are not experiments; > they're deployments. > > Even Schr?dinger's cat eventually sees the light of day (as much as there > is a cat in the first place). > > > (although there are rumours and reports of corner case problems) > > but i dont think either of these are "non tcp" - they are variants > > on CC behaviour.... > > Which is a specified standard, which these mechanisms violate. > > You do bring up a valid point about the subject line, so I've changed it > for this thread going forward. > > > also - the ability to do any deployment testing of a new tcp in > > anger _requires you_ to be wireline compatible with TCP because of > > the "non stadard" but ubiquitous NATs and other middleboxes > > The environment doesn't support safe experiments, but that is not a valid > excuse for unsafe ones. > ... > > so the gold standard you quite reasonably want to hold people to, > > to show their work doesn't do harm in the wild, > > requires them to do "harm" by making > > their new variant TCP appear chameleon like, > > vanilla TCP, so they can get results > > So they do harm to avoid doing harm? > > They have failed because of their first step. > > Joe > > From jeanjour at comcast.net Sun Jul 28 04:57:12 2013 From: jeanjour at comcast.net (John Day) Date: Sun, 28 Jul 2013 07:57:12 -0400 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: The responsibility though is to identify as many as possible before going to in vivo. It is not clear that we are seeing that. Nor is it clear that the deployments that do occur are considering the overall benefit to the network. Many of these "improvements" are only improvements because not everyone is doing them. At 6:35 AM +0200 7/28/13, Randy Bush wrote: > > One never does experiments with a production network. > >some of us do. many things are impossible to see in vitro. > >randy From jeanjour at comcast.net Sun Jul 28 04:54:34 2013 From: jeanjour at comcast.net (John Day) Date: Sun, 28 Jul 2013 07:54:34 -0400 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: At 2:18 PM +1000 7/28/13, Lachlan Andrew wrote: >Greetings John, > >On 28 July 2013 11:25, John Day wrote: >> At 10:33 AM +1000 7/28/13, Lachlan Andrew wrote: >>> >>> You are absolutely right that testbed experiments should be performed >>> before "live" experiments. However, it is not so much the size of the >>> network as the mix of applications running on it that makes the test >>> representative. >> >> Are you telling me that we don't have good statistical models for the >> behavior of large numbers of users? I would suggest that this is just >> laziness, irresponsible behavior or both. > >It depends on what you mean by "good". We have lots of models of lots >of aspects of their behaviour, but cannot possibly hope to model all >of them. Law of Large Numbers. You are splitting hairs for the sake of argument. But then I have been told that all of you believe that traffic is self-similar, so it always has the same statistical behavior. >If you go to TCPM and say "Here is an enhancement of TCP. I know it >works because I have a statistical model of it", do you think they >will be happy, however "good" your model is? I don't. > >There is a need for testing at many levels. We need to test with >simulations / models, with testbeds with synthetic users, with >testbeds with real users, and with the real internet. Otherwise, we >don't understand how things will really behave. Precisely my point. And most of these people are skipping many of those. > >>> Of course, that doesn't excuse un-monitored deployments as occurred >>> when Linux started using BIC as the default. To my mind, the solution >>> would be for the IETF to provide more practical guidance on how to >>> perform limited-scale, monitored tests on the real Internet. >> >> That is not the job of an organization dedicated to the maintenance of a >> production network. > >True. That is why I don't suggest that NANOG provide such a list. I >see the role of the IETF as to promote and guide innovation in the >Internet, rather than to maintain it. > >> Have you ever seen what researchers in other fields do to ensure that they >> get accurate and reliable results? Triple distillations, putting precisely >> the same amount in 2000 test tubes, making your own reagents? > >In physics and chemistry, experiments are done on the real physical >and chemical world. In medicine, they run clinical trials on real >people. > >As I said, I'm not proposing that people should run tests on the >public internet *instead* of doing testbed studies. I'm saying that >there comes a time when things need to be tested in reality. The IETF >has an opportunity to guide these tests. If it chooses not to guide >them, then they will be carried out without its guidance. To take your point, there indeed comes a time when the community reviewing the test results should decide that the tests warrant in vivo trials. > >> I would question whether it is the job of *researchers* to get a protocol >> approved by the IETF for use in product. > >I thought that was the role of experimental RFCs. Who will do the >experiments, if not researchers? There is a difference between doing research and the process "to get a protocol approved by the IETF." One is scientific, the other has a definite self-serving purpose. I am not saying that is bad, but there is a conflict of interest between doing research and getting approval. > >Also, I was explicitly not talking about putting things in products. >I was talking about doing experiments on the Internet. I believe that >Cubic should not have been deployed as the Linux default until it had >been tested properly. I maintain that "proper testing" includes >properly monitored experiments on the Internet. > >The issue of what "experiments" are allowed on the Internet goes to >the heart of things like the ACM's IMC. Should CAIDA not be allowed >to experiment on the Internet because it is a production network? Should EPRI be allowed to experiment with the electric grid that comes to my house? Should CAIDA or Akamai be running some kind of stress test that just happens to affect traffic down the road as a remote surgery commences? >Should companies like Akamai be allowed to repeat the (sometimes quite >burdensome) measurements on a regular basis? How often? The IETF is >welcome not to care about such things, but I was suggested that they >may have useful things to say. If they don't care about such things, >why should they care about other experiments? Since so few TCP flows >are affected by the congestion control algorithm, why is it so >sacrosanct? The last thing anything should be should be sacrosanct. However, the distinction remains between production and research. > >>> The IETF will be >>> most relevant if its processes reflect its power. >> >> But you do have a good point. I have some interesting ideas for more >> efficient use of the power grid, I think I will deploy them to see how they >> affect the balance of the grid. Gee, what could go wrong! > >The makers of electric vehicles are doing exactly that. Of course, >EVs have to be approved for electrical safety, but that doesn't do >anything to show what effect they have on the balance of the grid. >EVs are a nightmare. Does make air-conditioning during a heat-wave look like an easy problem, doesn't it? ;-) > >(As an aside, last week, at last week's IEEE Power and Energy Society >General Meeting we were told that in Alberta, the network operator has >no say in what new generation is placed where. They just have to deal >with what happens. The situation is not ideal, but is not unique to >the Internet. I assume you were being sarcastic, but if you really do >have some ideas for more efficient energy use, I certainly hope that >you pursue them. That is much more likely to help the world than any >development in cyberspace ever could.) We can leave that for another time. Take care, John > >Cheers, >Lachlan > >-- >Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) >Swinburne University of Technology, Melbourne, Australia > >Ph +61 3 9214 4837 From touch at isi.edu Sun Jul 28 10:49:15 2013 From: touch at isi.edu (Joe Touch) Date: Sun, 28 Jul 2013 10:49:15 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: <67E0CBEA-7131-40DA-8FEA-43BF9A925A91@isi.edu> On Jul 28, 2013, at 1:09 AM, Jon Crowcroft wrote: > ...your (minor) correctness point about abuse of code points is unfortunately almost unavoidable The codepoints being abused are TCP options. Those options are assigned for IETF standards. There are two current abuses: - self-assigned (squatted) codepoints - widescale deployment of experimental codepoints See draft-ietf-tcpm-experimental-options-06 for a solution to the latter, and a summary of examples of both. These have been *entirely* avoidable. The new method lowers the bar for experiments, though they should still not be deployed into the wild without monitoring. Joe From touch at isi.edu Sun Jul 28 10:53:32 2013 From: touch at isi.edu (Joe Touch) Date: Sun, 28 Jul 2013 10:53:32 -0700 Subject: [e2e] Fwd: TCP ex Machina References: <1374849317.415132309@apps.rackspace.com> Message-ID: <054318A2-AC43-4CBF-BCE4-3CF62D3B66CA@isi.edu> Forwarded as requested. Joe (as list admin) Begin forwarded message: > From: dpreed at reed.com > Subject: Re: [e2e] TCP ex Machina > Date: July 26, 2013 7:35:17 AM PDT > To: "Jon Crowcroft" > Cc: "Joe Touch" , "" > > Jon - > > Personally, your presumption that anyone is proposing an orthodoxy here is a bit stretched. That's a "when did you stop beating your wife"-style comment. > > Research is one thing. Google, in this case, is not proposing research. It is proposing *deployment* in the real world of an untested technical improvement that is proposed to solve a problem that may or may not exist at present. > > Peer-review in our field (computer and communications systems research) is highly corrupt. I realize this is a serious charge. It is not that the peer-reviewers (the people) are corrupt individuals. What I mean is that passing peer-review is taken as a stamp of "certification" by those who eagerly want to deploy systems, make money, regulate systems, etc. > > In the scientific method (Baconian or whatever) there is no "peer review" step. The gold standard is a mix of replication (which implies replicability) and correlation with other results, and a vast and deep examination of the context and validity of applicability of the results in the "real world". Peer review *CANNOT* replace this, and should never do so. (otherwise, we should hold peer-reviewers accountable for any and all harm caused by the papers they review. This cannot be achieved when most reviewers delegate the reviews to graduate students, as is the norm in our profession. > > Getting back from peer-review, I believe Joe (and I) are expressing a concern that these proposals by Google and various academics are not *research proposals*. One does not issue a research proposal by issuing a press release, or promoting the idea prior to testing on your blog as if it were *policy*. > > Given the power Google has to "just deploy it, dammit", it is, I think, responsible engineering (not science) practice for engineers whose professional concern includes impact on the public to point out that the proposals are absolutely untested, have certain risks of misbehavior, and may damage the performance or behavior of a vast number of *already deployed* applications. > > It would please me greatly if Joe would allow this message to be posted on e2e, just this once. > > I think we should be very clear that the Internet as it is today is not a science-project, but a real public work, even though it is not the work of a government and probably should never be "governed" (even by the ITU). > > Engineering responsibility (not the scientific method alone) should be at least as important in its evolution. > > That doesn't mean that new ideas should not be tested scientifically and deployed experimentally. But we should be *careful* not to conflate "orthodoxy" with engineering caution. > > And it is perfectly fine, in my mind, to challenge those who would *promote* ideas that are not even tested (only simulated in toy environments or run in testbeds that are far from the real world). > > The sad thing about our "science" is that the presumption that network traffic is generated by independent, simple Markov processes and are Gaussian (in the sense that aggregation reduces variance) is not even questioned by those who do peer review, much less those who have the experimental disproof of this staring them in the face (academics). > > Our field has a bad case of *dogmatism* in its experimental assumptions. Not orthodoxy, but an arrogance that because an assumption makes a problem appear to be tractable mathematically that the assumption is true. > > That dogmatism is shared with neo-classical economic theory. Despite years of being horribly wrong, we still suffer from the arrogance of that field. > > Let's not let the field we share become like that. It's a danger to society now that the Internet is central. > > > > On Friday, July 26, 2013 12:53am, "Jon Crowcroft" said: > > > I'm sorry, but this smacks of "orthodoxy" > > which is anathema to good science. > > > > the reality is that the nature of > > most of the "optimization" work that makes it into good conferences > > (nsdi, icnp, sigcomm, pick your fave) > > are in the nature of the word you use - extensions > > > > they do not "sidestep" anything - they change the behaviour in the > > situation where the extesion applies and not otherwise, and are > > typically "backwards compatible" as much as possible, as the one > > true orthodoxy that is around is that any decent paper evaluates > > the tcp-atlantis or tcp-erewhon both in the new scenario, and in a > > mix with tcp-vanilla or tcp-realworld, to see what the impact of > > each on the other is, because thats what scientists do when they do > > their evaluation properly - they want to know what happens in the > > real world, not just in some bubble world > > > > however, it is also legitimate science > > to ask what happens in some brave new world, > > if one has the capability to change everything - > > > > some of these brave new worlds aren't small - > > they are places like all cell phones, > > or in a data center with a few million cores, > > where one might actually have the capability > > to have a flag day > > (ship the tcp-flagrante to all devices and enable)... > > > > that new world might not share the same primary goal - > > > > for example, in some data center it might be > > perfectly reasonable to sidestep your goal > > as the modest owner of the data center might actually > > want to maximise total crosssectional throughput > > for some odd reason... > > > > I just dont buy the homogenous trope any more - > > what is more, we have dynamic systems that > > rebuild, relink, reload, reboot - > > we can decide which is the appropriate behaviour - > > indeed, many of these systems do this anyhow at some level > > (when switching from wifi to 3g/4g, > > when switching from hadoop using tcp to do reduce/shuffle phase, > > to video streaming over tcp to a customer - > > in lots of other cases) > > > > for these people, > > what is wrong with experimenting and publishing > > papers on the results of the experiments? it would be wrong > > not to publically propose and expose evaluations of these > > extensions or modifications. > > > > in any case, > > TCP and IP were not perfect at birth > > through some sort of intelligent design > > which magically foresaw all these new scenarios- > > > > they were prototypes that expressed some very cool ideas, > > but missed some details and I believe we learned some new stuff > > since anyhow... > > > > so of course there might be extensions which would kick in, > > in the wrong circumstances and do harm - > > > > so do you believe any OS vendor or > > community would ship those? > > > > do you believe that unless the papers about them showed a net win, > > anyone would use them? > > > > do you think we shouldn't propose ideas, > > that occasionally turn out to be wrong? > > how else will things move on? > > > > in any case, the TCP ex Machina paper isn't this at all > > so on the topic of evolving (and evaluating in a complex rich way) > > new variants of CC, I think one way you could think about this paper > > is the following > > > > instead of waiting a duty cycle of 1 grad-student-year for a next > > TCP congestion control idea, > > you use a big cluster of machines and evolutionary computing > > to run at the rate of around 100 graduate students-per-week > > and pick the best according to some metric - > > > > you could, if you like, pick a different metric - Fred's proposed > > one - that's fine - the paper just illustrates that we have a > > capability to do some thing which was probably rather difficult to > > consider in 1976 or 1988 (pick your preferred incept date here)... > > > > I'd quite like DNS lookup and search to go faster anyhow- but i > > dont think the people working on that want the download times of > > the PDFs of their papers to slow down or be unrelable as a > > result... > > > > Do you have specific examples of work that would do that? > > > > cheers > > jon > > > > > > In missive <51F19708.3050206 at isi.edu>, Joe Touch typed: > > > > >>FWIW, I've been concerned about a number of proposed "extensions" to > > >>TCP, either to optimize it for DNS (TCP-CT) or for Google (various). > > >> > > >>My concern is because these optimizations sidestep my view of the > > >>primary goal of TCP as described below. > > >> > > >>Joe > > >> > > >>On 7/25/2013 2:00 PM, Joe Touch wrote: > > >>> > > >>> > > >>> On 7/21/2013 11:35 PM, Fred Baker (fred) wrote: > > >>>> But I think maximizing throughput while minimizing the > > probability of > > >>>> loss is probably the holy grail. > > >>> > > >>> It also depends on whether you measure that as an individual or as a > > group. > > >>> > > >>> I tend to think of TCP as supporting the most participants having > > >>> reliable communication as a primary goal, and a secondary goal to > > >>> maximize (though not optimize) the aggregate data per unit time -- > > >>> subject to the primary goal. > > >>> > > >>> I.e., it's not about max BW for a given connection, and it also > > almost > > >>> never means minimizing the transfer time for a single connection. > > >>> > > >>> Joe > > > > cheers > > > > jon > > > > From Alexander.Zimmermann at netapp.com Sun Jul 28 11:43:44 2013 From: Alexander.Zimmermann at netapp.com (Zimmermann, Alexander) Date: Sun, 28 Jul 2013 18:43:44 +0000 Subject: [e2e] TCP ex Machina In-Reply-To: <51F2B580.9040703@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> Message-ID: <12208140-C0CF-4D83-915C-B070532ACB1D@netapp.com> Hi Joe, Am 26.07.2013 um 19:44 schrieb Joe Touch : > Hi, Jon, > > On 7/25/2013 9:53 PM, Jon Crowcroft wrote: >> I'm sorry, but this smacks of "orthodoxy" >> which is anathema to good science. > > I'm not referring to science, or research. I'm referring to changes to TCP that are being pushed for deployment or already deployed. > > E.g., TCP-CT is not an approved IETF standard. It was not taken to through the IETF process. But it is *deployed* in Linux. Fortunately, TCP-CT is not part of Linux anymore. http://www.spinics.net/lists/netdev/msg229446.html Alex > As an example of bad behavior, TCP-CT uses a TCP option codepoint that is reserved for experiments that are not widely deployed. > >> the reality is that the nature of >> most of the "optimization" work that makes it into good conferences >> (nsdi, icnp, sigcomm, pick your fave) >> are in the nature of the word you use - extensions > > There are different kinds of TCP extensions, falling into at least two broad categories: > > A- extensions to TCP itself > A1- some intended to make TCP more robust or address > a corner case > A2- some intended to correct a problem in TCP's design > A3- some intended to support a specific use case > > B- using the TCP protocol ID and coarsely conforming to TCP's > connection establishment > > I'm not referring to A1 or A2, which includes some published research. > > I'm concerned more about B, and somewhat about A3. > > B-style extensions are really new transport protocols. That's a valid area of research, but the deployment reality is that no such new transports can traverse widely-deployed non-standard network elements (e.g., NATs). So B is really a new transport masquerading as TCP. > > A3 are extensions to optimize specific use cases. The trouble with such optimizations is that they can easily undermine a key feature of TCP - that it falls back to working nearly all the time it possibly can. > > I don't consider breaking that key property of TCP a good thing. > > Experimentation is fine, as is research. But the medical community doesn't start playing with new meds by handing them out on street corners like candy. > > ... >> so of course there might be extensions which would kick in, >> in the wrong circumstances and do harm - >> >> so do you believe any OS vendor or >> community would ship those? > > Linux. Cisco. Alcatel-Lucent. This is a partial list of vendors who have deployed code that uses *the same* experimental codepoint. > >> do you believe that unless the papers about them showed a net win, >> anyone would use them? > > Yes. Linux has done this repeatedly. > > ... >> instead of waiting a duty cycle of 1 grad-student-year for a next >> TCP congestion control idea, >> you use a big cluster of machines and evolutionary computing >> to run at the rate of around 100 graduate students-per-week >> and pick the best according to some metric - >> >> you could, if you like, pick a different metric > > I did, and you took issue with it. > > Joe > From touch at isi.edu Mon Jul 29 10:25:12 2013 From: touch at isi.edu (Joe Touch) Date: Mon, 29 Jul 2013 10:25:12 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> Message-ID: <51F6A578.5010000@isi.edu> On 7/27/2013 7:20 PM, Matt Mathis wrote: > The real issue is the diversity of implementations in the Internet that > allege to be standard IP and TCP NAT, but contain undocumented "features". > No level of simulation has any hope of predicting how well a new protocol > feature or congestion control algorithm will actually function in the real > Internet - you have to measure it. > > Furthermore: given that Google gets most of its revenue from clicks, how > much might it cost us to "deploy" a protocol feature that caused 0.01% > failure rate? If you were Google management, how large of sample size > would you want to have before you might be willing actually deploy > something globally? I don't understand the logic above, summarized IMO as: - big deployment is required to find issues - issues are found by measurement - Google doesn't want to deploy protocols that cause failures Take that with the current actions: - Google management is comfortable deploying protocols that are not instrumented Do you see why the rest of us are concerned? Besides, there are different kinds of failures. I doubt Google wants a 0.01% failure of its current base, but if the feature increases its base by 0.02%, do you really think it wouldn't go ahead and deploy? And worse, what if the 0.01% failure wasn't to Google connections, but a competitor? Or just the rest of the Internet? My view has been that we need to protect the Internet for *when* there is no more Google. I don't trust Google (or any company) to do that. Joe From Jon.Crowcroft at cl.cam.ac.uk Mon Jul 29 14:53:18 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Mon, 29 Jul 2013 22:53:18 +0100 Subject: [e2e] TCP "experiments" In-Reply-To: <51F6A578.5010000@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> Message-ID: In missive <51F6A578.5010000 at isi.edu>, Joe Touch typed: >>My view has been that we need to protect the Internet for *when* there >>is no more Google. I don't trust Google (or any company) to do that. why do we trust you? jon From touch at isi.edu Mon Jul 29 16:05:07 2013 From: touch at isi.edu (Joe Touch) Date: Mon, 29 Jul 2013 16:05:07 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> Message-ID: <51F6F523.6060107@isi.edu> On 7/29/2013 2:53 PM, Jon Crowcroft wrote: > In missive <51F6A578.5010000 at isi.edu>, Joe Touch typed: > > >>My view has been that we need to protect the Internet for *when* there > >>is no more Google. I don't trust Google (or any company) to do that. > > why do we trust you? You shouldn't. More specifically, "trust but verify". Un-instrumented changes are not verifiable. Joe From mattmathis at google.com Mon Jul 29 22:26:59 2013 From: mattmathis at google.com (Matt Mathis) Date: Mon, 29 Jul 2013 22:26:59 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: <51F6A578.5010000@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> Message-ID: Good point about potential unintended consequences of shipping new features with the instrumentation stripped.... Perhaps we should rethink that one little detail. Global scale experiments are always geometric spirals, iterating between developing and testing across an exponentially growing pool of users. As the pool grows you keep looking for unexpected behaviors etc. and move on when you convince yourselves that they have been addressed and all old features have been properly regression tested (e.g. packetdrill). The really critical question is at what point do you (the researcher/developer) lose control of your own legacy and we all become forced to maintain compatibility with code that has escaped. If you look at all of the stuff that we have done (both in transport and above), I think you will find that we have a pretty good handle on making sure that obsolete code actually gets deprecated and does not persist. And that is what justifies calling it an experiment. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. On Mon, Jul 29, 2013 at 10:25 AM, Joe Touch wrote: > > > On 7/27/2013 7:20 PM, Matt Mathis wrote: > >> The real issue is the diversity of implementations in the Internet that >> allege to be standard IP and TCP NAT, but contain undocumented "features". >> No level of simulation has any hope of predicting how well a new >> protocol >> feature or congestion control algorithm will actually function in the real >> Internet - you have to measure it. >> >> Furthermore: given that Google gets most of its revenue from clicks, how >> much might it cost us to "deploy" a protocol feature that caused 0.01% >> failure rate? If you were Google management, how large of sample size >> would you want to have before you might be willing actually deploy >> something globally? >> > > I don't understand the logic above, summarized IMO as: > > - big deployment is required to find issues > - issues are found by measurement > - Google doesn't want to deploy protocols that cause failures > > Take that with the current actions: > > - Google management is comfortable deploying protocols > that are not instrumented > > Do you see why the rest of us are concerned? > > Besides, there are different kinds of failures. I doubt Google wants a > 0.01% failure of its current base, but if the feature increases its base by > 0.02%, do you really think it wouldn't go ahead and deploy? > > And worse, what if the 0.01% failure wasn't to Google connections, but a > competitor? Or just the rest of the Internet? > > My view has been that we need to protect the Internet for *when* there is > no more Google. I don't trust Google (or any company) to do that. > > Joe > > From Jon.Crowcroft at cl.cam.ac.uk Mon Jul 29 23:14:44 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Tue, 30 Jul 2013 07:14:44 +0100 Subject: [e2e] TCP "experiments" In-Reply-To: <51F6F523.6060107@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> Message-ID: google have extremely good instrumentation - you need to go visit them unfortunately, you'd have to sign NDAs but believe me, they know what they are about a lot more than most academics and a whole lot of other industry - some of the stuff is quite public too (lots of SPDY testing is on their blogs and vairous public facing documents) their shareholders would get upset if they deployed things that broke the world badly - that's the NSA's job. In missive <51F6F523.6060107 at isi.edu>, Joe Touch typed: >> >> >>On 7/29/2013 2:53 PM, Jon Crowcroft wrote: >>> In missive <51F6A578.5010000 at isi.edu>, Joe Touch typed: >>> >>> >>My view has been that we need to protect the Internet for *when* there >>> >>is no more Google. I don't trust Google (or any company) to do that. >>> >>> why do we trust you? >> >>You shouldn't. >> >>More specifically, "trust but verify". >> >>Un-instrumented changes are not verifiable. >> >>Joe cheers jon From touch at isi.edu Mon Jul 29 23:39:59 2013 From: touch at isi.edu (Joe Touch) Date: Mon, 29 Jul 2013 23:39:59 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> Message-ID: <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> On Jul 29, 2013, at 11:14 PM, Jon Crowcroft wrote: > google have extremely good instrumentation - They can now see how my PC talks to amazon.com now? > their shareholders would get upset if they deployed things that broke > the world badly - Even if they made Google 0.01% faster? And increased ad revenue as a result? And stock value or dividends? > that's the NSA's job. Sound like you're referring to my first point above :-) Joe From ashish.makani at gmail.com Tue Jul 30 00:05:18 2013 From: ashish.makani at gmail.com (ashish makani) Date: Tue, 30 Jul 2013 00:05:18 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> Message-ID: On Mon, Jul 29, 2013 at 11:14 PM, Jon Crowcroft wrote: > google have extremely good instrumentation - you need to go visit them > unfortunately, you'd have to sign NDAs but believe me, they know > what they are about a lot more than most academics and a whole lot of > other industry - some of the stuff is quite public too (lots of SPDY > testing is on their blogs and vairous public facing documents) > > > their shareholders would get upset if they deployed things that broke >> the world badly - that's the NSA's job. > > That's the NSA's job - lol :) True that Jon :) > > In missive <51F6F523.6060107 at isi.edu>, Joe Touch typed: > > >> > >> > >>On 7/29/2013 2:53 PM, Jon Crowcroft wrote: > >>> In missive <51F6A578.5010000 at isi.edu>, Joe Touch typed: > >>> > >>> >>My view has been that we need to protect the Internet for *when* > there > >>> >>is no more Google. I don't trust Google (or any company) to do > that. > >>> > >>> why do we trust you? > >> > >>You shouldn't. > >> > >>More specifically, "trust but verify". > >> > >>Un-instrumented changes are not verifiable. > >> > >>Joe > > cheers > > jon > > From Jon.Crowcroft at cl.cam.ac.uk Tue Jul 30 01:34:21 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Tue, 30 Jul 2013 09:34:21 +0100 Subject: [e2e] TCP "experiments" In-Reply-To: <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> Message-ID: the nice thing about a lot of the big cloud/web service outfits is that they run a huge _range_ of applications so a point solution tcp optimised for just one thing that harmed the common case uses of TCP would be ruled out - if you think about the mix of map/reduce, video streaming, social net, searches, gmail, gfs, google docs, etc etc (and similar mixes for microsoft bing + hotmail + azure etc; and similar for apple iCloud, appstore, and similar for, oh, i dunno, maybe even facebook), then I doubt very much we'd see them do something as narrowly dumb as people seem to imply here (or deploy someone else's dumb narrow hack either).... as for impact between different cloud/web service providers (e.g. google being smart enough to optimise a TCP hack for all that stuff but still harm Amazon's traffic for book/cd buying, EC2, cloud music player etc etc, that would be incredibly ingeneiously dumb, but also unlikely, since a major google's revenue depends on people finding stuff on _other servers, and finding those other services useful, so it would be kind of silly to auction advert space to the highest bidder iand deliver those adverts in a way that broke the advertised things.... [I suppose you could deliver adverts that broke TCP services to servers that hadn't paid you to advertise them - that'd be pretty super duper whacky type of cyberwarfare game some folks at the NSA probably have fun thinking up..... maybe Huawei could deploy some DPI-based filter to harm succesful businesses only so as to bring down the whole capitalist imperialist western civilisation....oh no, wait, they need our net to work so they can sell us their routers... In missive <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7 at isi.edu>, Joe Touch typed: >> >> >>On Jul 29, 2013, at 11:14 PM, Jon Crowcroft wro= >>te: >> >>> google have extremely good instrumentation - >> >>They can now see how my PC talks to amazon.com now? >> >>> their shareholders would get upset if they deployed things that broke >>> the world badly - >> >>Even if they made Google 0.01% faster? And increased ad revenue as a result?= >> And stock value or dividends? >> >>> that's the NSA's job. >> >>Sound like you're referring to my first point above :-) >> >>Joe=20 >> cheers jon From touch at isi.edu Tue Jul 30 10:24:57 2013 From: touch at isi.edu (Joe Touch) Date: Tue, 30 Jul 2013 10:24:57 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> Message-ID: <51F7F6E9.4040107@isi.edu> On 7/30/2013 1:34 AM, Jon Crowcroft wrote: > the nice thing about a lot of the big cloud/web service outfits > is that they run a huge _range_ of applications so a point solution > tcp optimised for just one thing that > harmed the common case uses of TCP would be ruled out - Large back-end stuff can run privately optimized TCP - or even opt out of TCP and use WDM (see the UCSD Sigcomm pubs). My concern is that Google would easily trade 0.01% benefit from 1 billion users over even a 10% decrease in performance internally if the economics demanded it. IMO, too many mods to TCP are made in the name of "it's only 0.01% improvement, but that's important to X" (where X is Google, Netflix, etc.). And not enough of these mods are instrumented in a way that we can see their impact. For the difference in approach, see RFC 6928 vs. draft-touch-automatic-iw. The former increases the IW as a fixed value because it helps Google's short connections. The latter is self-instrumented to react to whether the large IW works and adapt it accordingly. Joe From mellia at tlc.polito.it Tue Jul 30 12:13:15 2013 From: mellia at tlc.polito.it (Marco Mellia) Date: Tue, 30 Jul 2013 12:13:15 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> Message-ID: Jon, I think no one is saying that big companies are doing dumb things. It's just that the Internet is a really shared infrastructure. Unless resources are infinite (which may be actually the case for Google :)), gaining something somewhere comes at the expenses of loosing something else somewhere else. Considering TCP, it is easy to show that "I can gain". It is much harder to show "who else is loosing". And weighting the plus and the minus is even more complicated. M -- Marco Mellia - Assistant Professor Dipartimento di Elettronica e Telecomunicazioni Politecnico di Torino Corso Duca Degli Abruzzi 24 10129 - Torino - IT Tel: +39-011-090-4173 Cel: +39-331-6714789 Skype: mgmellia Home page: http://www.tlc-networks.polito.it/mellia On Jul 30, 2013, at 1:34 AM, Jon Crowcroft wrote: > the nice thing about a lot of the big cloud/web service outfits > is that they run a huge _range_ of applications so a point solution > tcp optimised for just one thing that > harmed the common case uses of TCP would be ruled out - > > if you think about the mix of map/reduce, > video streaming, social net, searches, gmail, gfs, google docs, etc etc > (and similar mixes for microsoft bing + hotmail + azure etc; > and similar for apple iCloud, appstore, and similar for, > oh, i dunno, maybe even facebook), > then I doubt very much we'd see them do something > as narrowly dumb as people seem to imply here > (or deploy someone else's dumb narrow hack either).... > > as for impact between different cloud/web service providers > (e.g. google being smart enough to optimise a TCP hack for all that stuff > but still harm Amazon's traffic for book/cd buying, EC2, cloud music player > etc etc, that would be incredibly ingeneiously dumb, > but also unlikely, since a major google's revenue depends on > people finding stuff on _other servers, > and finding those other services useful, > so it would be kind of silly to auction advert space to the highest bidder > iand deliver those adverts in a way that broke the advertised things.... > > [I suppose you could deliver adverts that broke TCP services to > servers that hadn't paid you to advertise them - that'd be pretty > super duper whacky type of cyberwarfare game some folks at the NSA > probably have fun thinking up..... > > maybe Huawei could deploy some DPI-based filter to harm succesful businesses only > so as to bring down the whole capitalist imperialist western civilisation....oh no, > wait, they need our net to work so they can sell us their routers... > > In missive <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7 at isi.edu>, Joe Touch typed: > >>> >>> >>> On Jul 29, 2013, at 11:14 PM, Jon Crowcroft wro= >>> te: >>> >>>> google have extremely good instrumentation - >>> >>> They can now see how my PC talks to amazon.com now? >>> >>>> their shareholders would get upset if they deployed things that broke >>>> the world badly - >>> >>> Even if they made Google 0.01% faster? And increased ad revenue as a result?= >>> And stock value or dividends? >>> >>>> that's the NSA's job. >>> >>> Sound like you're referring to my first point above :-) >>> >>> Joe=20 >>> > > cheers > > jon > From Jon.Crowcroft at cl.cam.ac.uk Wed Jul 31 01:01:39 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Wed, 31 Jul 2013 09:01:39 +0100 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> Message-ID: if someone is just making the sender side send faster no matter what the circumstance then you'd be right - but a lot of TCP changes make it more _efficient_ which benefits everyone - for example TCP Fast Open means less packets on the net, which is a win-win scenario (lower latency for this user, less in the queues for other users)... so a lot of the things people at google and other places are trying to do are universally beneficial imho.....why would they not be? In missive , Marco Mellia typed: >>Jon, >> >>I think no one is saying that big companies are doing dumb things. It's = >>just that the Internet is a really shared infrastructure. >>Unless resources are infinite (which may be actually the case for Google = >>:)), gaining something somewhere comes at the expenses of loosing = >>something else somewhere else. >> >>Considering TCP, it is easy to show that "I can gain". It is much harder = >>to show "who else is loosing". >>And weighting the plus and the minus is even more complicated. >> >>M >> >> >>--=20 >>Marco Mellia - Assistant Professor >>Dipartimento di Elettronica e Telecomunicazioni >>Politecnico di Torino >>Corso Duca Degli Abruzzi 24 >>10129 - Torino - IT >>Tel: +39-011-090-4173 >>Cel: +39-331-6714789 >>Skype: mgmellia >>Home page: http://www.tlc-networks.polito.it/mellia >> >>On Jul 30, 2013, at 1:34 AM, Jon Crowcroft = >>wrote: >> >>> the nice thing about a lot of the big cloud/web service outfits=20 >>> is that they run a huge _range_ of applications so a point solution=20 >>> tcp optimised for just one thing that=20 >>> harmed the common case uses of TCP would be ruled out -=20 >>>=20 >>> if you think about the mix of map/reduce,=20 >>> video streaming, social net, searches, gmail, gfs, google docs, etc = >>etc=20 >>> (and similar mixes for microsoft bing + hotmail + azure etc;=20 >>> and similar for apple iCloud, appstore, and similar for, >>> oh, i dunno, maybe even facebook),=20 >>> then I doubt very much we'd see them do something >>> as narrowly dumb as people seem to imply here=20 >>> (or deploy someone else's dumb narrow hack either).... >>>=20 >>> as for impact between different cloud/web service providers >>> (e.g. google being smart enough to optimise a TCP hack for all that = >>stuff >>> but still harm Amazon's traffic for book/cd buying, EC2, cloud music = >>player >>> etc etc, that would be incredibly ingeneiously dumb,=20 >>> but also unlikely, since a major google's revenue depends on=20 >>> people finding stuff on _other servers,=20 >>> and finding those other services useful,=20 >>> so it would be kind of silly to auction advert space to the highest = >>bidder=20 >>> iand deliver those adverts in a way that broke the advertised = >>things.... >>>=20 >>> [I suppose you could deliver adverts that broke TCP services to >>> servers that hadn't paid you to advertise them - that'd be pretty >>> super duper whacky type of cyberwarfare game some folks at the NSA >>> probably have fun thinking up..... >>>=20 >>> maybe Huawei could deploy some DPI-based filter to harm succesful = >>businesses only >>> so as to bring down the whole capitalist imperialist western = >>civilisation....oh no, >>> wait, they need our net to work so they can sell us their routers... >>>=20 >>> In missive <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7 at isi.edu>, Joe Touch = >>typed: >>>=20 >>>>>=20 >>>>>=20 >>>>> On Jul 29, 2013, at 11:14 PM, Jon Crowcroft = >> wro=3D >>>>> te: >>>>>=20 >>>>>> google have extremely good instrumentation - >>>>>=20 >>>>> They can now see how my PC talks to amazon.com now? >>>>>=20 >>>>>> their shareholders would get upset if they deployed things that = >>broke >>>>>> the world badly - >>>>>=20 >>>>> Even if they made Google 0.01% faster? And increased ad revenue as a = >>result?=3D >>>>> And stock value or dividends? >>>>>=20 >>>>>> that's the NSA's job. >>>>>=20 >>>>> Sound like you're referring to my first point above :-) >>>>>=20 >>>>> Joe=3D20 >>>>>=20 >>>=20 >>> cheers >>>=20 >>> jon >>>=20 >> cheers jon From ymjiang at ieee.org Wed Jul 31 02:34:57 2013 From: ymjiang at ieee.org (Yuming Jiang) Date: Wed, 31 Jul 2013 11:34:57 +0200 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> Message-ID: <51F8DA41.8030409@ieee.org> Hi, For your possible interest, we have conducted a simulation study to see if/how different TCP variants may be fair to each other under coexistence. Some early results can be found from the following paper available from ACM DL: Addisu Eshete, Yuming Jiang, Lars Landmark: Fairness among high speed and traditional TCP under different queue management mechanisms. AINTEC 2012: 39-46 Yuming -- Yuming Jiang Department of Telematics Norwegian University of Science and Technology (NTNU) Phone: +47 73592724 Fax: +47 73596973 E-mail: jiang at item.ntnu.no http://www.item.ntnu.no/people/personalpages/fac/jiang/ From richard at richardclegg.org Wed Jul 31 05:47:23 2013 From: richard at richardclegg.org (Richard G. Clegg) Date: Wed, 31 Jul 2013 13:47:23 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> Message-ID: <51F9075B.9050404@richardclegg.org> On 22/07/2013 08:04, Jon Crowcroft wrote: > Richard Clegg at UCL presented a nice study recently where he has being > trying to fit TCP equations to lots if big packet traces...he found > precious few loss events, and the best explanation appears to be that the > world is made of many zebras and mice....zebras are flows that are > application rate limited...e.g video streaming...mice are flows that just > don't make it out if slow start before they are done...eg web transactions, > social media chat etc...it appears TCP CC is rarely invoked these days...... Jon, Thanks for the kind mention. I didn't reply earlier as I was travelling in parts of the world with no Internet coverage. The work I presented comes from two papers, one publicly available, one under submission right now: This paper was presented at ICC: On the relationship between fundamental measurements in TCP flows: http://www.richardclegg.org/pubs/rgc_icc_2013.pdf It's a simple model fitting exercise where we look at a large number of publicly available TCP traces (mainly CAIDA but some MAWI) and try to fit models inspired by the Padhye et al form that a TCP connection bandwidth is proportional to 1/RTT and 1/sqrt(p) where p is probability of packet loss. There was a burst of papers of this form some years ago which took a mathematical model of TCP and tried to get a formula for throughput given assumptions. We took the reverse approach and fitted a large number of packet traces (several billion packets in all) to see how the throughput of a flow depended on loss, RTT and length of flow. So the idea was to take the observational statistics approach of assuming that we knew nothing about TCP and trying to reverse engineer what factors we can observe in a TCP flow which cause it to have high or low throughput. What was interesting to me was that 1) Length of flow was very important to the model fit -- even if you filter out short flows to get rid of slow start. 2) Proportion of loss in a flow was not so important, indeed hardly relevant at all. There was little correlation between loss and throughput (we used a variety of traces, some low loss, some high loss). 3) Often it is sufficient to simply know RTT and length of flow in order to make a good prediction of the completion time for a flow. The second paper (under submission) was led by Joao Araujo here at UCL. It looks at mechanisms which affect TCP throughput but are not loss or delay based. E.g. throttling by application (youtube, clickhosts and P2P hosts often do this by various mechanims), hitting max window size (either because of OS limits at client or because server limits it to cap bandwidth) or throttling by middleboxes. This used the MAWI data from Japan and showed that more than half of modern TCP traffic in that data is not really "traditional" TCP as we think of it (attempting to "fill the pipe" using loss or delay as a signal to stop doing so) and the fraction of TCP controlled by other mechanisms appears to be growing. Joao would be the best person to answer questions about the work j.araujo at ucl.ac.uk -- but I found it very interesting. Others would have better insight than I but I was surprised. If more than half the traffic is not at heart controlled by loss/delay then what does this imply for all of our insights on fairness between flows? It seems the majority of traffic we considered is in various ways either deliberately (youtube, clickhost) or accidentally (OS limited at client) controlled in a different way to the way that we teach in the class room. -- Richard G. Clegg, Dept of Elec. Eng., University College London http://www.richardclegg.org/ From mellia at tlc.polito.it Wed Jul 31 12:05:34 2013 From: mellia at tlc.polito.it (Marco Mellia) Date: Wed, 31 Jul 2013 12:05:34 -0700 Subject: [e2e] TCP "experiments" In-Reply-To: References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> Message-ID: <2A89DA46-85CA-4261-AF2E-D15DE4222ED4@tlc.polito.it> > if someone is just making the sender side send faster no matter what the circumstance > then you'd be right - but a lot of TCP changes make it more _efficient_ which benefits > everyone - for example TCP Fast Open means less packets on the net, which is a > win-win scenario (lower latency for this user, less in the queues for other users)? > Agree > so a lot of the things people at google and other places are trying to do > are universally beneficial imho.....why would they not be? If you want an example where a change would be not "fair", you can consider YouTube. Now they start to use more than one TCP connection to serve the video. This avoids that the application gets stuck because the TCP connection gets stuck. In case of competing traffic (and congestion), YouTube client will download in parallel from multiple connections (we have seen up to 5). Thus sharing capacity is not fair against other concurrent traffic. You may say that Video is not best effort, so why not. But in a scenario where you use YouTube and I use MyTube, you'll see, and I'll not see :( You are more aggressive, and get a larger share of capacity. Same for TCP initial window. You use 10, I use 2. You are more aggressive. I suffer from additional losses/delay because of you (in case we share the same link). As said, weighting the pros against the cons is not easy. Anyway, I found it funny that we can discuss this forever. But actually no one has the right/power/control of imposing anything on the Internet. Basically, it's a perfect anarchy, where everyone is allowed to do/deploy whatever he thinks it's good for him. I push more, I get more. I don't care if you get hurt? Would be nice to have a system where some judge can say that you are pushing too much, and provide a punishment for you. But this seems quite impossible here? Ciao M > In missive , Marco Mellia typed: > >>> Jon, >>> >>> I think no one is saying that big companies are doing dumb things. It's = >>> just that the Internet is a really shared infrastructure. >>> Unless resources are infinite (which may be actually the case for Google = >>> :)), gaining something somewhere comes at the expenses of loosing = >>> something else somewhere else. >>> >>> Considering TCP, it is easy to show that "I can gain". It is much harder = >>> to show "who else is loosing". >>> And weighting the plus and the minus is even more complicated. >>> >>> M >>> >>> >>> --=20 >>> Marco Mellia - Assistant Professor >>> Dipartimento di Elettronica e Telecomunicazioni >>> Politecnico di Torino >>> Corso Duca Degli Abruzzi 24 >>> 10129 - Torino - IT >>> Tel: +39-011-090-4173 >>> Cel: +39-331-6714789 >>> Skype: mgmellia >>> Home page: http://www.tlc-networks.polito.it/mellia >>> >>> On Jul 30, 2013, at 1:34 AM, Jon Crowcroft = >>> wrote: >>> >>>> the nice thing about a lot of the big cloud/web service outfits=20 >>>> is that they run a huge _range_ of applications so a point solution=20 >>>> tcp optimised for just one thing that=20 >>>> harmed the common case uses of TCP would be ruled out -=20 >>>> =20 >>>> if you think about the mix of map/reduce,=20 >>>> video streaming, social net, searches, gmail, gfs, google docs, etc = >>> etc=20 >>>> (and similar mixes for microsoft bing + hotmail + azure etc;=20 >>>> and similar for apple iCloud, appstore, and similar for, >>>> oh, i dunno, maybe even facebook),=20 >>>> then I doubt very much we'd see them do something >>>> as narrowly dumb as people seem to imply here=20 >>>> (or deploy someone else's dumb narrow hack either).... >>>> =20 >>>> as for impact between different cloud/web service providers >>>> (e.g. google being smart enough to optimise a TCP hack for all that = >>> stuff >>>> but still harm Amazon's traffic for book/cd buying, EC2, cloud music = >>> player >>>> etc etc, that would be incredibly ingeneiously dumb,=20 >>>> but also unlikely, since a major google's revenue depends on=20 >>>> people finding stuff on _other servers,=20 >>>> and finding those other services useful,=20 >>>> so it would be kind of silly to auction advert space to the highest = >>> bidder=20 >>>> iand deliver those adverts in a way that broke the advertised = >>> things.... >>>> =20 >>>> [I suppose you could deliver adverts that broke TCP services to >>>> servers that hadn't paid you to advertise them - that'd be pretty >>>> super duper whacky type of cyberwarfare game some folks at the NSA >>>> probably have fun thinking up..... >>>> =20 >>>> maybe Huawei could deploy some DPI-based filter to harm succesful = >>> businesses only >>>> so as to bring down the whole capitalist imperialist western = >>> civilisation....oh no, >>>> wait, they need our net to work so they can sell us their routers... >>>> =20 >>>> In missive <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7 at isi.edu>, Joe Touch = >>> typed: >>>> =20 >>>>>> =20 >>>>>> =20 >>>>>> On Jul 29, 2013, at 11:14 PM, Jon Crowcroft = >>> wro=3D >>>>>> te: >>>>>> =20 >>>>>>> google have extremely good instrumentation - >>>>>> =20 >>>>>> They can now see how my PC talks to amazon.com now? >>>>>> =20 >>>>>>> their shareholders would get upset if they deployed things that = >>> broke >>>>>>> the world badly - >>>>>> =20 >>>>>> Even if they made Google 0.01% faster? And increased ad revenue as a = >>> result?=3D >>>>>> And stock value or dividends? >>>>>> =20 >>>>>>> that's the NSA's job. >>>>>> =20 >>>>>> Sound like you're referring to my first point above :-) >>>>>> =20 >>>>>> Joe=3D20 >>>>>> =20 >>>> =20 >>>> cheers >>>> =20 >>>> jon >>>> =20 >>> > > cheers > > jon > From lachlan.andrew at gmail.com Wed Jul 31 17:10:04 2013 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Thu, 1 Aug 2013 10:10:04 +1000 Subject: [e2e] TCP "experiments" In-Reply-To: <2A89DA46-85CA-4261-AF2E-D15DE4222ED4@tlc.polito.it> References: <51EC4468.1010105@web.de> <7C62F84D-A881-4609-ABB9-9E700F3896FF@isi.edu> <8C48B86A895913448548E6D15DA7553B94661D@xmb-rcd-x09.cisco.com> <51F191D0.6030807@isi.edu> <51F19708.3050206@isi.edu> <51F2B580.9040703@isi.edu> <1CCA5689-7C66-4F43-856B-F3CC8E0B6548@isi.edu> <51F6A578.5010000@isi.edu> <51F6F523.6060107@isi.edu> <2EBB07CA-C011-4DC4-AACD-5A9D959C59D7@isi.edu> <2A89DA46-85CA-4261-AF2E-D15DE4222ED4@tlc.polito.it> Message-ID: On 1 August 2013 05:05, Marco Mellia wrote: > > If you want an example where a change would be not "fair", you can consider YouTube. > Now they start to use more than one TCP connection to serve the video. That is the canonical example of what is wrong with the approach of TCPM. TCPM is extremely reluctant to allow improvements to the behaviour of a single TCP flow, and so users open multiple connections. Each flow is "standards compliant", but the net result is IMHO much worse than if a single TCP flow could have more flexibility. Bob Briscoe has been arguing for a long time that we need to take a broader view and consider "regulating" (or charging for) multiple connections between a given pair of entities. His particular solution has had strong push-back from operators, but his point is entirely valid. Excessive inflexibility about what a single TCP flow is allowed to do hinders progress without contributing substantially to real-world fairness or performance. Again, I'm not saying that there should be a free-for-all, with unmonitored experiments. We just need to remember that the aspect of TCP congestion control that prevents congestion collapse is primarily the shift from go-back-N to selective-repeat i.e., packet conservation. Changing almost any other aspect is unlikely to bring the Internet crashing down. ((Standard thought-experiment disclaimer applies.)) Cheers, Lachlan -- Lachlan Andrew Centre for Advanced Internet Architectures (CAIA) Swinburne University of Technology, Melbourne, Australia Ph +61 3 9214 4837