From frasker at hotmail.com Wed Jun 4 00:13:10 2008 From: frasker at hotmail.com (frasker@hotmail.com) Date: Wed, 04 Jun 2008 07:13:10 -0000 Subject: [e2e] good game, I've played for a moment for free. = Message-ID: http://203.155.74.91/elib/blog/blog.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080604/8da2f921/attachment.html From hsanson at gmail.com Mon Jun 2 18:45:47 2008 From: hsanson at gmail.com (hsanson) Date: Tue, 03 Jun 2008 10:45:47 +0900 Subject: [e2e] P2P Streaming alternatives?? Message-ID: <4844A24B.7020105@gmail.com> I have been researching on scalable real time streaming architectures and everybody seems to be betting on P2P streaming. If you see the current alternatives then the P2P streaming idea does not look so bad. - Classic server/client architecture simply accept as many clients as server resources allow and refuse connection to any new clients. - Content Delivery Network, increase capacity as needed by adding more servers to the CDN, that is costly and normally wastes those resources on non peak times. - IP multicast, that would be the optimal solution but as I understand only works on corporate environments and so far will never expand to cover the Internet. - P2P sounds like the perfect solution, keep enough resources to serve a few clients and in peak sessions use client resources to distribute the media. As demand increases so the available resources (bandwidth) increase. But the technical challenges we need to overcome in order to enable P2P streaming are too high: - dynamic clients can disrupt others connections. - security, any client can inject other data to the stream. - heterogeneous clients, not all clients have equal capacity. - P2P distributed, scalable and fast management algorithms. So I wonder if there exist any other alternative to cheap, scalable media streaming apart from the architectures mentioned above? or shall I focus my research somewhere else. comments are welcome, Horacio From Jon.Crowcroft at cl.cam.ac.uk Wed Jun 4 09:25:56 2008 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Wed, 04 Jun 2008 17:25:56 +0100 Subject: [e2e] P2P Streaming alternatives?? In-Reply-To: <4844A24B.7020105@gmail.com> References: <4844A24B.7020105@gmail.com> Message-ID: hybrid p2p and multicast? neflix? In missive <4844A24B.7020105 at gmail.com>, hsanson typed: >> >>I have been researching on scalable real time streaming architectures >>and everybody seems to be betting on P2P streaming. If you see the >>current alternatives then the P2P streaming idea does not look so bad. >> >>- Classic server/client architecture simply accept as many clients as >>server resources allow and refuse connection to any new clients. >> >>- Content Delivery Network, increase capacity as needed by adding more >>servers to the CDN, that is costly and normally wastes those resources >>on non peak times. >> >>- IP multicast, that would be the optimal solution but as I understand >>only works on corporate environments and so far will never expand to >>cover the Internet. >> >>- P2P sounds like the perfect solution, keep enough resources to serve a >>few clients and in peak sessions use client resources to distribute the >>media. As demand increases so the available resources (bandwidth) increase. >> >>But the technical challenges we need to overcome in order to enable P2P >>streaming are too high: >> >> - dynamic clients can disrupt others connections. >> - security, any client can inject other data to the stream. >> - heterogeneous clients, not all clients have equal capacity. >> - P2P distributed, scalable and fast management algorithms. >> >>So I wonder if there exist any other alternative to cheap, scalable >>media streaming apart from the architectures mentioned above? or shall I >>focus my research somewhere else. >> >>comments are welcome, >>Horacio cheers jon From brough.35790502 at bloglines.com Wed Jun 4 12:06:08 2008 From: brough.35790502 at bloglines.com (brough.35790502@bloglines.com) Date: 4 Jun 2008 19:06:08 -0000 Subject: [e2e] P2P Streaming alternatives?? Message-ID: <1212606368.2511670273.27522.sendItem@bloglines.com> I don't think the "technical challenges...are too high." There have been P2P streaming TV systems running in China at least since 2004 (www.pplive.com & www.pps.tv) and Joost has been running (since late 2006) a hybrid network with long tail content served from central systems and popular content served from peers. There's a good presentation of the Joost network architecture here: http://www.uknof.org/uknof7/MacCarthaigh-Joost.pdf Thanks, Brough Turner -- Mobile: +1 617 285-0433 Skype: brough http://blogs.nmss.com/communications/ --- hsanson > - P2P sounds like the perfect solution, keep enough resources to serve a > few clients and in peak sessions use client resources to distribute the > media. As demand increases so the available resources (bandwidth) increase. > > But the technical challenges we need to overcome in order to enable P2P > streaming are too high: > From dpreed at reed.com Wed Jun 4 13:32:41 2008 From: dpreed at reed.com (David P. Reed) Date: Wed, 04 Jun 2008 16:32:41 -0400 Subject: [e2e] P2P Streaming alternatives?? In-Reply-To: <1212606368.2511670273.27522.sendItem@bloglines.com> References: <1212606368.2511670273.27522.sendItem@bloglines.com> Message-ID: <4846FBE9.5000105@reed.com> Brough, some of these solutions are hybrids. P2P among "peer servers" near the edge, and P2P among clients supported by the servers. That makes it a bit less unstable, and also friendlier to access network providers, if the access networks support local-local communications capacity that exceeds backbone uplink capacity. brough.35790502 at bloglines.com wrote: > I don't think the "technical challenges...are too high." There have been > P2P streaming TV systems running in China at least since 2004 (www.pplive.com > & www.pps.tv) and Joost has been running (since late 2006) a hybrid network > with long tail content served from central systems and popular content served > from peers. There's a good presentation of the Joost network architecture > here: > http://www.uknof.org/uknof7/MacCarthaigh-Joost.pdf > > Thanks, > Brough > Turner > > From kmp at cs.unc.edu Fri Jun 6 09:10:24 2008 From: kmp at cs.unc.edu (Ketan Mayer-Patel) Date: Fri, 6 Jun 2008 12:10:24 -0400 Subject: [e2e] CFP: Multimedia Computing and Networking (MMCN) 2009 Message-ID: <3EE4C493-2992-4783-87C8-FDE0108BE962@cs.unc.edu> ===================================================================== Call for Papers and Announcement SPIE/ACM MMCN 2009 Multimedia Computing and Networking 2009 http://mirage.cs.uoregon.edu/mmcn2009 Part of the IS&T/SPIE International Symposium on Electronic Imaging 18-22 January 2009 San Jose Marriott and San Jose Convention Center San Jose, CA, USA ===================================================================== For 16 years, the multimedia computing and networking conference has brought together researchers, practitioners and developers to contribute new ideas in all facets of multimedia systems, networking, applications, and other related areas of computing. Traditionally the conference features presentations of full and short papers, a keynote talk, and a panel of experts. Presenters are encouraged to make multimedia presentations and demonstrate their proposed solutions. Authors of a few accepted papers with the highest quality are invited to submit the extended version of their papers to a special issue of ACM/Springer Multimedia Systems Journal. Original papers on all emerging technologies and traditional areas of multimedia, including but not limited to: Multimedia Computing . multimedia OS services . power-aware systems . video-on-demand services . mixed and augmented reality systems Measurement and Modeling . performance measurement of multimedia systems . statistical modeling of server traffic and server software . multimedia system simulations and benchmark comparisons Multimedia Networking . home, mobile and broadband networks . QoS control and scheduling . push technologies and content distribution . peer-to-peer media systems . Internet data streaming, delivery and wide-area caching . multimedia security and rights management Case Studies and Applications . multimedia search engines . entertainment and networked games . distributed augmented and virtual reality . multimedia authoring Authors are invited to submit both research and industrial papers on original, unpublished work that is not currently under submission at any other conference. Papers whose contributions are supported by experimental evaluations are strongly encouraged. Both full and short papers are considered. Full paper submissions should not exceed 12 single-spaced, single column pages including figures, tables, and references, using a typeface no smaller than 10 points. Short paper submissions should not exceed 8 pages. All papers must be electronically submitted to the conference website. Further information about MMCN'09 can be found at http://mirage.cs.uoregon.edu/mmcn2009. * Paper Registration: 24 June 2008, 11:59pm PDT * Paper Submission: 01 July 2008, 11:59pm PDT * Notification of Acceptance: 20 August 2008 * Camera Ready Version: 27 October 2008 * 200-word Final Summary: 27 October 2008 * Conference 18-22 January 2009 Proceedings of this conference will be published and available at the meeting. Conference Chairs: Ketan Mayer-Patel, Univ. of North Carolina/Chapel Hill (USA) Reza Rejaje, Univ. of Oregon (USA) Program Committee: Tarek Abdelzaher, Univ. of Illinois/Urbana-Champaign (USA) Surendar Chandra, Univ. of Notre Dame (USA) Songqing Chen, George Mason Univ. (USA) Mark Claypool, Worcester Polytechnic Institute (USA) Chitra Dorai, IBM Thomas J. Watson Research Ctr. (USA) Wu-chi Feng, Portland State Univ. (USA) Pascal Frossard, Swiss Federal Institute of Technology/Lausanne (Switzerland) Christos Gkantsidis, Microsoft Research (UK) Carsten Griwodz, Univ. of Oslo (Norway) Yang Guo, Thomson Lab (USA) Ahsan Habib, Siemens TTB Center/Berkeley (USA) P?l Halvorsen, Simula Research Laboratory (Norway) Mohamed Hefeeda, Simon Fraser Univ. (Canada) Seon Ho Kim, Univ. of Denver (USA) Nikolaos Laoutaris, Telefonica Research, Barcelona (Spain) Baochun Li, Univ. of Toronto (Canada) Jin Li, Microsoft Research (USA) Kang Li, Univ. of Georgia (USA) Andreas Mauthe, Lancaster Univ. (UK) Wei Tsang Ooi, National Univ. of Singapore (Singapore) Nabil Sarhan, Wayne State Univ. (USA) Karsten Schwan, Georgia Institute of Technology (USA) Prashant Shenoy, Univ. of Massachusetts/Amherst (USA) Shervin Shirmohammadi, Univ. of Ottawa (Canada) Nalini Venkatasubramanian, Univ. of California/Irvine (USA) Roger Zimmermann, National Univ. of Singapore (Singapore) Web Chair: Ghulam Memon, Univ. of Oregon (USA) From craig at aland.bbn.com Tue Jun 10 14:21:02 2008 From: craig at aland.bbn.com (Craig Partridge) Date: Tue, 10 Jun 2008 17:21:02 -0400 Subject: [e2e] end of interest In-Reply-To: Your message of "Mon, 12 May 2008 12:37:32 EDT." Message-ID: <20080610212102.5AB5A28E155@aland.bbn.com> Finally getting to read a rich collection of postings at a time I can think and had a quick comment: In message , John Day writes: > ... > >I tend to think of this in terms of "loci" of shared state and their >scopes. (I use locus to indicate that the amount of shared state >could be small or large) The problem with our early thinking on >layering whether OSI or not was confusing the layer as a model of >distributed components with the implementation of such a thing. (I >tend to be proponent of letting the problem tell me what is going on >rather me imposing what I think is going on. ;-) It is less >embarrassing.) > >Also, it seems that if networking is composed of "loci of shared >state with different scopes that we want to treat as black boxes" >then there seems to be something vaguely like a layer going on from >one perspective. Given the problems we have had with layers this >would seem to imply that there is something about them that we aren't >getting right. Something we aren't seeing. Let me suggest that there is no "loci of shared state" -- or perhaps, better said, the "shared state" is incomplete. So, like many folks, I'm on a team playing with dynamic spectrum access: briefly the idea is N nodes want to talk to each other, they figure out what part of the spectrum is underutilized/available and meets their needs and use that. One challenge is that, to well and fully understand what parts of the spectrum might feasible, each of the N nodes could need to see a snapshot of the current spectrum from the N-1 neighbors (for instance, frequency F may look clear of noise at node 7 but see a tremendous level of noise [power] at node 19). The snapshots can easily get quite large (read megabytes). So you rapidly conclude that you can't send around complete information -- instead you send snippets -- and everyone's information about the collective state is incomplete... (I'll note in passing, there are ways to evade this problem but they often end up with lots of exchanges of information -- so the radios are chattering incessantly about what they will do rather than doing it. In some ways, it is a question of RIP vs. SPF through a carnival looking glass.) Anyway, so we find ourselves thinking hard about how well we can communicate with incomplete information and what fallback schemes to use when communication fails, not due to physics but confusion. Thanks! Craig From fernando at gont.com.ar Sun Jun 22 03:35:04 2008 From: fernando at gont.com.ar (Fernando Gont) Date: Sun, 22 Jun 2008 07:35:04 -0300 Subject: [e2e] Papers on IPv6 fragmentation Message-ID: <200806221038.m5MAcOnc027455@venus.xmundo.net> Hello folks, Is anybody aware of any papers with real-world data about IPv6 fragmentation? I'd be interested in something along the lines of: Shannon, C., Moore, D., and Claffy, K.C. 2001. Characteristics of Fragmented IP Traffic on Internet Links. (but that focuses on IPv6 fragmentation, rather than v4 fragmentation). Thanks! Kind regards, -- Fernando Gont e-mail: fernando at gont.com.ar || fgont at acm.org PGP Fingerprint: 7809 84F5 322E 45C7 F1C9 3945 96EE A9EF D076 FFF1 From michael.scharf at ikr.uni-stuttgart.de Thu Jun 26 00:38:37 2008 From: michael.scharf at ikr.uni-stuttgart.de (Michael Scharf) Date: Thu, 26 Jun 2008 09:38:37 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? Message-ID: <20080626073836.GA13918@ikr.uni-stuttgart.de> Hi, maybe this is a stupid question: Is there really a need for the TCP flow control, i. e., for signaling the receiver window back to the sender? It is well known that TCP realizes both congestion control and flow control, and that a TCP sender therefore maintains two different windows (cwnd and rwnd). Obviously, the congestion control protects the path from overload, while the flow control protects the receiver from overload. However, I have some difficulties to understand why the flow control part and receiver advertized window is actually needed. Instead of reducing rwnd, an overloaded receiver running out of buffer space could simply drop (or mark) new arriving packets, or just refrain from sending acknowledgements. As a reaction to this, the sender probably times out and the TCP congestion control significantly reduces the sending rate, which reduces the load on the receiver, too. To my understanding, a fine granular receiver advertized window is much more efficient if the buffer sizes are of the order of a few packets only. But I guess that most of today's Internet hosts have larger buffers, and therefore they hardly need a fine granular flow control. Are there reasons why TCP can't just use its congestion control to handle slow receivers? Do I overlook some aspect? Any hint or reference would be welcome. Michael From michael.welzl at uibk.ac.at Thu Jun 26 04:40:42 2008 From: michael.welzl at uibk.ac.at (Michael Welzl) Date: Thu, 26 Jun 2008 13:40:42 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626073836.GA13918@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <1214480442.3484.11.camel@guidolin.dit.unitn.it> Hi, If the receiver limits the rate, you don't get the sawtooth behavior, with packets being lost whenever you exceed the congestion limit. No fluctuating rate, but a static one - this is more efficient. Cheers, Michael On Thu, 2008-06-26 at 09:38 +0200, Michael Scharf wrote: > Hi, > > maybe this is a stupid question: Is there really a need for the TCP > flow control, i. e., for signaling the receiver window back to > the sender? > > It is well known that TCP realizes both congestion control and flow > control, and that a TCP sender therefore maintains two different > windows (cwnd and rwnd). Obviously, the congestion control protects > the path from overload, while the flow control protects the receiver > from overload. > > However, I have some difficulties to understand why the flow control > part and receiver advertized window is actually needed. > > Instead of reducing rwnd, an overloaded receiver running out of buffer > space could simply drop (or mark) new arriving packets, or just > refrain from sending acknowledgements. As a reaction to this, the > sender probably times out and the TCP congestion control significantly > reduces the sending rate, which reduces the load on the receiver, too. > > To my understanding, a fine granular receiver advertized window is > much more efficient if the buffer sizes are of the order of a few > packets only. But I guess that most of today's Internet hosts have > larger buffers, and therefore they hardly need a fine granular flow > control. > > Are there reasons why TCP can't just use its congestion control to > handle slow receivers? Do I overlook some aspect? Any hint or > reference would be welcome. > > Michael > From craig at aland.bbn.com Thu Jun 26 04:52:35 2008 From: craig at aland.bbn.com (Craig Partridge) Date: Thu, 26 Jun 2008 07:52:35 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: Your message of "Thu, 26 Jun 2008 09:38:37 +0200." <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <20080626115235.7DA0A28E155@aland.bbn.com> In message <20080626073836.GA13918 at ikr.uni-stuttgart.de>, Michael Scharf writes : >... >However, I have some difficulties to understand why the flow control >part and receiver advertized window is actually needed. > >Instead of reducing rwnd, an overloaded receiver running out of buffer >space could simply drop (or mark) new arriving packets, or just >refrain from sending acknowledgements. >... So, try the thought experiment. Suppose the receiver does drop the newly arriving segment. Eventually the segment is retransmitted. Two (non-exclusive) situations to consider: * the segment's packet encounters congestion and causes another packet to be dropped -- now the decision by the receiver to drop the original transmission has caused a third party harm... (See the Floyd/Romanow paper from SIGCOMM several years back for an analogous situation in ATM and the harm it causes). * the segment's packet fails to get to the receiver (congestion loss or transmission error) -- this can be repeated for each retransmission, such that the receiver's decision to drop the original segment means it never gets the segment and the connection dies... So dropping a segment is bad. Let's try retaining but not sending acks... * without an ack, the sender eventually retransmits and the retransmitted segment can, in a case of congestion, again cause loss for a third party (to no purpose, as the retransmission is clearly redundant -- if only the receiver had acked...) * if acks are suppressed too long, the sender times out the connection and the connection fails * another consequence is that the sender increases its round-trip timeout, so when a true loss occurs later in the connection, the sender will respond less promptly (harming performance). In conclusion, the receiver needs to ack, and ack promptly. But the receiver is not ready for more data... Craig From dpreed at reed.com Thu Jun 26 05:23:42 2008 From: dpreed at reed.com (David P. Reed) Date: Thu, 26 Jun 2008 08:23:42 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626073836.GA13918@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <48638A4E.806@reed.com> This is an interesting question. However, the assertion that many hosts have large buffer pools is too broad and of the wrong dimensionality to be useful. The key value is the ratio: size_rpool * rate_service ------------------------------------ size_inflite * rate_input where size_rpool is the size of the buffer pool, the size_inflite is the total size of packets that will be in flight before a control signal can slow the source, rate_service is the service rate of the receiving system in terms of work it can handle, and rate_input is the input rate of work to the target machine (rate_input is the product of input NIC rate times the work-per-byte-of-input). (Derived from Little's Lemma/Theorem). That's an indicator of the probability of discarded work requests that will arise because the It's a dimensionless parameter. And in particular it illustrates that you need to understand how much work is done per byte and how much work the endpoint can actually do. So the rwnd parameter is NOT actually measuring buffer pool size. It is actually a control loop that measures the endpoint application's ability to do work. Michael Scharf wrote: > Hi, > > maybe this is a stupid question: Is there really a need for the TCP > flow control, i. e., for signaling the receiver window back to > the sender? > > It is well known that TCP realizes both congestion control and flow > control, and that a TCP sender therefore maintains two different > windows (cwnd and rwnd). Obviously, the congestion control protects > the path from overload, while the flow control protects the receiver > from overload. > > However, I have some difficulties to understand why the flow control > part and receiver advertized window is actually needed. > > Instead of reducing rwnd, an overloaded receiver running out of buffer > space could simply drop (or mark) new arriving packets, or just > refrain from sending acknowledgements. As a reaction to this, the > sender probably times out and the TCP congestion control significantly > reduces the sending rate, which reduces the load on the receiver, too. > > To my understanding, a fine granular receiver advertized window is > much more efficient if the buffer sizes are of the order of a few > packets only. But I guess that most of today's Internet hosts have > larger buffers, and therefore they hardly need a fine granular flow > control. > > Are there reasons why TCP can't just use its congestion control to > handle slow receivers? Do I overlook some aspect? Any hint or > reference would be welcome. > > Michael > > From faber at ISI.EDU Thu Jun 26 10:41:32 2008 From: faber at ISI.EDU (Ted Faber) Date: Thu, 26 Jun 2008 10:41:32 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626073836.GA13918@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <20080626174132.GC77132@zod.isi.edu> On Thu, Jun 26, 2008 at 09:38:37AM +0200, Michael Scharf wrote: > Hi, > > maybe this is a stupid question: Is there really a need for the TCP > flow control, i. e., for signaling the receiver window back to > the sender? Several folks have weighed in with useful explanations of this, but I'm going to say it another way. TCP Flow control is an end-to-end feature; TCP congestion control is not. Just like other end-to-end transport features, flow control is a communication of endpoint state, just like the initial synchronization of sequence numbers or options. The protocol is designed to send only data it knows the receiver is willing to accept regardless of the state of the network. That's a design decision, and other protocols may make different ones, but TCP designers decided that sending data that they knew would be dropped was wasteful of network and endpoint resources. Because it's explicit communication, it can be used to set sending rates (over a window time). Congestion control attempts to infer network capacity and avoid collapse. Because of the coarse granularity of its inference mechanism and because it's a global safety feature (it's avoiding network collapse (or maybe path collapse)), it operates conservatively. Because it's detecting current state, it probes for additional capacity. The two features have very different motivations and goals. Congestion control will treat an endpoint that doesn't do flow control as a congestion point, but that's subtly different than doing flow control. Here's a somewhat contrived example to illustrate the difference. It's simplified a little to draw the distinction. Consider endpoints using flow control to rate limit senders (by limiting the drain rate of the TCP buffer). A single connection (on an uncongested low BDP net) would probably only see second order effects between using congestion control and flow control for this purpose. That assumes that endpoint TCP buffering was sufficient to smooth out the sawtoothed sending rate that TCP congestion control creates and that the BDP was reasonable to prevent underruns at the bottom of the tooth. As I say, it's easy enough to make that all work out, but congestion controlled senders keep probing the network and exhibit the saw toothed sending rate. These are problems that you don't have with flow controled senders. Flow controlled sources stop probing the network when they're sending at the flow rate. Their rate rises to the flow controled limit and plateaus. As more connections use that network, the saw toothed sending rates may combine unpleasently. Because losses are likely to be correlated, there's a reasonable chance that multiple sources will synchronize and that overall, fewer connections will be able to use the network because of the disruptions to the network. Multiple sources hitting their peak at the same time create longer periods of congestion. More flow controlled sources can fit in the same network because there are no disruptive probes to combine. (You can always stack n flat rate senders, but the number of sawtooth rate senders depends on how the teeth align.) As I say, this is simplified. When resources become scarce, even flow controled sources will enter the congestion controled regime and the system becomes tricky to model or understand. But the distinction between a flow controlled source that knows when to stop probing and a congestion controlled source that doesn't is the distinction I'm drawing. -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://mailman.postel.org/pipermail/end2end-interest/attachments/20080626/56ff5668/attachment.bin From michael.scharf at ikr.uni-stuttgart.de Fri Jun 27 01:08:35 2008 From: michael.scharf at ikr.uni-stuttgart.de (Michael Scharf) Date: Fri, 27 Jun 2008 10:08:35 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626115235.7DA0A28E155@aland.bbn.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> Message-ID: <20080627080835.GA14740@ikr.uni-stuttgart.de> On Thu, 26 Jun 2008 at 07:52:35, Craig Partridge wrote: > >Instead of reducing rwnd, an overloaded receiver running out of buffer > >space could simply drop (or mark) new arriving packets, or just > >refrain from sending acknowledgements. > > So, try the thought experiment. > > Suppose the receiver does drop the newly arriving segment. Eventually > the segment is retransmitted. Two (non-exclusive) situations to consider: > > * the segment's packet encounters congestion and causes another packet > to be dropped -- now the decision by the receiver to drop the > original transmission has caused a third party harm... > (See the Floyd/Romanow paper from SIGCOMM several years back for > an analogous situation in ATM and the harm it causes). > > * the segment's packet fails to get to the receiver (congestion loss or > transmission error) -- this can be repeated for each retransmission, > such that the receiver's decision to drop the original segment means > it never gets the segment and the connection dies... > > So dropping a segment is bad. Let's try retaining but not sending acks... > > * without an ack, the sender eventually retransmits and the retransmitted > segment can, in a case of congestion, again cause loss for a third > party (to no purpose, as the retransmission is clearly redundant -- > if only the receiver had acked...) > > * if acks are suppressed too long, the sender times out the connection > and the connection fails > > * another consequence is that the sender increases its round-trip > timeout, so when a true loss occurs later in the connection, the > sender will respond less promptly (harming performance). > > In conclusion, the receiver needs to ack, and ack promptly. But the > receiver is not ready for more data... Instead of dropping arriving packets or not sending acks, the receiver could also send an ack with ECN marking (assuming ECN usage is negotiated). This would also throttle the sender, but not require a retransmission. To my understanding, ECN marking would not cause all these problems. Thus, a receiver running out of buffer space could just use ECN instead of shrinking rwnd. (Of course, the ECN solution is more coarse grained and does not offer certain features, such as zero window updates.) Michael From mascolo at poliba.it Fri Jun 27 03:01:30 2008 From: mascolo at poliba.it (Saverio Mascolo) Date: Fri, 27 Jun 2008 12:01:30 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? Message-ID: <00b401c8d83c$c9c075d0$723bccc1@HPSM> dear michael, you are missing main points here: 1. flow control is aimed at avoiding overflow of the receiver buffer. the receiver buffer is assigned on a per-flow base, i.e. it is not a shared resource. This makes flow control a mechanism that is 100% perfect from the point of view of control, i mean that all the feedback required for perfect control is available; 2. congestion control does not know buffer available at routers because they are shared; this is the reason you need a probing mechanism to estimate cwnd. you do not need this probing mechanism with the receiver buffer since the advertised window tells you the exact available buffer. this is why we need flow control. moreover, saturating cwnd with receiver-buffer size (f.i. 64Kb) avoids that any single flow congest network using probing. best saverio On Fri, Jun 27, 2008 at 10:08 AM, Michael Scharf wrote: On Thu, 26 Jun 2008 at 07:52:35, Craig Partridge wrote: > >Instead of reducing rwnd, an overloaded receiver running out of buffer > >space could simply drop (or mark) new arriving packets, or just > >refrain from sending acknowledgements. > > So, try the thought experiment. > > Suppose the receiver does drop the newly arriving segment. Eventually > the segment is retransmitted. Two (non-exclusive) situations to consider: > > * the segment's packet encounters congestion and causes another packet > to be dropped -- now the decision by the receiver to drop the > original transmission has caused a third party harm... > (See the Floyd/Romanow paper from SIGCOMM several years back for > an analogous situation in ATM and the harm it causes). > > * the segment's packet fails to get to the receiver (congestion loss or > transmission error) -- this can be repeated for each retransmission, > such that the receiver's decision to drop the original segment means > it never gets the segment and the connection dies... > > So dropping a segment is bad. Let's try retaining but not sending acks... > > * without an ack, the sender eventually retransmits and the retransmitted > segment can, in a case of congestion, again cause loss for a third > party (to no purpose, as the retransmission is clearly redundant -- > if only the receiver had acked...) > > * if acks are suppressed too long, the sender times out the connection > and the connection fails > > * another consequence is that the sender increases its round-trip > timeout, so when a true loss occurs later in the connection, the > sender will respond less promptly (harming performance). > > In conclusion, the receiver needs to ack, and ack promptly. But the > receiver is not ready for more data... Instead of dropping arriving packets or not sending acks, the receiver could also send an ack with ECN marking (assuming ECN usage is negotiated). This would also throttle the sender, but not require a retransmission. To my understanding, ECN marking would not cause all these problems. Thus, a receiver running out of buffer space could just use ECN instead of shrinking rwnd. (Of course, the ECN solution is more coarse grained and does not offer certain features, such as zero window updates.) Michael -- Prof. Saverio Mascolo Dipartimento di Elettrotecnica ed Elettronica Politecnico di Bari Tel. +39 080 5963621 Fax. +39 080 5963410 email:mascolo at poliba.it http://www-dee.poliba.it/dee-web/Personale/mascolo.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080627/379d3712/attachment.html From mascolo at poliba.it Fri Jun 27 03:02:09 2008 From: mascolo at poliba.it (Saverio Mascolo) Date: Fri, 27 Jun 2008 12:02:09 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? Message-ID: <00be01c8d83c$de6fb7c0$723bccc1@HPSM> dear michael, you are missing some big points here: 1. flow control is aimed at avoiding overflow On Fri, Jun 27, 2008 at 10:08 AM, Michael Scharf wrote: On Thu, 26 Jun 2008 at 07:52:35, Craig Partridge wrote: > >Instead of reducing rwnd, an overloaded receiver running out of buffer > >space could simply drop (or mark) new arriving packets, or just > >refrain from sending acknowledgements. > > So, try the thought experiment. > > Suppose the receiver does drop the newly arriving segment. Eventually > the segment is retransmitted. Two (non-exclusive) situations to consider: > > * the segment's packet encounters congestion and causes another packet > to be dropped -- now the decision by the receiver to drop the > original transmission has caused a third party harm... > (See the Floyd/Romanow paper from SIGCOMM several years back for > an analogous situation in ATM and the harm it causes). > > * the segment's packet fails to get to the receiver (congestion loss or > transmission error) -- this can be repeated for each retransmission, > such that the receiver's decision to drop the original segment means > it never gets the segment and the connection dies... > > So dropping a segment is bad. Let's try retaining but not sending acks... > > * without an ack, the sender eventually retransmits and the retransmitted > segment can, in a case of congestion, again cause loss for a third > party (to no purpose, as the retransmission is clearly redundant -- > if only the receiver had acked...) > > * if acks are suppressed too long, the sender times out the connection > and the connection fails > > * another consequence is that the sender increases its round-trip > timeout, so when a true loss occurs later in the connection, the > sender will respond less promptly (harming performance). > > In conclusion, the receiver needs to ack, and ack promptly. But the > receiver is not ready for more data... Instead of dropping arriving packets or not sending acks, the receiver could also send an ack with ECN marking (assuming ECN usage is negotiated). This would also throttle the sender, but not require a retransmission. To my understanding, ECN marking would not cause all these problems. Thus, a receiver running out of buffer space could just use ECN instead of shrinking rwnd. (Of course, the ECN solution is more coarse grained and does not offer certain features, such as zero window updates.) Michael -- Prof. Saverio Mascolo Dipartimento di Elettrotecnica ed Elettronica Politecnico di Bari Tel. +39 080 5963621 Fax. +39 080 5963410 email:mascolo at poliba.it http://www-dee.poliba.it/dee-web/Personale/mascolo.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080627/2827cff2/attachment.html From day at std.com Fri Jun 27 05:04:43 2008 From: day at std.com (John Day) Date: Fri, 27 Jun 2008 08:04:43 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <00b401c8d83c$c9c075d0$723bccc1@HPSM> References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> Message-ID: The definitions I have come to use are: Flow control is a feedback mechanism colocated with the resource being controlled. Congestion control is a feedback mechanism not co-located with the resource being controlled. And there in lies the rub. At 12:01 +0200 2008/06/27, Saverio Mascolo wrote: >dear michael, > >you are missing main points here: > >1. flow control is aimed at avoiding overflow of the receiver >buffer. the receiver buffer is assigned on a per-flow base, i.e. it >is not a shared resource. This makes flow control a mechanism that >is 100% perfect from the point of view of control, i mean that >all the feedback required for perfect control is available; > >2. congestion control does not know buffer available at routers >because they are shared; this is the reason you need a probing >mechanism to estimate cwnd. you do not need this probing mechanism >with the receiver buffer since the advertised window tells you the >exact available buffer. > >this is why we need flow control. moreover, saturating cwnd with >receiver-buffer size (f.i. 64Kb) avoids that any single flow >congest network using probing. > >best > >saverio > > >On Fri, Jun 27, 2008 at 10:08 AM, Michael Scharf > wrote: > >On Thu, 26 Jun 2008 at 07:52:35, Craig Partridge wrote: >> >Instead of reducing rwnd, an overloaded receiver running out of buffer >> >space could simply drop (or mark) new arriving packets, or just >> >refrain from sending acknowledgements. >> >> So, try the thought experiment. >> >> Suppose the receiver does drop the newly arriving segment. Eventually >> the segment is retransmitted. Two (non-exclusive) situations to consider: >> >> * the segment's packet encounters congestion and causes another packet >> to be dropped -- now the decision by the receiver to drop the >> original transmission has caused a third party harm... >> (See the Floyd/Romanow paper from SIGCOMM several years back for >> an analogous situation in ATM and the harm it causes). >> >> * the segment's packet fails to get to the receiver (congestion loss or >> transmission error) -- this can be repeated for each retransmission, >> such that the receiver's decision to drop the original segment means >> it never gets the segment and the connection dies... >> >> So dropping a segment is bad. Let's try retaining but not sending acks... >> >> * without an ack, the sender eventually retransmits and the >>retransmitted >> segment can, in a case of congestion, again cause loss for a third >> party (to no purpose, as the retransmission is clearly redundant -- >> if only the receiver had acked...) >> >> * if acks are suppressed too long, the sender times out the connection >> and the connection fails >> >> * another consequence is that the sender increases its round-trip >> timeout, so when a true loss occurs later in the connection, the >> sender will respond less promptly (harming performance). >> >> In conclusion, the receiver needs to ack, and ack promptly. But the >> receiver is not ready for more data... > >Instead of dropping arriving packets or not sending acks, the receiver >could also send an ack with ECN marking (assuming ECN usage is >negotiated). This would also throttle the sender, but not require a >retransmission. To my understanding, ECN marking would not cause all >these problems. > >Thus, a receiver running out of buffer space could just use ECN >instead of shrinking rwnd. (Of course, the ECN solution is more coarse >grained and does not offer certain features, such as zero window >updates.) > >Michael > > > > >-- >Prof. Saverio Mascolo >Dipartimento di Elettrotecnica ed Elettronica >Politecnico di Bari >Tel. +39 080 5963621 >Fax. +39 080 5963410 >email:mascolo at poliba.it > >http://www-dee.poliba.it/dee-web/Personale/mascolo.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080627/4605cc2c/attachment-0001.html From michael.scharf at ikr.uni-stuttgart.de Fri Jun 27 05:17:07 2008 From: michael.scharf at ikr.uni-stuttgart.de (Michael Scharf) Date: Fri, 27 Jun 2008 14:17:07 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <00b401c8d83c$c9c075d0$723bccc1@HPSM> References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> Message-ID: <20080627121707.GA9797@ikr.uni-stuttgart.de> On Fri, 27 Jun 2008 at 12:01:30, Saverio Mascolo wrote: > you are missing main points here: > > 1. flow control is aimed at avoiding overflow of the receiver buffer. the receiver buffer is assigned on a per-flow base, i.e. it is not a shared resource. This makes flow control a mechanism that is 100% perfect from the point of view of control, i mean that all the feedback required for perfect control is available; > > 2. congestion control does not know buffer available at routers because they are shared; this is the reason you need a probing mechanism to estimate cwnd. you do not need this probing mechanism with the receiver buffer since the advertised window tells you the exact available buffer. > > this is why we need flow control. moreover, saturating cwnd with receiver-buffer size (f.i. 64Kb) avoids that any single flow congest network using probing. Very generally speaking, memory _is_ a shared resource on a host. On the one hand, you are probably right, since most network stacks will have a buffer allocation strategy that somehow ensures that the free buffer space, which is signaled in the receiver advertized window, is indeed available. But since memory allocation in an operating system is a rather complex issue, I am not sure whether there is a 100% guarantee that the receive buffer has really (at least) the announced size. Note that modern TCP stacks announce windows much larger than 64K (e. g., up to 16MB), and this could be an incentive to somehow share buffer space if there are many parallel connections. On the other hand, the flow control is not 100% perfect, because of the inherent delay of feedback signals. For instance, many TCP stacks use receive window auto-tuning and dynamically increase their buffer size during the lifetime of a connection. This means that, at a given point in time, there might be more buffer space allocated in the receiver than the sender is aware of. BTW, if window scaling is negotiated and receiver window auto-tuning is enabled, single TCP flows should be able to fill almost any pipe. And, this propobably just what an app expects from TCP... Michael From michael.scharf at ikr.uni-stuttgart.de Fri Jun 27 07:50:11 2008 From: michael.scharf at ikr.uni-stuttgart.de (Michael Scharf) Date: Fri, 27 Jun 2008 16:50:11 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626174132.GC77132@zod.isi.edu> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626174132.GC77132@zod.isi.edu> Message-ID: <20080627145010.GA17578@ikr.uni-stuttgart.de> On Thu, 26 Jun 2008 at 10:41:32, Ted Faber wrote: > Because it's explicit communication, it can be used to set sending rates > (over a window time). > > [...] > > Consider endpoints using flow control to rate limit senders (by limiting > the drain rate of the TCP buffer). A single connection (on an > uncongested low BDP net) would probably only see second order effects > between using congestion control and flow control for this purpose. > That assumes that endpoint TCP buffering was sufficient to smooth out > the sawtoothed sending rate that TCP congestion control creates and that > the BDP was reasonable to prevent underruns at the bottom of the tooth. > As I say, it's easy enough to make that all work out, but congestion > controlled senders keep probing the network and exhibit the saw toothed > sending rate. These are problems that you don't have with flow > controled senders. Flow controlled sources stop probing the network > when they're sending at the flow rate. Their rate rises to the flow > controled limit and plateaus. Just to uphold my point a little bit: This could also be achieved by a sender-side mechanism that just clamps the congestion window at some reasonable value (for instance, Linux allows to configure such a threshold independent of the flow control). True, the receiver-driven flow control inherently realizes an equivalent function if the receive buffer sizes are small. But, isn't it up to the sending side to decide when to stop probing the network? Michael From faber at ISI.EDU Fri Jun 27 10:07:38 2008 From: faber at ISI.EDU (Ted Faber) Date: Fri, 27 Jun 2008 10:07:38 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080627145010.GA17578@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626174132.GC77132@zod.isi.edu> <20080627145010.GA17578@ikr.uni-stuttgart.de> Message-ID: <20080627170738.GC4651@zod.isi.edu> On Fri, Jun 27, 2008 at 04:50:11PM +0200, Michael Scharf wrote: > On Thu, 26 Jun 2008 at 10:41:32, Ted Faber wrote: > > Because it's explicit communication, it can be used to set sending rates > > (over a window time). > > > > [...] > > > > Consider endpoints using flow control to rate limit senders (by limiting > > the drain rate of the TCP buffer). A single connection (on an > > uncongested low BDP net) would probably only see second order effects > > between using congestion control and flow control for this purpose. > > That assumes that endpoint TCP buffering was sufficient to smooth out > > the sawtoothed sending rate that TCP congestion control creates and that > > the BDP was reasonable to prevent underruns at the bottom of the tooth. > > As I say, it's easy enough to make that all work out, but congestion > > controlled senders keep probing the network and exhibit the saw toothed > > sending rate. These are problems that you don't have with flow > > controled senders. Flow controlled sources stop probing the network > > when they're sending at the flow rate. Their rate rises to the flow > > controled limit and plateaus. > > Just to uphold my point a little bit: This could also be achieved by a > sender-side mechanism that just clamps the congestion window at some > reasonable value (for instance, Linux allows to configure such a > threshold independent of the flow control). But "reasonable value" is sometimes decided by the receiver. Don't think workstation; think iPhone. Or embedded webserver on an MP3 player. Or my $70 OpenWRT-enabled router. If the sender has a rate limit, it just calls send at a frequency appropriate to that rate. This may mean that the sender has many fewer outstanding packets that cwnd allows; it's a reason for the FlightSize variable in RFC2581. A rate-limited sender on an uncongested network will not show a saw-toothed sending rate (in steady state). > True, the receiver-driven flow control inherently realizes an > equivalent function if the receive buffer sizes are small. But, isn't > it up to the sending side to decide when to stop probing the network? I don't think so. It's an agreement between the sender and receiver. TCP doesn't make any assumptions about which (if either) endpoint is more constrained. Having the sender probe for capacity it cannot use seems unnecessarily disruptive to me. The receiver sets one bound (the receiver window) and the sender sets another (send buffers + rate at which it tries to send traffic). Once either the unconstrained sender reaches rwnd or the constrained sender has no traffic to queue the probing stops. -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://mailman.postel.org/pipermail/end2end-interest/attachments/20080627/51a45f4f/attachment.bin From faber at ISI.EDU Fri Jun 27 10:22:08 2008 From: faber at ISI.EDU (Ted Faber) Date: Fri, 27 Jun 2008 10:22:08 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> Message-ID: <20080627172208.GD4651@zod.isi.edu> On Fri, Jun 27, 2008 at 08:04:43AM -0400, John Day wrote: > The definitions I have come to use are: > > Flow control is a feedback mechanism colocated with the resource > being controlled. > > Congestion control is a feedback mechanism not co-located with the > resource being controlled. > > And there in lies the rub. For TCP this is a key distinction. I'm going to pick nits below, but I agree that the difference in information age and accuracy is key to the function of these components in TCP. However, with things like ECN (or re-ECN or XCP) it gets fuzzier. And if you get outside TCP one might use a strict reservation policy to avoid congestion (stop-and-wait queueing anyone?), which isn't feedback at all. There is a distinction in goals between the two in addition to the distinction in mechanism. Flow control is 2 endpoints cooperating to properly use a connection-specific resource (specifically receive buffers). Congestion control is an unknown number of players trying to avoid exhausting shared resource(s) and compromising everyone's connectivity. The goals are different - efficiency vs. safety. In TCP an endpoint makes it's congestion control decisions using feedback based on a time-shifted sample of network buffer state. And I agree with you: therein lies the rub. -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://mailman.postel.org/pipermail/end2end-interest/attachments/20080627/30ed32e2/attachment.bin From matta at cs.bu.edu Fri Jun 27 10:52:19 2008 From: matta at cs.bu.edu (Abraham Matta) Date: Fri, 27 Jun 2008 13:52:19 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080627145010.GA17578@ikr.uni-stuttgart.de> Message-ID: > isn't it up to the sending side to decide when > to stop probing the network? I don't see a good reason for the sender *not* to stop probing the network if it is told *explicitly* by the receiver about the rate at which the sender should send? Ideally, if the network *explicitly* tells the sender about a (fair or not) rate at which the sender should send, then no probing by the sender is needed (similar to the good old ATM explicit rate control ;) Best, ibrahim -----Original Message----- From: end2end-interest-bounces at postel.org [mailto:end2end-interest-bounces at postel.org] On Behalf Of Michael Scharf Sent: Friday, June 27, 2008 10:50 AM To: Ted Faber Cc: end2end-interest at postel.org Subject: Re: [e2e] Why do we need TCP flow control (rwnd)? On Thu, 26 Jun 2008 at 10:41:32, Ted Faber wrote: > Because it's explicit communication, it can be used to set sending > rates (over a window time). > > [...] > > Consider endpoints using flow control to rate limit senders (by > limiting the drain rate of the TCP buffer). A single connection (on > an uncongested low BDP net) would probably only see second order > effects between using congestion control and flow control for this purpose. > That assumes that endpoint TCP buffering was sufficient to smooth out > the sawtoothed sending rate that TCP congestion control creates and > that the BDP was reasonable to prevent underruns at the bottom of the tooth. > As I say, it's easy enough to make that all work out, but congestion > controlled senders keep probing the network and exhibit the saw > toothed sending rate. These are problems that you don't have with > flow controled senders. Flow controlled sources stop probing the > network when they're sending at the flow rate. Their rate rises to > the flow controled limit and plateaus. Just to uphold my point a little bit: This could also be achieved by a sender-side mechanism that just clamps the congestion window at some reasonable value (for instance, Linux allows to configure such a threshold independent of the flow control). True, the receiver-driven flow control inherently realizes an equivalent function if the receive buffer sizes are small. But, isn't it up to the sending side to decide when to stop probing the network? Michael From michael.scharf at ikr.uni-stuttgart.de Fri Jun 27 11:34:21 2008 From: michael.scharf at ikr.uni-stuttgart.de (Michael Scharf) Date: Fri, 27 Jun 2008 20:34:21 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <1e41a3230806270918lbcd4860r620d38a59d15db08@mail.gmail.com> References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> <20080627121707.GA9797@ikr.uni-stuttgart.de> <1e41a3230806270918lbcd4860r620d38a59d15db08@mail.gmail.com> Message-ID: <20080627183421.GA31509@ikr.uni-stuttgart.de> On Fri, 27 Jun 2008 at 12:18:35, John Heffner wrote: > TCP's receiver window works far better for flow control than the AIMD > controller used for congestion control because it is able to quickly > announce the availability of additional capacity. It's possible to > announce a full BDP of available window in a single ack, whereas > congestion control will only increase cwnd by a small amount with the > same ack, if it is controlling the rate. This is particularly useful > since applications often consume data in bursts, based on availability > of resources in the host. They tend to be far more bursty than > network devices that congestion control probes. Congestion control > does not adapt well to quickly varying capacity. That's a good point. (But, a high-speed TCP congestion control variant may be quite fast in grabbing additional capacity.) > Turning the question around, what is the reason *not* to use a receive window? Honestly speaking, I don't know. Maybe there is no reason. But I'll at least try to name some interesting aspects: * Less protocol complexity. Receive window auto-tuning seems to be a non-trivial problem. No need to agree on a fixed window scaling factor during connection setup, when the BDP is hardly known. Middleboxes have less features to mess around with. * Users and applications would probably have to care less about buffer sizing. Less Web sites with TCP tweaking tips. * Speculatively, memory management in the stack could be simpler: When a packet arrives, a receiver either has buffer, then the data is stored, or it is just dropped/marked. Buffer space can be shared among several connections, no need to worry about per-flow allocations, window shrinking, etc. (but, I am not an OS expert) * And, well, a couple of bits in the TCP header that could be used for better purposes ;) Michael From ldecicco at gmail.com Thu Jun 26 07:56:16 2008 From: ldecicco at gmail.com (Luca De Cicco) Date: Thu, 26 Jun 2008 16:56:16 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626073836.GA13918@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <20080626165616.2344ab33@gmail.com> Dear Michael, an elegant explanation of the flow control is given in the paper: S. Mascolo, "Modeling the Internet congestion control using a Smith controller with input shaping", Control Engineering Practice, Vol 14, Issue 4, April 2006 In particular, in the paper it is shown how the flow control can be modeled as the outer loop of the congestion control machinery. Flow control is more efficient than AIMD since it relies on explicit feedback as opposed to the implicit feedback that congestion control algorithms normally take. Moreover, since flow control happens when receiver acts as the bottleneck, isn't dropping at the receiver side (not counting retransmissions) a waste of network resources? However, flow control just asks for 16 bit each ACK to work, isn't it cheap enough? :) Best, Luca De Cicco On Thu, 26 Jun 2008 09:38:37 +0200 Michael Scharf wrote: > Hi, > > maybe this is a stupid question: Is there really a need for the TCP > flow control, i. e., for signaling the receiver window back to > the sender? > > It is well known that TCP realizes both congestion control and flow > control, and that a TCP sender therefore maintains two different > windows (cwnd and rwnd). Obviously, the congestion control protects > the path from overload, while the flow control protects the receiver > from overload. > > However, I have some difficulties to understand why the flow control > part and receiver advertized window is actually needed. > > Instead of reducing rwnd, an overloaded receiver running out of buffer > space could simply drop (or mark) new arriving packets, or just > refrain from sending acknowledgements. As a reaction to this, the > sender probably times out and the TCP congestion control significantly > reduces the sending rate, which reduces the load on the receiver, too. > > To my understanding, a fine granular receiver advertized window is > much more efficient if the buffer sizes are of the order of a few > packets only. But I guess that most of today's Internet hosts have > larger buffers, and therefore they hardly need a fine granular flow > control. > > Are there reasons why TCP can't just use its congestion control to > handle slow receivers? Do I overlook some aspect? Any hint or > reference would be welcome. > > Michael -- Luca De Cicco PhD Candidate Politecnico di Bari (Italy) http://c3lab.poliba.it/index.php/LDC From lachlan.andrew at gmail.com Thu Jun 26 10:29:11 2008 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Thu, 26 Jun 2008 10:29:11 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <48638A4E.806@reed.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <48638A4E.806@reed.com> Message-ID: On 26/06/2008, David P. Reed wrote: > > So the rwnd parameter is NOT actually measuring buffer pool size. It is > actually a control loop that measures the endpoint application's ability to > do work. That is a good point. However, most systems still do set rwnd based on the buffer pool size (as, I believe, the RFCs still require). There has been work on using rwnd for congestion control instead of to signal the buffer pool size. Mukhtar, Hanly and I did some , but were by no means the first. There has also been work on senders selectively ignoring rwnd when the buffer pool is small compared to the BDP, but the receiver is still keeping pace with the sending rate. Sorry, I don't have refs for that. Cheers, Lachlan -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Ph: +1 (626) 395-8820 Fax: +1 (626) 568-3603 http://netlab.caltech.edu/lachlan From johnwheffner at gmail.com Fri Jun 27 09:18:35 2008 From: johnwheffner at gmail.com (John Heffner) Date: Fri, 27 Jun 2008 12:18:35 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080627121707.GA9797@ikr.uni-stuttgart.de> References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> <20080627121707.GA9797@ikr.uni-stuttgart.de> Message-ID: <1e41a3230806270918lbcd4860r620d38a59d15db08@mail.gmail.com> On Fri, Jun 27, 2008 at 8:17 AM, Michael Scharf wrote: > On the other hand, the flow control is not 100% perfect, because of > the inherent delay of feedback signals. For instance, many TCP stacks > use receive window auto-tuning and dynamically increase their buffer > size during the lifetime of a connection. This means that, at a given > point in time, there might be more buffer space allocated in the > receiver than the sender is aware of. This feedback delay is intrinsic, and exactly as long as for ECN or loss feedback. This is where buffers help. TCP's receiver window works far better for flow control than the AIMD controller used for congestion control because it is able to quickly announce the availability of additional capacity. It's possible to announce a full BDP of available window in a single ack, whereas congestion control will only increase cwnd by a small amount with the same ack, if it is controlling the rate. This is particularly useful since applications often consume data in bursts, based on availability of resources in the host. They tend to be far more bursty than network devices that congestion control probes. Congestion control does not adapt well to quickly varying capacity. TCP implementations also use the receive window to ensure that they don't need to drop packets if they are unable or unwilling to commit additional memory, since the penalty for dropping a packet is relatively high. Turning the question around, what is the reason *not* to use a receive window? -John From day at std.com Fri Jun 27 18:31:05 2008 From: day at std.com (John Day) Date: Fri, 27 Jun 2008 21:31:05 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080627172208.GD4651@zod.isi.edu> References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> <20080627172208.GD4651@zod.isi.edu> Message-ID: At 10:22 -0700 2008/06/27, Ted Faber wrote: >On Fri, Jun 27, 2008 at 08:04:43AM -0400, John Day wrote: >> The definitions I have come to use are: >> >> Flow control is a feedback mechanism colocated with the resource >> being controlled. >> >> Congestion control is a feedback mechanism not co-located with the >> resource being controlled. >> >> And there in lies the rub. > >For TCP this is a key distinction. I'm going to pick nits below, but >I agree that the difference in information age and accuracy is key to >the function of these components in TCP. > >However, with things like ECN (or re-ECN or XCP) it gets fuzzier. And >if you get outside TCP one might use a strict reservation policy to >avoid congestion (stop-and-wait queueing anyone?), which isn't feedback >at all. Well, strictly speaking it is. ;-) Not very interesting feedback. You can't send until there is an ack. No ack, retransmit. > >There is a distinction in goals between the two in addition to the >distinction in mechanism. True. I wasn't trying to claim that was all there was to say! ;-) Just the seminal property. Although, a good textbook is process control would tell you that you should always put the feedback as close to the resource to be controlled as you can get it. It is just not practical sometimes. And it is also good to have . . . shall we say, a little slop in the system. ;-) For safety. Too much control is as bad as too little. > >Flow control is 2 endpoints cooperating to properly use a >connection-specific resource (specifically receive buffers). > >Congestion control is an unknown number of players trying to avoid >exhausting shared resource(s) and compromising everyone's connectivity. In some cases, this is true. > >The goals are different - efficiency vs. safety. Well, ;-) safety is in the eye of the beholder. Flow control can be just as much for safety as efficiency, especially with pooled buffers. > >In TCP an endpoint makes it's congestion control decisions using >feedback based on a time-shifted sample of network buffer state. And I >agree with you: therein lies the rub. Strictly speaking, flow control is time shifted as well. By the time, information gets to the sender it is out of date and may be wrong (something unexpected may have happened back at the receiver). With congestion control, things are much more out of date and hence even more chance that information could be very wrong. If something can go wrong in the meantime, it will! ;-) But largely I think we agree. >-- >Ted Faber >http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc >Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG > >Attachment converted: Macintosh HD:Untitled 1 ( / ) (002CF8F0) From weixl at caltech.edu Fri Jun 27 21:13:02 2008 From: weixl at caltech.edu (Xiaoliang "David" Wei) Date: Fri, 27 Jun 2008 21:13:02 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626073836.GA13918@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> I really enjoyed reading this thread -- lots of wisdom and history from what all the gurus have said. :) I would add two cents with my experience of playing with the rwnd: 1. rwnd is not critical for the correctness of TCP. So, yes, we can remove rwnd without breaking TCP's correctness. The TCP algorithm is very robust to guarantee reliability and avoid congestion. If we remove rwnd, the receiver buffer will just be viewed as part of (the last hop of) the network model in the congestion control algorithm, and receiver dropping the packet (due to lack of buffer) will be a congestion signal to the sender to slow down. This will work, though the sender now has to "guess" the receiver's buffer with the *same* assumption of network congestion, and the guessing function will be the same congestion control algorithm such as AIMD or whatever loss based algorithms -- not necessary sawtooth if your algorithm is not AIMD. So, removing rwnd control will be OK (maybe less efficient), and works well when the receiving application is not the bottleneck, or the receiving application has a similar processing pattern as network processing patterns. 2. Why do we want to remove rwnd control (John's question)? rwnd has its own goodness and badness: Pro: rwnd control is very good in avoiding buffer overflow -- no loss will happen for lack of receiving buffer (unless OS some buffer negation). Con: However, rwnd is not very good in using the buffer efficiently, esp with small buffer cases. With rwnd control, we have to allocate BDP worth of buffer at the receiver to fully utilize the network capacity. However, this BDP worth of buffer is not always necessary at all -- Think about an extreme case that the receiving application has a much larger processing capacity, and each packet arrives at the receiver side can be immediately consumed by the application: we only need to have 1 packet worth of buffer to hold that received packet. But with rwnd control, sender will only send a maximum of rwnd packets each RTT, even there is no queue-up at the receiver side at all! (As David Reed pointed out, the rwnd should indicate the receiving app's processing capacity, but unfortunately, the current way of indication is through available buffer size, which is not always an accurate indication.) This trouble is particular obvious with the majority OS implementations of the last generation. As many research (e.g. web100) pointed out a few years ago, most of the TCP connections are bounded by a very small default buffer of Windows and also Linux. While it is easy to change the server's sending buffer, the clients' receiver buffer (usually lies in millions of customers Windows boxes) is hard to change. So, if we can remove the rwnd control (e.g. having the sender ignore the rwnd and only rely on congestion control), we might improve the connection speed and don't even have extra loss if the receivers can process all the packets quickly. I remember some of the network enhancement units on the market actually do such a feature (with other features to reduce the negative effect of ignoring rwnd). This reason, however, will probably be weaken as Vista and Linux 2.6 both come with buffer auto-tuning. 3. rwnd is very important for the responsiveness and adaptability of TCP. So, no, please don't remove rwnd until you get a good solution for all TCP usages.:) TCP are used almost universally in all reliability traffic. Bulk traffic where network is bottle-necked usually satisfies above conditions that receiver is not a bottleneck. However, there are also many cases that the receiver is slow, or the receiver's processing pattern is completely different from network router (and hence congestion control algorithm's estimation will completely go off). Just give an example of networked-printer. When a networked-printer runs out of paper, it is data processing capability quickly drops to zero and lasts for minutes, then after feeding paper, its capacity quickly jumps back to normal. This on-off pattern is very different from most network congestion, and I don't see TCP congestion control algorithms can handle such case responsively. In this case, rwnd control has its own advantage: great responsiveness (by preventive control, explicit notification when the buffer opens up and etc). Note that to achieve such great responsiveness, rwnd control is designed to be very conservative and preventive -- sender (at this moment) can at most send data up to whatever the receiver (half RTT ago) could receive. This conservativeness guarantees that no packet will be dropped even application completely shut down its processing after announcing the rwnd. ECN and other explicit congestion control provide no such guarantee and cannot achieve the same responsiveness to a sudden capacity shutdown. I think there are a lot other applications that have very different processing patterns and it is very hard to have one algorithm to predict all these patterns efficiently. So, my understanding here is that: A. if the receiver is very fast, we don't need rwnd control at all; B. if the receiver's processing pattern is similar to network congestion and if tcp congestion does a good job, we don't need rwnd either. C. The two "if" in A and B might stand in some cases, but not all the usage cases. I don't expect TCP will work as universally well as it currently does if we don't have rwnd control. -David On Thu, Jun 26, 2008 at 12:38 AM, Michael Scharf < michael.scharf at ikr.uni-stuttgart.de> wrote: > Hi, > > maybe this is a stupid question: Is there really a need for the TCP > flow control, i. e., for signaling the receiver window back to > the sender? > > It is well known that TCP realizes both congestion control and flow > control, and that a TCP sender therefore maintains two different > windows (cwnd and rwnd). Obviously, the congestion control protects > the path from overload, while the flow control protects the receiver > from overload. > > However, I have some difficulties to understand why the flow control > part and receiver advertized window is actually needed. > > Instead of reducing rwnd, an overloaded receiver running out of buffer > space could simply drop (or mark) new arriving packets, or just > refrain from sending acknowledgements. As a reaction to this, the > sender probably times out and the TCP congestion control significantly > reduces the sending rate, which reduces the load on the receiver, too. > > To my understanding, a fine granular receiver advertized window is > much more efficient if the buffer sizes are of the order of a few > packets only. But I guess that most of today's Internet hosts have > larger buffers, and therefore they hardly need a fine granular flow > control. > > Are there reasons why TCP can't just use its congestion control to > handle slow receivers? Do I overlook some aspect? Any hint or > reference would be welcome. > > Michael > > -- Xiaoliang "David" Wei http://davidwei.org *********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080627/36acaa2f/attachment-0001.html From michael.scharf at ikr.uni-stuttgart.de Sat Jun 28 02:00:16 2008 From: michael.scharf at ikr.uni-stuttgart.de (Michael Scharf) Date: Sat, 28 Jun 2008 11:00:16 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> Message-ID: <20080628090015.GA1399@ikr.uni-stuttgart.de> On Fri, 27 Jun 2008 at 21:13:02, Xiaoliang David Wei wrote: > So, my understanding here is that: > A. if the receiver is very fast, we don't need rwnd control at all; > B. if the receiver's processing pattern is similar to network congestion > and if tcp congestion does a good job, we don't need rwnd either. > C. The two "if" in A and B might stand in some cases, but not all the > usage cases. I don't expect TCP will work as universally well as > it currently does if we don't have rwnd control. Maybe we can detect the cases "A" and "B" by some heuristic? To come up with something simple: What about if a sender considered rwnd only when it is small: if (rwnd<=thresh) /* flow control enabled */ wnd = min(cwnd,rwnd) else /* flow control disabled */ wnd = cwnd; The parameter "thresh" could be e. g. 64 KB. Or, more disruptive, "thresh" could be set to the initial window (typically, 3 MSS). The rational behind the latter would be that any host should be able to buffer at least an initial window of data, and if this is not the case, it is definitively running out of buffer space. Such a scheme should reduce the risk of buffer overflows in slow (e. g. embedded) devices, while faster endpoints wouldn't have to care much about the precise value of the advertized window unless they run out of (shared) buffer space as well. Michael From a.grieco at poliba.it Sat Jun 28 03:52:33 2008 From: a.grieco at poliba.it (Alfredo) Date: Sat, 28 Jun 2008 12:52:33 +0200 Subject: [e2e] R: Why do we need TCP flow control (rwnd)? In-Reply-To: <20080628090015.GA1399@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de><7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> <20080628090015.GA1399@ikr.uni-stuttgart.de> Message-ID: <9E4C331E69DE43F0838ED0B2FCE36C46@PCgrieco> Hi All, In my opinion, in this discussion about flow control we should also consider that the speed of the receiver is not the only issue that determines the need for flow control. If you have a very large number of outstanding segments (high BDP) and a segment gets lost in the network, the next segments that will be received by the receiver will be buffered till the lost segment is received (we need to wait the retransmitted version of the lost segment). This is mandatory to ensure in sequence data delivery. In this case, regardless of the speed of the receiver, you need flow control. Best Regards Alfredo -----Messaggio originale----- Da: end2end-interest-bounces at postel.org [mailto:end2end-interest-bounces at postel.org] Per conto di Michael Scharf Inviato: Saturday, June 28, 2008 11:00 AM A: Xiaoliang David Wei Cc: end2end-interest at postel.org Oggetto: Re: [e2e] Why do we need TCP flow control (rwnd)? On Fri, 27 Jun 2008 at 21:13:02, Xiaoliang David Wei wrote: > So, my understanding here is that: > A. if the receiver is very fast, we don't need rwnd control at all; > B. if the receiver's processing pattern is similar to network > congestion and if tcp congestion does a good job, we don't need rwnd either. > C. The two "if" in A and B might stand in some cases, but not all > the usage cases. I don't expect TCP will work as universally well as > it currently does if we don't have rwnd control. Maybe we can detect the cases "A" and "B" by some heuristic? To come up with something simple: What about if a sender considered rwnd only when it is small: if (rwnd<=thresh) /* flow control enabled */ wnd = min(cwnd,rwnd) else /* flow control disabled */ wnd = cwnd; The parameter "thresh" could be e. g. 64 KB. Or, more disruptive, "thresh" could be set to the initial window (typically, 3 MSS). The rational behind the latter would be that any host should be able to buffer at least an initial window of data, and if this is not the case, it is definitively running out of buffer space. Such a scheme should reduce the risk of buffer overflows in slow (e. g. embedded) devices, while faster endpoints wouldn't have to care much about the precise value of the advertized window unless they run out of (shared) buffer space as well. Michael Internal Virus Database is out-of-date. Checked by AVG. Version: 7.5.524 / Virus Database: 270.4.1/1512 - Release Date: 6/21/2008 9:27 AM Internal Virus Database is out-of-date. Checked by AVG. Version: 7.5.524 / Virus Database: 270.4.1/1512 - Release Date: 6/21/2008 9:27 AM From vitacaishun at gmail.com Sat Jun 28 05:33:17 2008 From: vitacaishun at gmail.com (vitacai shun) Date: Sat, 28 Jun 2008 20:33:17 +0800 Subject: [e2e] Flow control or congestion control in application-level? Message-ID: <94d9d84a0806280533r721d710fyac24918e6d49efae@mail.gmail.com> When we talk about IP Multicast, much attention is paid on the flow contro l and error control. When we talk about Overlay Multicast(Application Layer Multicast), the memership management is important because its high dynamics. I am thinking, is it totally unnecessary to consider flow control or error control? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080628/f107d61c/attachment.html From mascolo at poliba.it Sat Jun 28 10:10:18 2008 From: mascolo at poliba.it (Saverio Mascolo) Date: Sat, 28 Jun 2008 19:10:18 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? Message-ID: <001801c8d941$dff360f0$723bccc1@HPSM> dear david, i think that using rwnd is important not only because the control of the receiver buffer is more efficient (otherwise the receiver buffer would be controlled as any other router buffers on the path through cwnd). i think that rwnd plays also the important role of setting a cap on the probing, which ends indeed when cwnd=rwnd. This is very important in practice. It avoids having networks on the verge of congestion that is detrimental for many reasons. best saverio - Hide quoted text - On Sat, Jun 28, 2008 at 6:13 AM, Xiaoliang David Wei wrote: I really enjoyed reading this thread -- lots of wisdom and history from what all the gurus have said. :) I would add two cents with my experience of playing with the rwnd: 1. rwnd is not critical for the correctness of TCP. So, yes, we can remove rwnd without breaking TCP's correctness. The TCP algorithm is very robust to guarantee reliability and avoid congestion. If we remove rwnd, the receiver buffer will just be viewed as part of (the last hop of) the network model in the congestion control algorithm, and receiver dropping the packet (due to lack of buffer) will be a congestion signal to the sender to slow down. This will work, though the sender now has to "guess" the receiver's buffer with the *same* assumption of network congestion, and the guessing function will be the same congestion control algorithm such as AIMD or whatever loss based algorithms -- not necessary sawtooth if your algorithm is not AIMD. So, removing rwnd control will be OK (maybe less efficient), and works well when the receiving application is not the bottleneck, or the receiving application has a similar processing pattern as network processing patterns. 2. Why do we want to remove rwnd control (John's question)? rwnd has its own goodness and badness: Pro: rwnd control is very good in avoiding buffer overflow -- no loss will happen for lack of receiving buffer (unless OS some buffer negation). Con: However, rwnd is not very good in using the buffer efficiently, esp with small buffer cases. With rwnd control, we have to allocate BDP worth of buffer at the receiver to fully utilize the network capacity. However, this BDP worth of buffer is not always necessary at all -- Think about an extreme case that the receiving application has a much larger processing capacity, and each packet arrives at the receiver side can be immediately consumed by the application: we only need to have 1 packet worth of buffer to hold that received packet. But with rwnd control, sender will only send a maximum of rwnd packets each RTT, even there is no queue-up at the receiver side at all! (As David Reed pointed out, the rwnd should indicate the receiving app's processing capacity, but unfortunately, the current way of indication is through available buffer size, which is not always an accurate indication.) This trouble is particular obvious with the majority OS implementations of the last generation. As many research (e.g. web100) pointed out a few years ago, most of the TCP connections are bounded by a very small default buffer of Windows and also Linux. While it is easy to change the server's sending buffer, the clients' receiver buffer (usually lies in millions of customers Windows boxes) is hard to change. So, if we can remove the rwnd control (e.g. having the sender ignore the rwnd and only rely on congestion control), we might improve the connection speed and don't even have extra loss if the receivers can process all the packets quickly. I remember some of the network enhancement units on the market actually do such a feature (with other features to reduce the negative effect of ignoring rwnd). This reason, however, will probably be weaken as Vista and Linux 2.6 both come with buffer auto-tuning. 3. rwnd is very important for the responsiveness and adaptability of TCP. So, no, please don't remove rwnd until you get a good solution for all TCP usages.:) TCP are used almost universally in all reliability traffic. Bulk traffic where network is bottle-necked usually satisfies above conditions that receiver is not a bottleneck. However, there are also many cases that the receiver is slow, or the receiver's processing pattern is completely different from network router (and hence congestion control algorithm's estimation will completely go off). Just give an example of networked-printer. When a networked-printer runs out of paper, it is data processing capability quickly drops to zero and lasts for minutes, then after feeding paper, its capacity quickly jumps back to normal. This on-off pattern is very different from most network congestion, and I don't see TCP congestion control algorithms can handle such case responsively. In this case, rwnd control has its own advantage: great responsiveness (by preventive control, explicit notification when the buffer opens up and etc). Note that to achieve such great responsiveness, rwnd control is designed to be very conservative and preventive -- sender (at this moment) can at most send data up to whatever the receiver (half RTT ago) could receive. This conservativeness guarantees that no packet will be dropped even application completely shut down its processing after announcing the rwnd. ECN and other explicit congestion control provide no such guarantee and cannot achieve the same responsiveness to a sudden capacity shutdown. I think there are a lot other applications that have very different processing patterns and it is very hard to have one algorithm to predict all these patterns efficiently. So, my understanding here is that: A. if the receiver is very fast, we don't need rwnd control at all; B. if the receiver's processing pattern is similar to network congestion and if tcp congestion does a good job, we don't need rwnd either. C. The two "if" in A and B might stand in some cases, but not all the usage cases. I don't expect TCP will work as universally well as it currently does if we don't have rwnd control. -David On Thu, Jun 26, 2008 at 12:38 AM, Michael Scharf wrote: Hi, maybe this is a stupid question: Is there really a need for the TCP flow control, i. e., for signaling the receiver window back to the sender? It is well known that TCP realizes both congestion control and flow control, and that a TCP sender therefore maintains two different windows (cwnd and rwnd). Obviously, the congestion control protects the path from overload, while the flow control protects the receiver from overload. However, I have some difficulties to understand why the flow control part and receiver advertized window is actually needed. Instead of reducing rwnd, an overloaded receiver running out of buffer space could simply drop (or mark) new arriving packets, or just refrain from sending acknowledgements. As a reaction to this, the sender probably times out and the TCP congestion control significantly reduces the sending rate, which reduces the load on the receiver, too. To my understanding, a fine granular receiver advertized window is much more efficient if the buffer sizes are of the order of a few packets only. But I guess that most of today's Internet hosts have larger buffers, and therefore they hardly need a fine granular flow control. Are there reasons why TCP can't just use its congestion control to handle slow receivers? Do I overlook some aspect? Any hint or reference would be welcome. Michael -- Xiaoliang "David" Wei http://davidwei.org *********************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080628/7804cd24/attachment-0001.html From dpreed at reed.com Sat Jun 28 13:43:50 2008 From: dpreed at reed.com (David P. Reed) Date: Sat, 28 Jun 2008 16:43:50 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080628090015.GA1399@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> <20080628090015.GA1399@ikr.uni-stuttgart.de> Message-ID: <4866A286.4080904@reed.com> I'm extremely confused as to why this is so exciting an idea. The whole point of TCP is to be a simple, robust standard way to do something across a wide variety of cases. Without fantasizing about possible imaginary justifications for which there is no evidence, what real problem exists? Michael Scharf wrote: > On Fri, 27 Jun 2008 at 21:13:02, Xiaoliang David Wei wrote: > >> So, my understanding here is that: >> A. if the receiver is very fast, we don't need rwnd control at all; >> B. if the receiver's processing pattern is similar to network congestion >> and if tcp congestion does a good job, we don't need rwnd either. >> C. The two "if" in A and B might stand in some cases, but not all the >> usage cases. I don't expect TCP will work as universally well as >> it currently does if we don't have rwnd control. >> > > Maybe we can detect the cases "A" and "B" by some heuristic? > > To come up with something simple: What about if a sender considered > rwnd only when it is small: > > if (rwnd<=thresh) /* flow control enabled */ > wnd = min(cwnd,rwnd) > else /* flow control disabled */ > wnd = cwnd; > > The parameter "thresh" could be e. g. 64 KB. Or, more disruptive, > "thresh" could be set to the initial window (typically, 3 MSS). The > rational behind the latter would be that any host should be able to > buffer at least an initial window of data, and if this is not the > case, it is definitively running out of buffer space. > > Such a scheme should reduce the risk of buffer overflows in slow > (e. g. embedded) devices, while faster endpoints wouldn't have to care > much about the precise value of the advertized window unless they run > out of (shared) buffer space as well. > > Michael > > From keshav at uwaterloo.ca Sat Jun 28 16:50:29 2008 From: keshav at uwaterloo.ca (S. Keshav) Date: Sat, 28 Jun 2008 19:50:29 -0400 Subject: [e2e] end2end-interest Digest, Vol 52, Issue 10 In-Reply-To: References: Message-ID: <3D29318C-D0CC-441C-8649-402D9C5B0911@uwaterloo.ca> Maybe I'm being really dumb, but I think that the reason for rwnd is simple: why should a sender bother sending something if there is no place to keep it when its received? By telling the sender how much buffer space it has left, the receiver avoids dropping data that has been sent over the network, which represents wasted effort. One could argue that the rwnd value should really be a predictor for the buffer capacity that is expected to be available one RTT from now, but setting that quibble aside, there isn't much more to it. keshav From detlef.bosau at web.de Mon Jun 30 07:18:46 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 16:18:46 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080626073836.GA13918@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> Message-ID: <4868EB46.5040000@web.de> Michael Scharf wrote: > However, I have some difficulties to understand why the flow control > part and receiver advertized window is actually needed. > Well, the path does not know anything about a receiver's capacity - and vice versa the receiver does not know anything about the path's capacity. > Instead of reducing rwnd, an overloaded receiver running out of buffer > space could simply drop (or mark) new arriving packets, or just > "Mark" is often a bad idea. Of course, you are always free to "mark" something. And then pray that the "mark" will reach its goal... What makes "implicit notification", i.e. drop, particularly appealing is that "loss cannot get lost." NACKs can, marks can, loss can't. > refrain from sending acknowledgements. No. A receiver's buffer may well be smaller than the path's capacity. Either way, you must not have more unacknowledged data on the fly than the path and the receiver are able to carry. So, if you omit the rwnd, you may have much more unacknowledged data on the fly, than a receiver my be able to accept. O.k., when the receiver's buffer is overrun, further packets will be dropped and the receiver thus will refrain from sending ACKs anyway. This will result in a delayed congestion action (perhaps without any actual network congestion) and retransmissions which could be avoided. > As a reaction to this, the > sender probably times out and the TCP congestion control significantly > reduces the sending rate, which reduces the load on the receiver, too. > Michael, I apologize for carrying cowls to newcastle, but there is no such thing as rate control or sending rate reduction in (general) TCP. ("general": Of course, I'm aware of rate controlled TCP flavours, e.g. Westwood. But these are experimental and not actually deployed.) > To my understanding, a fine granular receiver advertized window is > much more efficient if the buffer sizes are of the order of a few > packets only. But I guess that most of today's Internet hosts have > larger buffers, and therefore they hardly need a fine granular flow > control. > ?? Of course, you _can_ leave out flow control. But, what is this good for? Imagine a long fat line with 100 MByte capacity and an appropriately large congestion window and up to 100 MByte of unacknowledged data on the fly. Now, you have a queue overrun at the receiver. How long do you want to do retransmissions without sense (because even many of the retranmitted packets are likely to be dropped) until you decreased CWND appropriately? (This reminds me a bit on the flightsize discussion launched by Daniel Minder's question some time ago, which I did not understand for a long time. Perphas, these two issues point into a similar direction. However, the flightsize _may_ greatly exceed a receiver's buffer.) So, basically you can omit both, flightsize and rwnd. However, this happens to the cost of unnessecary retransmissions, drops and congestion actions. > Are there reasons why TCP can't just use its congestion control to > handle slow receivers? To my understanding, the reason is performance. > Do I overlook some aspect? Any hint or > reference would be welcome. > One "stupid" question: Do you have a concrete reason to launch this discussion? AFAIK TCP flow control is one of the oldest mechanisms in TCP, even part of RFC 793 (?). Of couse, it is always valid to put things in question. Some months ago, Dave Reed wrote: Science is about asking questions and answering them. But I'm a bit curious why we put in question a mechanism which is well proven for 30 years now? Detlef -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From detlef.bosau at web.de Mon Jun 30 07:23:00 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 16:23:00 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <1214480442.3484.11.camel@guidolin.dit.unitn.it> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <1214480442.3484.11.camel@guidolin.dit.unitn.it> Message-ID: <4868EC44.2010801@web.de> Michael Welzl wrote: > Hi, > > If the receiver limits the rate, you don't get the sawtooth > behavior, with packets being lost whenever you exceed the > congestion limit. No fluctuating rate, but a static one - > this is more efficient. > > IIRC thas was the clue how the TCP/Santa Cruz simulations (ICNP 99) were made to work ;-) *SCNR* Detlef -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From detlef.bosau at web.de Mon Jun 30 07:33:20 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 16:33:20 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <48638A4E.806@reed.com> Message-ID: <4868EEB0.1010306@web.de> Lachlan Andrew wrote: > On 26/06/2008, David P. Reed wrote: > >> So the rwnd parameter is NOT actually measuring buffer pool size. It is >> actually a control loop that measures the endpoint application's ability to >> do work. >> > > That is a good point. However, most systems still do set rwnd > based on the buffer pool size (as, I believe, the RFCs still require). > Isn't the buffer size a consequence of the application's ability to work? When a WWW server does not fetch requests from the incoming queue, doesn't this decrease the free buffer sprace and thus rwnd? > There has been work on using rwnd for congestion control instead of > to signal the buffer pool size. Mukhtar, Hanly and I did some > , but were > by no means the first. > I know. And my question is still: How does a receiver know the path capacity? Doesn't you (either silently or explicitely) assume the "last mile" of a network to be the bottleneck? > There has also been work on senders selectively ignoring rwnd when > the buffer pool is small compared to the BDP, but the receiver is > still keeping pace with the sending rate. Sorry, I don't have refs > for that. > Hm. Isn't this exactly the point? How does a receiver keep pace with the sending rate (whatever this may be in a self clocking system) when the application does not? And wouldn't particulary a large BDP cause unnesserary delays in a proper reaction of the receiver when the application hangs? (E.g. the search engines in Google are that much overloaded it would simply take several seconds to respond? And therefore, the search engine will intendedly throttle the number of requests?) Detlef -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From detlef.bosau at web.de Mon Jun 30 07:42:18 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 16:42:18 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080627080835.GA14740@ikr.uni-stuttgart.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> Message-ID: <4868F0CA.6000602@web.de> Michael Scharf wrote: > Instead of dropping arriving packets or not sending acks, the receiver > could also send an ack with ECN marking (assuming ECN usage is > negotiated). ECN marks can get lost. In addition: How many data may follow the first dropped packet? When the receiver is fed up, why shouldn't he simply tell the sender instead of seeing it waste network capacity for useless retransmits? And rwnd is _not_ lost - because it is part of any acknowledgement. When rwnd is lost, the whole ACK is lost, this causing the sender to stop. > This would also throttle the sender, but not require a > retransmission. To my understanding, ECN marking would not cause all > these problems. > As I said: What makes "ICN" (implicit congestion notificatin, i.e. by missing ACK) preferable over "ECN" is: Loss cannot get lost. > Thus, a receiver running out of buffer space could just use ECN > instead of shrinking rwnd. And what is the propper reaction at the sender's side? rwnd my be shrunk only temporarily for some reason. So, this may perhaps not even throttle the sender. ECN causes at least one congestion recovery action per "round", IIRC. -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From detlef.bosau at web.de Mon Jun 30 07:56:39 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 16:56:39 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080627121707.GA9797@ikr.uni-stuttgart.de> References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> <20080627121707.GA9797@ikr.uni-stuttgart.de> Message-ID: <4868F427.5050604@web.de> Michael Scharf wrote: > On Fri, 27 Jun 2008 at 12:01:30, Saverio Mascolo wrote: > >> you are missing main points here: >> >> 1. flow control is aimed at avoiding overflow of the receiver buffer. the receiver buffer is assigned on a per-flow base, i.e. it is not a shared resource. This makes flow control a mechanism that is 100% perfect from the point of view of control, i mean that all the feedback required for perfect control is available; >> >> 2. congestion control does not know buffer available at routers because they are shared; this is the reason you need a probing mechanism to estimate cwnd. you do not need this probing mechanism with the receiver buffer since the advertised window tells you the exact available buffer. >> >> this is why we need flow control. moreover, saturating cwnd with receiver-buffer size (f.i. 64Kb) avoids that any single flow congest network using probing. >> > > Very generally speaking, memory _is_ a shared resource on a host. > > On the one hand, you are probably right, since most network stacks > will have a buffer allocation strategy that somehow ensures that the > free buffer space, which is signaled in the receiver advertized > window, is indeed available. Hopefully, anyone will do that. I have a very critical position against overcommittment. Particularly, when it comes to kernel memory. > But since memory allocation in an > operating system is a rather complex issue, I am not sure whether > there is a 100% guarantee that the receive buffer has really (at > least) the announced size. I beg you pardon? > Note that modern TCP stacks announce > windows much larger than 64K (e. g., up to 16MB), and this could be an > incentive to somehow share buffer space if there are many parallel > connections. > > Michael, I did not count the times even _this_ year, when I released my computer here in my room from trashing and memory overcommittment - by power cycle. And I really hate a computer "going mad" when I want to work. > On the other hand, the flow control is not 100% perfect, because of > the inherent delay of feedback signals. Where is the problem? You always announce the amount of buffer space which is actually available. Actually, I think I do understand what you mean. Some years ago I thought about this problem for several weeks and I painted dozens of sketches and scenarios - until I convinced myself, that the "delay" is in fact no problem here. > For instance, many TCP stacks > use receive window auto-tuning and dynamically increase their buffer > size during the lifetime of a connection. Could you give a concrete example for "many"? And is this behaviour RFC conformant? Particularly, you well remember the "use it or loose it" principle that will cause a sender to _de_crease its window size , when a flow is inactive for a period of time. > This means that, at a given > point in time, there might be more buffer space allocated in the > receiver than the sender is aware of. > I don't think that this is a problem. > BTW, if window scaling is negotiated and receiver window auto-tuning > is enabled, single TCP flows should be able to fill almost any > pipe. And, this propobably just what an app expects from TCP... > Definitely not. At least not me. I don't see a justification for "auto-tuning" (what you wrote sounds highly questionable to me) and I do not expect TCP to fill pipes but I do expect TCP to be well behaved and not to cause problems by weird window experiments. -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From faber at ISI.EDU Mon Jun 30 08:41:57 2008 From: faber at ISI.EDU (Ted Faber) Date: Mon, 30 Jun 2008 08:41:57 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: References: <00b401c8d83c$c9c075d0$723bccc1@HPSM> <20080627172208.GD4651@zod.isi.edu> Message-ID: <20080630154157.GA2477@zod.isi.edu> On Fri, Jun 27, 2008 at 09:31:05PM -0400, John Day wrote: > At 10:22 -0700 2008/06/27, Ted Faber wrote: > >On Fri, Jun 27, 2008 at 08:04:43AM -0400, John Day wrote: > >> The definitions I have come to use are: > >> > >> Flow control is a feedback mechanism colocated with the resource > >> being controlled. > >> > >> Congestion control is a feedback mechanism not co-located with the > >> resource being controlled. > >> > >> And there in lies the rub. > > > >For TCP this is a key distinction. I'm going to pick nits below, but > >I agree that the difference in information age and accuracy is key to > >the function of these components in TCP. > > > >However, with things like ECN (or re-ECN or XCP) it gets fuzzier. And > >if you get outside TCP one might use a strict reservation policy to > >avoid congestion (stop-and-wait queueing anyone?), which isn't feedback > >at all. > > Well, strictly speaking it is. ;-) Not very interesting feedback. > You can't send until there is an ack. No ack, retransmit. I was thinking of a hard reservation system. You could send, say streaming voice in such a system without waiting for an ACK - lost packets would be a gap in the voice. > > > > >There is a distinction in goals between the two in addition to the > >distinction in mechanism. > > True. I wasn't trying to claim that was all there was to say! ;-) > Just the seminal property. I understand. :-) > > Although, a good textbook is process control would tell you that you > should always put the feedback as close to the resource to be > controlled as you can get it. It is just not practical sometimes. > And it is also good to have . . . shall we say, a little slop in the > system. ;-) For safety. Too much control is as bad as too little. Feedback congestion control is a tricky problem to solve correctly. :-) Not only is all the data an endpoint has to go on out of date (and probably inferred), the different endpoints may be out of date by different amounts. Worst cases can get very exciting. :-) Fortunately, slowing down when there's trouble and being conservative seem to provide a reasonably fail-safe environment. Today. > >The goals are different - efficiency vs. safety. > > Well, ;-) safety is in the eye of the beholder. Flow control can be > just as much for safety as efficiency, especially with pooled buffers. Sure. But with flow control someone knows all the players (in this case the receiver) and you know who will lose if your flow control fails. In a reservationless network like the Internet, you don't (generally) know who will suffer if a router collapsed from congestion, nor do you know the set of endpoints that are cooperating (loosely) to keep a router from collapsing. > > > > >In TCP an endpoint makes it's congestion control decisions using > >feedback based on a time-shifted sample of network buffer state. And I > >agree with you: therein lies the rub. > > Strictly speaking, flow control is time shifted as well. By the time, > information gets to the sender it is out of date and may be wrong > (something unexpected may have happened back at the receiver). With > congestion control, things are much more out of date and hence even > more chance that information could be very wrong. > > If something can go wrong in the meantime, it will! ;-) > > But largely I think we agree. Indeed. Flow control information is time shifted just as much as congestion information, but at least it isn't a guess as well. :-) -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://mailman.postel.org/pipermail/end2end-interest/attachments/20080630/afe1c041/attachment.bin From faber at ISI.EDU Mon Jun 30 09:07:11 2008 From: faber at ISI.EDU (Ted Faber) Date: Mon, 30 Jun 2008 09:07:11 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <4868F0CA.6000602@web.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> <4868F0CA.6000602@web.de> Message-ID: <20080630160711.GC2477@zod.isi.edu> On Mon, Jun 30, 2008 at 04:42:18PM +0200, Detlef Bosau wrote: > As I said: What makes "ICN" (implicit congestion notificatin, i.e. by > missing ACK) preferable over "ECN" is: Loss cannot get lost. Outside of some bugs in say, tunneling ECN packets, if the ECN is lost, the packet is lost. The missing packet transmits the congestion signal perfectly accurately (as you point out). Bugs where a router or endpoint deletes an ECN mark are of the same order of acknowleging a lost packet (losing a loss). Losing an ECN marked packet sends the exact same congestion signal as delivering it. That aspect of the system is remarkably robust. -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 195 bytes Desc: not available Url : http://mailman.postel.org/pipermail/end2end-interest/attachments/20080630/b306ed7f/attachment-0001.bin From detlef.bosau at web.de Mon Jun 30 10:12:10 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 19:12:10 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <20080630160711.GC2477@zod.isi.edu> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> <4868F0CA.6000602@web.de> <20080630160711.GC2477@zod.isi.edu> Message-ID: <486913EA.5040208@web.de> Ted Faber wrote: > Losing an ECN marked packet sends the exact same congestion signal as > delivering it. That aspect of the system is remarkably robust. > > You're correct. So, ECN is an _additional_ information. As long as packet loss is still reckognized as an indication for congestion, anything is fine :-) -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From dpreed at reed.com Mon Jun 30 12:11:26 2008 From: dpreed at reed.com (David P. Reed) Date: Mon, 30 Jun 2008 15:11:26 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <4868F0CA.6000602@web.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> <4868F0CA.6000602@web.de> Message-ID: <48692FDE.10909@reed.com> ECN marks share the fate of the packets carrying them. So what's your point? Detlef Bosau wrote: > Michael Scharf wrote: >> Instead of dropping arriving packets or not sending acks, the receiver >> could also send an ack with ECN marking (assuming ECN usage is >> negotiated). > > > ECN marks can get lost. > > In addition: How many data may follow the first dropped packet? > > When the receiver is fed up, why shouldn't he simply tell the sender > instead of seeing it waste network capacity for useless retransmits? > And rwnd is _not_ lost - because it is part of any acknowledgement. > When rwnd is lost, the whole ACK is lost, this causing the sender to > stop. > >> This would also throttle the sender, but not require a >> retransmission. To my understanding, ECN marking would not cause all >> these problems. >> > > As I said: What makes "ICN" (implicit congestion notificatin, i.e. by > missing ACK) preferable over "ECN" is: Loss cannot get lost. > >> Thus, a receiver running out of buffer space could just use ECN >> instead of shrinking rwnd. > And what is the propper reaction at the sender's side? rwnd my be > shrunk only temporarily for some reason. > So, this may perhaps not even throttle the sender. > ECN causes at least one congestion recovery action per "round", IIRC. > > > > From detlef.bosau at web.de Mon Jun 30 12:55:10 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Mon, 30 Jun 2008 21:55:10 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <48692FDE.10909@reed.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> <4868F0CA.6000602@web.de> <48692FDE.10909@reed.com> Message-ID: <48693A1E.90403@web.de> David P. Reed wrote: > ECN marks share the fate of the packets carrying them. So what's your > point? O.k., o.k. see <20080630160711.GC2477 at zod.isi.edu> <486913EA.5040208 at web.de> ECN marks cannot fully replace loss as congestion indicator (and they don't, I realized this. However, I do not see, why ECN marks should replace the use of TCP flow control (what I understand as one of Michael's thoughts, <20080627080835.GA14740 at ikr.uni-stuttgart.de>). Particularly because ECN marks can get lost. Wrt. CC, this is no problem because the packet loss is a congestion indicator itself. Particularly, a lost packet does not clock out anything from the sender. In contrast to that, an ACK packet with vaid rwnd will inform the sender correctly about the available receiver capacity. Executive summary: TCP flow control works fine, we should leave it alone. > > Detlef Bosau wrote: >> Michael Scharf wrote: >>> Instead of dropping arriving packets or not sending acks, the receiver >>> could also send an ack with ECN marking (assuming ECN usage is >>> negotiated). >> >> >> ECN marks can get lost. >> >> In addition: How many data may follow the first dropped packet? >> >> When the receiver is fed up, why shouldn't he simply tell the sender >> instead of seeing it waste network capacity for useless retransmits? >> And rwnd is _not_ lost - because it is part of any acknowledgement. >> When rwnd is lost, the whole ACK is lost, this causing the sender to >> stop. >> >>> This would also throttle the sender, but not require a >>> retransmission. To my understanding, ECN marking would not cause all >>> these problems. >>> >> >> As I said: What makes "ICN" (implicit congestion notificatin, i.e. by >> missing ACK) preferable over "ECN" is: Loss cannot get lost. >> >>> Thus, a receiver running out of buffer space could just use ECN >>> instead of shrinking rwnd. >> And what is the propper reaction at the sender's side? rwnd my be >> shrunk only temporarily for some reason. >> So, this may perhaps not even throttle the sender. >> ECN causes at least one congestion recovery action per "round", IIRC. >> >> >> >> -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From detlef.bosau at web.de Mon Jun 30 15:23:59 2008 From: detlef.bosau at web.de (Detlef Bosau) Date: Tue, 01 Jul 2008 00:23:59 +0200 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: References: Message-ID: <48695CFF.4000908@web.de> Abraham Matta wrote: >> isn't it up to the sending side to decide when >> to stop probing the network? >> > > I don't see a good reason for the sender *not* to stop probing the > network if it is told *explicitly* by the receiver about the rate at > which the sender should send? > > Ideally, if the network *explicitly* tells the sender about a (fair or > not) rate at which the sender should send, then no probing by the sender > is needed (similar to the good old ATM explicit rate control ;) > However: TCP does not probe for a "rate" but it probes for _storage_ capacity. The "rate" is a result of selfclocking: A TCP sender is clocked / paced by ACK packets. -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From matta at cs.bu.edu Mon Jun 30 15:30:40 2008 From: matta at cs.bu.edu (Abraham Matta) Date: Mon, 30 Jun 2008 18:30:40 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <48695CFF.4000908@web.de> Message-ID: Sure -- just making the point of explicit feedback... -----Original Message----- From: detlef.bosau at web.de [mailto:detlef.bosau at web.de] Sent: Monday, June 30, 2008 6:24 PM To: e2e Cc: Abraham Matta Subject: Re: [e2e] Why do we need TCP flow control (rwnd)? Abraham Matta wrote: >> isn't it up to the sending side to decide when to stop probing the >> network? >> > > I don't see a good reason for the sender *not* to stop probing the > network if it is told *explicitly* by the receiver about the rate at > which the sender should send? > > Ideally, if the network *explicitly* tells the sender about a (fair or > not) rate at which the sender should send, then no probing by the > sender is needed (similar to the good old ATM explicit rate control ;) > However: TCP does not probe for a "rate" but it probes for _storage_ capacity. The "rate" is a result of selfclocking: A TCP sender is clocked / paced by ACK packets. -- Detlef Bosau Mail: detlef.bosau at web.de Galileistrasse 30 Web: http://www.detlef-bosau.de 70565 Stuttgart Skype: detlef.bosau Mobile: +49 172 681 9937 From pganti at gmail.com Mon Jun 30 17:56:30 2008 From: pganti at gmail.com (Paddy Ganti) Date: Mon, 30 Jun 2008 17:56:30 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> Message-ID: <2ff1f08a0806301756p6a0e7d8bm93fc49e5e3a82081@mail.gmail.com> Nice to learn from your write-up as well. Just one more point where rwnd helps: Imagine a middle box that acts as a tcp relay/forwarder or a PEP. In this case rwnd control provides adequate buffering on both ends (read and write sockets) thus increasing the total throughput. If not for rwnd, the optimal throughput for each individual connection is never reached. -Paddy Ganti On Fri, Jun 27, 2008 at 9:13 PM, Xiaoliang David Wei wrote: > I really enjoyed reading this thread -- lots of wisdom and history from > what all the gurus have said. :) > > I would add two cents with my experience of playing with the rwnd: > > 1. rwnd is not critical for the correctness of TCP. So, yes, we can remove > rwnd without breaking TCP's correctness. > > The TCP algorithm is very robust to guarantee reliability and avoid > congestion. If we remove rwnd, the receiver buffer will just be viewed as > part of (the last hop of) the network model in the congestion control > algorithm, and receiver dropping the packet (due to lack of buffer) will be > a congestion signal to the sender to slow down. This will work, though the > sender now has to "guess" the receiver's buffer with the *same* assumption > of network congestion, and the guessing function will be the same congestion > control algorithm such as AIMD or whatever loss based algorithms -- not > necessary sawtooth if your algorithm is not AIMD. So, removing rwnd control > will be OK (maybe less efficient), and works well when the receiving > application is not the bottleneck, or the receiving application has a > similar processing pattern as network processing patterns. > > > 2. Why do we want to remove rwnd control (John's question)? rwnd has its > own goodness and badness: > > Pro: rwnd control is very good in avoiding buffer overflow -- no loss > will happen for lack of receiving buffer (unless OS some buffer negation). > > Con: However, rwnd is not very good in using the buffer efficiently, > esp with small buffer cases. With rwnd control, we have to allocate BDP > worth of buffer at the receiver to fully utilize the network capacity. > However, this BDP worth of buffer is not always necessary at all -- Think > about an extreme case that the receiving application has a much larger > processing capacity, and each packet arrives at the receiver side can be > immediately consumed by the application: we only need to have 1 packet worth > of buffer to hold that received packet. But with rwnd control, sender will > only send a maximum of rwnd packets each RTT, even there is no queue-up at > the receiver side at all! (As David Reed pointed out, the rwnd should > indicate the receiving app's processing capacity, but unfortunately, the > current way of indication is through available buffer size, which is not > always an accurate indication.) > This trouble is particular obvious with the majority OS implementations > of the last generation. As many research (e.g. web100) pointed out a few > years ago, most of the TCP connections are bounded by a very small default > buffer of Windows and also Linux. While it is easy to change the server's > sending buffer, the clients' receiver buffer (usually lies in millions of > customers Windows boxes) is hard to change. So, if we can remove the rwnd > control (e.g. having the sender ignore the rwnd and only rely on congestion > control), we might improve the connection speed and don't even have extra > loss if the receivers can process all the packets quickly. I remember some > of the network enhancement units on the market actually do such a feature > (with other features to reduce the negative effect of ignoring rwnd). This > reason, however, will probably be weaken as Vista and Linux 2.6 both come > with buffer auto-tuning. > > 3. rwnd is very important for the responsiveness and adaptability of TCP. > So, no, please don't remove rwnd until you get a good solution for all TCP > usages.:) > > TCP are used almost universally in all reliability traffic. Bulk > traffic where network is bottle-necked usually satisfies above conditions > that receiver is not a bottleneck. However, there are also many cases that > the receiver is slow, or the receiver's processing pattern is completely > different from network router (and hence congestion control algorithm's > estimation will completely go off). > > Just give an example of networked-printer. When a networked-printer > runs out of paper, it is data processing capability quickly drops to zero > and lasts for minutes, then after feeding paper, its capacity quickly jumps > back to normal. This on-off pattern is very different from most network > congestion, and I don't see TCP congestion control algorithms can handle > such case responsively. In this case, rwnd control has its own advantage: > great responsiveness (by preventive control, explicit notification when the > buffer opens up and etc). > > Note that to achieve such great responsiveness, rwnd control is > designed to be very conservative and preventive -- sender (at this moment) > can at most send data up to whatever the receiver (half RTT ago) could > receive. This conservativeness guarantees that no packet will be dropped > even application completely shut down its processing after announcing the > rwnd. ECN and other explicit congestion control provide no such guarantee > and cannot achieve the same responsiveness to a sudden capacity shutdown. > > I think there are a lot other applications that have very different > processing patterns and it is very hard to have one algorithm to predict all > these patterns efficiently. > > So, my understanding here is that: > A. if the receiver is very fast, we don't need rwnd control at all; > B. if the receiver's processing pattern is similar to network > congestion and if tcp congestion does a good job, we don't need rwnd either. > C. The two "if" in A and B might stand in some cases, but not all the > usage cases. I don't expect TCP will work as universally well as > it currently does if we don't have rwnd control. > > > -David > > > On Thu, Jun 26, 2008 at 12:38 AM, Michael Scharf < > michael.scharf at ikr.uni-stuttgart.de> wrote: > >> Hi, >> >> maybe this is a stupid question: Is there really a need for the TCP >> flow control, i. e., for signaling the receiver window back to >> the sender? >> >> It is well known that TCP realizes both congestion control and flow >> control, and that a TCP sender therefore maintains two different >> windows (cwnd and rwnd). Obviously, the congestion control protects >> the path from overload, while the flow control protects the receiver >> from overload. >> >> However, I have some difficulties to understand why the flow control >> part and receiver advertized window is actually needed. >> >> Instead of reducing rwnd, an overloaded receiver running out of buffer >> space could simply drop (or mark) new arriving packets, or just >> refrain from sending acknowledgements. As a reaction to this, the >> sender probably times out and the TCP congestion control significantly >> reduces the sending rate, which reduces the load on the receiver, too. >> >> To my understanding, a fine granular receiver advertized window is >> much more efficient if the buffer sizes are of the order of a few >> packets only. But I guess that most of today's Internet hosts have >> larger buffers, and therefore they hardly need a fine granular flow >> control. >> >> Are there reasons why TCP can't just use its congestion control to >> handle slow receivers? Do I overlook some aspect? Any hint or >> reference would be welcome. >> >> Michael >> >> > > > -- > Xiaoliang "David" Wei > http://davidwei.org > *********************************************** > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mailman.postel.org/pipermail/end2end-interest/attachments/20080630/7701b6a0/attachment-0001.html From dpreed at reed.com Mon Jun 30 19:42:12 2008 From: dpreed at reed.com (David P. Reed) Date: Mon, 30 Jun 2008 22:42:12 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <83F80BF7-B927-419D-9E47-FC8C1C61A265@cisco.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> <4868F0CA.6000602@web.de> <48692FDE.10909@reed.com> <48693A1E.90403@web.de> <83F80BF7-B927-419D-9E47-FC8C1C61A265@cisco.com> Message-ID: <48699984.9000809@reed.com> Fred: TCP flow control (as opposed to congestion control) is not the subject of any significant genaralizable performance research papers I know of (well, Jeff Mogul did a nice paper more than a decade ago with others about HTTP/TCP performance in WWW servers, then, but it's OOOOOOLD, and I once did an unpublished[able] study paper evaluating performance issues in POP3/TCP server flow control). As it should be: there are no "standard users" of TCP in terms of applications running on standardized end systems. TCP congestion control can be isolated from flow control by presuming a system where the input source and consumption sink are test programs like "iperf" that have reproducible and standardized simple behaviors, and carefully controlling the OS so its control algorithm idiosyncracies (scheduler parameters, memory managers, etc.) are kept out of the picture. The rwnd question (one simple question that considers the flow control part of TCP) is interesting, as I said. But to consider it one has to consider externalities that this group and more generally the researchers in the field of "networking" have no good models for. As just one example: does a server version of Linux running a mix of server applications manage its kernel buffer pool in any way similarly to a laptop version of MacOSX running a mix of client applications? And since server implementations choose how they manage buffering and acknowledgments from a mixture of OS-dependent and TCP-dependent design space choices, how can a paper claim to talk about flow control problems as if they are a unitary and comprehensive research topic? Fred Baker wrote: > > On Jun 30, 2008, at 12:55 PM, Detlef Bosau wrote: > >> Executive summary: TCP flow control works fine, we should leave it >> alone. > > If it did, people would. The problem is that it doesn't; there are > some very good papers on the reasons that statement is true. So they > (we) try to fix it. > From lachlan.andrew at gmail.com Sun Jun 29 15:53:14 2008 From: lachlan.andrew at gmail.com (Lachlan Andrew) Date: Sun, 29 Jun 2008 15:53:14 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <4866A286.4080904@reed.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> <20080628090015.GA1399@ikr.uni-stuttgart.de> <4866A286.4080904@reed.com> Message-ID: 2008/6/28 David P. Reed : > > Without fantasizing about possible imaginary justifications for which there > is no evidence, what real problem exists? Greetings David, Perhaps the biggest current problem is the one that David Wei mentioned, which occurs when the buffer is small relative the the BDP, but the receiver is fast. As you pointed out, RWND should not really measure the free buffer size, but that is (I believe) what it is currently defined to do. If the receiver has a negligible buffer, but can keep up with the transmission rate, setting RWND to the free buffer space needlessly restricts the transmission rate. Even if the buffer is approximately one BDP, a single loss will cause this buffer to fill up, causing the sending rate in subsequent RTTs to drop to approximately zero, even though SACK can allow full utilisation to be maintained without overrunning the buffer. As well as wasting one RTT, this breaks the ACK clock. Also, as Michael pointed out, OSes (at least Linux) currently allocate individual buffers per flow, wasting a lot of memory. (Perhaps the small-buffer problem would go away if a large common buffer was allocated, in which case it is an implementation rather than protocol problem...) Cheers, Lachlan -- Lachlan Andrew Dept of Computer Science, Caltech 1200 E California Blvd, Mail Code 256-80, Pasadena CA 91125, USA Ph: +1 (626) 395-8820 Fax: +1 (626) 568-3603 http://netlab.caltech.edu/lachlan From johnwheffner at gmail.com Mon Jun 30 08:46:24 2008 From: johnwheffner at gmail.com (John Heffner) Date: Mon, 30 Jun 2008 11:46:24 -0400 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <7335583a0806272113r53fea014n8ce8f99f404d0310@mail.gmail.com> Message-ID: <1e41a3230806300846r766ff44cs86923efb6bd4e3ff@mail.gmail.com> On Sat, Jun 28, 2008 at 12:13 AM, Xiaoliang David Wei wrote: > 1. rwnd is not critical for the correctness of TCP. So, yes, we can remove > rwnd without breaking TCP's correctness. This is only mostly true -- TCP can maintain a zero window indefinitely, but the receiver is dropping segments, the sender cannot distinguish between the network's and the receiver's drops, and will eventually time out and reset the connection. > 2. Why do we want to remove rwnd control (John's question)? rwnd has its > own goodness and badness: > > Pro: rwnd control is very good in avoiding buffer overflow -- no loss > will happen for lack of receiving buffer (unless OS some buffer negation). > Con: However, rwnd is not very good in using the buffer efficiently, esp > with small buffer cases. With rwnd control, we have to allocate BDP worth of > buffer at the receiver to fully utilize the network capacity. However, this > BDP worth of buffer is not always necessary at all -- Think about an extreme > case that the receiving application has a much larger processing capacity, > and each packet arrives at the receiver side can be immediately consumed by > the application: we only need to have 1 packet worth of buffer to hold that > received packet. But with rwnd control, sender will only send a maximum of > rwnd packets each RTT, even there is no queue-up at the receiver side at In all modern operating systems, receiver buffer isn't really "allocated" when the window is announced. A TCP receiver will allocate memory only as segments arrive and must be buffered. The announced window is used to indicate how much memory TCP might be willing to commit to buffering data should it suddenly need to. The only real reason for not announcing more window is if you know that typical behavior such as a single loss in the network or a brief application pause will cause you to drop lots of segments because you are unwilling or unable to commit the memory. > all! (As David Reed pointed out, the rwnd should indicate the receiving > app's processing capacity, but unfortunately, the current way of indication > is through available buffer size, which is not always an accurate > indication.) For what it's worth, this is the idea that receive autotuning uses, though it has never been appropriately documented (mea culpa). It measures the amount of data consumed by the application in an RTT, and uses the maximum as the app's processing capacity. (Past behavior is not always a good indication of future behavior, but it's the best thing we've got.) -John From fred at cisco.com Mon Jun 30 15:56:22 2008 From: fred at cisco.com (Fred Baker) Date: Mon, 30 Jun 2008 15:56:22 -0700 Subject: [e2e] Why do we need TCP flow control (rwnd)? In-Reply-To: <48693A1E.90403@web.de> References: <20080626073836.GA13918@ikr.uni-stuttgart.de> <20080626115235.7DA0A28E155@aland.bbn.com> <20080627080835.GA14740@ikr.uni-stuttgart.de> <4868F0CA.6000602@web.de> <48692FDE.10909@reed.com> <48693A1E.90403@web.de> Message-ID: <83F80BF7-B927-419D-9E47-FC8C1C61A265@cisco.com> On Jun 30, 2008, at 12:55 PM, Detlef Bosau wrote: > Executive summary: TCP flow control works fine, we should leave it > alone. If it did, people would. The problem is that it doesn't; there are some very good papers on the reasons that statement is true. So they (we) try to fix it.