From touch at isi.edu Thu Sep 5 11:30:44 2013 From: touch at isi.edu (Joe Touch) Date: Thu, 05 Sep 2013 11:30:44 -0700 Subject: [e2e] testing Message-ID: <5228CDD4.4000403@isi.edu> please ignore From touch at isi.edu Thu Sep 5 11:53:06 2013 From: touch at isi.edu (Joe Touch) Date: Thu, 05 Sep 2013 11:53:06 -0700 Subject: [e2e] list problem query Message-ID: <5228D312.101@isi.edu> Hi, all, I had a report that this list wasn't posting submissions since Aug 1, 2013. I checked the archives, and there's nothing there since that date. However, I sent a test message a few minutes ago, and it made it to the archive fine. Can you please let me know off-list if you have been having any problems posting since Aug 1? If so, you may want to re-post and let me know if you have any problems this time. Thanks, Joe (list admin) From jamel at cin.ufpe.br Fri Sep 6 04:01:39 2013 From: jamel at cin.ufpe.br (Djamel Sadok) Date: Fri, 6 Sep 2013 08:01:39 -0300 Subject: [e2e] testing In-Reply-To: <5228CDD4.4000403@isi.edu> References: <5228CDD4.4000403@isi.edu> Message-ID: Hi, We could think that in the future we could have e2e subnets and non e2e subnets such as ICNs living side by side sharing the same infra-structure in a virtualized SDN world. Can we think of services that use one paradigm or both? Djamel On Thu, Sep 5, 2013 at 3:30 PM, Joe Touch wrote: > please ignore > From touch at isi.edu Fri Sep 6 15:46:23 2013 From: touch at isi.edu (Joe Touch) Date: Fri, 06 Sep 2013 15:46:23 -0700 Subject: [e2e] testing again - please ignore Message-ID: <522A5B3F.1070905@isi.edu> From mattmathis at google.com Sat Sep 7 18:04:52 2013 From: mattmathis at google.com (Matt Mathis) Date: Sat, 7 Sep 2013 18:04:52 -0700 Subject: [e2e] testing 1/2 - as requested by list admin, please ignore Message-ID: Thanks, --MM-- From jamel at cin.ufpe.br Mon Sep 9 06:09:00 2013 From: jamel at cin.ufpe.br (Djamel Sadok) Date: Mon, 9 Sep 2013 10:09:00 -0300 Subject: [e2e] testing In-Reply-To: <1378508332.701616578@apps.rackspace.com> References: <5228CDD4.4000403@isi.edu> <1378508332.701616578@apps.rackspace.com> Message-ID: Hi, What percentage of those who bought a ticket to a show went on to discuss this on a twitter or other social page later on? A transport provider may agree to increase QoS (bandwidth share) for a video streaming service (an end-to-end service) if it could include some advertising material in real time from a third party (the result is a none2e composed service). Both scenarios show that a non-e2e service is required and gives value added information or new business model. My question is: can we think of interesting services that cannot be met by e2e network operation? note that the scope is that of a networking. Thanks, Djamel On Fri, Sep 6, 2013 at 7:58 PM, wrote: > Why should they be differentiated? (I'm not denying that they could be, > but "should" is a different matter). > > > > Also, I would suggest that when SDN's are used to balkanize networks, that > has little to do with "internetworking". Remember "internetworking" is > different from "networking" in a fundamental way. If you have any > background in Abrahamic religions, the Tower of Babel is a relevant > metaphor to think with (I use that only because I don't know if non > Abrahamic religions have traditions that consider the many problems of > non-interoperation as key issues). > > > > > > > > > > On Friday, September 6, 2013 7:01am, "Djamel Sadok" > said: > > > Hi, > > > > We could think that in the future we could have e2e subnets and non e2e > > subnets such as ICNs living side by side sharing the same infra-structure > > in a virtualized SDN world. Can we think of services that use one > paradigm > > or both? > > > > Djamel > > > > > > > > On Thu, Sep 5, 2013 at 3:30 PM, Joe Touch wrote: > > > > > please ignore > > > > > > From Jon.Crowcroft at cl.cam.ac.uk Mon Sep 9 06:36:10 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Mon, 09 Sep 2013 14:36:10 +0100 Subject: [e2e] testing In-Reply-To: References: <5228CDD4.4000403@isi.edu> <1378508332.701616578@apps.rackspace.com> Message-ID: see Lexicon by Max Berry for a novel with non-abrhamic references for Babel (see also Jorge Luis Borges for library of babel (and lottery of babylon) stories that give you some scope and range bounds on the price of anarchy and the cost of over organisation, respectively... In missive , Djamel Sadok ty ped: >>Hi, >> >>What percentage of those who bought a ticket to a show went on to discuss >>this on a twitter or other social page later on? >> >>A transport provider may agree to increase QoS (bandwidth share) for a >>video streaming service (an end-to-end service) if it could include some >>advertising material in real time from a third party (the result is a >>none2e composed service). >> >>Both scenarios show that a non-e2e service is required and gives value >>added information or new business model. >> >>My question is: can we think of interesting services that cannot be met by >>e2e network operation? note that the scope is that of a networking. >> >>Thanks, >> >>Djamel >> >> >> >> >> >>On Fri, Sep 6, 2013 at 7:58 PM, wrote: >> >>> Why should they be differentiated? (I'm not denying that they could be, >>> but "should" is a different matter). >>> >>> >>> >>> Also, I would suggest that when SDN's are used to balkanize networks, that >>> has little to do with "internetworking". Remember "internetworking" is >>> different from "networking" in a fundamental way. If you have any >>> background in Abrahamic religions, the Tower of Babel is a relevant >>> metaphor to think with (I use that only because I don't know if non >>> Abrahamic religions have traditions that consider the many problems of >>> non-interoperation as key issues). >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> On Friday, September 6, 2013 7:01am, "Djamel Sadok" >>> said: >>> >>> > Hi, >>> > >>> > We could think that in the future we could have e2e subnets and non e2e >>> > subnets such as ICNs living side by side sharing the same infra-structure >>> > in a virtualized SDN world. Can we think of services that use one >>> paradigm >>> > or both? >>> > >>> > Djamel >>> > >>> > >>> > >>> > On Thu, Sep 5, 2013 at 3:30 PM, Joe Touch wrote: >>> > >>> > > please ignore >>> > > >>> > >>> cheers jon From mail at detlef-bosau.de Mon Sep 9 08:25:46 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Mon, 09 Sep 2013 17:25:46 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: Message-ID: <522DE87A.4020801@detlef-bosau.de> (my apologies for repost) I finally managed to read the paper in more detail. What I don't yet understand is the RemyCC algorithm. To my understanding, this is an end-to-end algorithm, like in traditional tcp, employing a multiplicative decrease / additive increase scheme? So, the tuple (m,b,r) describes the algorithm? -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From hari at csail.mit.edu Mon Sep 9 09:15:03 2013 From: hari at csail.mit.edu (Hari Balakrishnan) Date: Mon, 9 Sep 2013 12:15:03 -0400 Subject: [e2e] TCP ex Machina In-Reply-To: <522DE87A.4020801@detlef-bosau.de> References: <522DE87A.4020801@detlef-bosau.de> Message-ID: On Sep 9, 2013, at 11:25 AM, Detlef Bosau wrote: > (my apologies for repost) > > I finally managed to read the paper in more detail. > > What I don't yet understand is the RemyCC algorithm. > > To my understanding, this is an end-to-end algorithm, like in > traditional tcp, employing a multiplicative decrease / additive increase > scheme? > > So, the tuple (m,b,r) describes the algorithm? A RemyCC is described by a set of "match-action" rules mapping an input signal to an action. Each rule has a signal and an action. The signal we explored in the paper has three components: , where ack_ewma is the EWMA (low-pass filter) of the interarrival time between new ACKs, send_ewma is the EWMA of the TCP sender timestamps (echoed in the received ACKs), and rtt_ratio is the ratio of the most recent RTT to the smallest seen so far. The action has three components: , where new_cwnd = old_cwnd * m + b, and packets are sent (paced) at a rate not exceeding 1 packet every r milliseconds (assuming equi-sized packets for simplicity). When an ACK arrives, the sender computes the signal, matches it to the best rule, and performs the resulting action. The rules partition the space of all possible signals into disjoint subspaces, so any input signal observed matches exactly one rule. This sort of structure is general enough to accommodate a variety of signals and actions, not just the ones we proposed. You can see that the structure of the specific match-action rules we investigated makes the resulting RemyCC not linear in its temporal behavior (like TCP), though it is piecewise linear in time. Hari > > -- > ------------------------------------------------------------------ > Detlef Bosau > Galileistra?e 30 70565 Stuttgart Tel.: > +49 711 5208031 > mobile: +49 172 6819937 > skype: detlef.bosau > ICQ: 566129673 > detlef.bosau at web.de http://www.detlef-bosau.de > From hari at csail.mit.edu Mon Sep 9 11:20:39 2013 From: hari at csail.mit.edu (Hari Balakrishnan) Date: Mon, 9 Sep 2013 14:20:39 -0400 Subject: [e2e] TCP ex Machina In-Reply-To: <522DFE0B.2020901@web.de> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> Message-ID: <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> On Sep 9, 2013, at 12:57 PM, Detlef Bosau wrote: > Am 09.09.2013 18:15, schrieb Hari Balakrishnan: >> A RemyCC is described by a set of "match-action" rules mapping an >> input signal to an action. Each rule has a signal and an action. The >> signal we explored in the paper has three components: > send_ewma, and rtt_ratio>, where ack_ewma is the EWMA (low-pass >> filter) of the interarrival time between new ACKs, send_ewma is the >> EWMA of the TCP sender timestamps (echoed in the received ACKs), and >> rtt_ratio is the ratio of the most recent RTT to the smallest seen so >> far. > > So, my understanding is correct that the only congestion signalling in > your algorithm is achieved by > - the interarrival time of ACKs, > - the RTT > - the minimum RTT? Yes. But please note that the ACKs also tell us the sender's transmission timings of the corresponding packets (segments, if used in TCP). We tried some others but these worked the best. Explaining why that is trickier, and is a question we are investigating currently. >> The action has three components: , where new_cwnd = old_cwnd >> * m + b, and packets are sent (paced) at a rate not exceeding 1 > > How do you obtain ? That's the "superrational" learning stage, described in Section 4.3 of the paper. May I please request you to read that part -- it's actually only less than a page long? Happy to answer more specific questions about the mechanics of the learning. And if you are so inclined, the source code to play with the learning method is available from our web site -- mit.edu/remy (Please note: the learning stage isn't just doing a simulation of a specific network that is identical to the one under test/evaluation, but is attempting a clever way to efficiently navigate a large design space, while computing an explicit objective function that quantifies the goodness of any given match-action rule. Thanks for your interest and questions, Hari > > > -- > ------------------------------------------------------------------ > Detlef Bosau > Galileistra?e 30 > 70565 Stuttgart Tel.: +49 711 5208031 > mobile: +49 172 6819937 > skype: detlef.bosau > ICQ: 566129673 > detlef.bosau at web.de http://www.detlef-bosau.de > From mail at detlef-bosau.de Tue Sep 10 07:12:11 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Tue, 10 Sep 2013 16:12:11 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> Message-ID: <522F28BB.20108@detlef-bosau.de> Am 09.09.2013 20:20, schrieb Hari Balakrishnan: > On Sep 9, 2013, at 12:57 PM, Detlef Bosau wrote: > >> Am 09.09.2013 18:15, schrieb Hari Balakrishnan: >>> A RemyCC is described by a set of "match-action" rules mapping an >>> input signal to an action. Each rule has a signal and an action. The >>> signal we explored in the paper has three components: >> send_ewma, and rtt_ratio>, where ack_ewma is the EWMA (low-pass >>> filter) of the interarrival time between new ACKs, send_ewma is the >>> EWMA of the TCP sender timestamps (echoed in the received ACKs), and >>> rtt_ratio is the ratio of the most recent RTT to the smallest seen so >>> far. >> So, my understanding is correct that the only congestion signalling in >> your algorithm is achieved by >> - the interarrival time of ACKs, >> - the RTT >> - the minimum RTT? > Yes. But please note that the ACKs also tell us the sender's transmission timings of the corresponding packets (segments, if used in TCP). We tried some others but these worked the best. Explaining why that is trickier, and is a question we are investigating currently. That's exactly the problem in the whole approach. It's the "classical" conclusion in the wrong direction, AKA "ratio ex post". There are quite a few factors which have influence on the aforementioned statistics, amongst these are: - network load - in case of wireless ?etworks: recovery delay, varying throughput, e.g. in HSDPA the net data rate may change every 2 ms becuase line and channel coding may change - user behaviour (honestly spoken: unpredictable) - software behaviour an additional problem (and IIRC Nilsson, Martin, Ree pointed to that problem in 2003) in hetereogeneous networks you will see highly different time scales, e.g. a cross traffic in the Tier 2 network may cause a load dependent transport time variation of perhaps 10 ns, while the clock used in the end nodes has a resolution of say 2 ms. So, a variation in the observed statistics may be caused by several reasons - nevertheless they are interpreted as load indicators. When you think of the loss differentiation problem, there are literally thousands of papers which observe - RTT - RTT variation - Interpacket delay - Interpacket delay variation / jitter - RTT distribution, average, median, skew which well exhibit load dependent behaviour. And which are used to determine whether an observed packet loss was due to congestion drop or due to corruption. All these approaches are condemned to fail because they share THAT ONE common weakness: Conclusion in the wrong direction. When a certain symptom arises, you know (by good luck or divine inspiration) which one of the 100 possible causes does apply. So the problem is whether the observed statistics are at least suited to determine changes in network load - and hence are suited to serve as an input for a closed loop network congestion control. I spent years of my life in trying to detect network congestion by delay observation and it took me a long and painful process to finally understand and to admit that delay observation statistics are simply not suitable for that purpose. Now, for the simulations, you mainly optimize a target function in n variables in a perfectly reproducible scenario. (In general, and there are few exceptions, neither user behaviour nor wireless network conditions are reproducible at all. Particularly in small time scales.) And of course, you will find minima and maxima here by use of some optimization algorithm. However: Your parameters (m,b,r in the set of rules) optimize a mathematical function. And I don't know whether these optimum will hold, i.e. stay in place, when a user behaviour or wireless network property or some other parameter beyond your control will change. Hence, I'm not completely convinced that this approach actually yields a congestion control algorithm which can be generalized and works with an arbitrary scenario - as long as some assumptions will hold. -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From dhavey at yahoo.com Wed Sep 11 00:10:39 2013 From: dhavey at yahoo.com (Daniel Havey) Date: Wed, 11 Sep 2013 00:10:39 -0700 (PDT) Subject: [e2e] list problem query In-Reply-To: <5228D312.101@isi.edu> References: <5228D312.101@isi.edu> Message-ID: <1378883439.65301.YahooMailNeo@web163003.mail.bf1.yahoo.com> Hi Joe, I don't seem to be able to post. ...Daniel ________________________________ From: Joe Touch To: "end2end-interest at postel.org" Sent: Thursday, September 5, 2013 11:53 AM Subject: [e2e] list problem query Hi, all, I had a report that this list wasn't posting submissions since Aug 1, 2013. I checked the archives, and there's nothing there since that date. However, I sent a test message a few minutes ago, and it made it to the archive fine. Can you please let me know off-list if you have been having any problems posting since Aug 1? If so, you may want to re-post and let me know if you have any problems this time. Thanks, Joe (list admin) From loa at pi.nu Wed Sep 11 00:59:32 2013 From: loa at pi.nu (Loa Andersson) Date: Wed, 11 Sep 2013 09:59:32 +0200 Subject: [e2e] list problem query In-Reply-To: <1378883439.65301.YahooMailNeo@web163003.mail.bf1.yahoo.com> References: <5228D312.101@isi.edu> <1378883439.65301.YahooMailNeo@web163003.mail.bf1.yahoo.com> Message-ID: <523022E4.3020301@pi.nu> Daniel, seems that you are able to post, since I received this mail :). /Loa On 2013-09-11 09:10, Daniel Havey wrote: > Hi Joe, > I don't seem to be able to post. > > ...Daniel > > > ________________________________ > From: Joe Touch > To: "end2end-interest at postel.org" > Sent: Thursday, September 5, 2013 11:53 AM > Subject: [e2e] list problem query > > > Hi, all, > > I had a report that this list wasn't posting submissions since Aug 1, > 2013. I checked the archives, and there's nothing there since that date. > > However, I sent a test message a few minutes ago, and it made it to the > archive fine. > > Can you please let me know off-list if you have been having any problems > posting since Aug 1? If so, you may want to re-post and let me know if > you have any problems this time. > > Thanks, > > Joe (list admin) > -- Loa Andersson email: loa at mail01.huawei.com Senior MPLS Expert loa at pi.nu Huawei Technologies (consultant) phone: +46 739 81 21 64 From mail at detlef-bosau.de Wed Sep 11 10:43:27 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Wed, 11 Sep 2013 19:43:27 +0200 Subject: [e2e] list problem query In-Reply-To: <1378883439.65301.YahooMailNeo@web163003.mail.bf1.yahoo.com> References: <5228D312.101@isi.edu> <1378883439.65301.YahooMailNeo@web163003.mail.bf1.yahoo.com> Message-ID: <5230ABBF.5030801@detlef-bosau.de> Am 11.09.2013 09:10, schrieb Daniel Havey: > Hi Joe, > I don't seem to be able to post. Me neither. When I post from my preferred mail address, the posts don't appear. Some of the posts bounce at my mail provider. I reported the problem to the customer's support but did not receive an answer yet. As a work around, I post from a second mail account of mine run by a different provider. Detlef > ...Daniel > > > ________________________________ > From: Joe Touch > To: "end2end-interest at postel.org" > Sent: Thursday, September 5, 2013 11:53 AM > Subject: [e2e] list problem query > > > Hi, all, > > I had a report that this list wasn't posting submissions since Aug 1, > 2013. I checked the archives, and there's nothing there since that date. > > However, I sent a test message a few minutes ago, and it made it to the > archive fine. > > Can you please let me know off-list if you have been having any problems > posting since Aug 1? If so, you may want to re-post and let me know if > you have any problems this time. > > Thanks, > > Joe (list admin) From mail at detlef-bosau.de Thu Sep 12 04:24:36 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Thu, 12 Sep 2013 13:24:36 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: <51EC4468.1010105@web.de> Message-ID: <5231A474.4070501@detlef-bosau.de> Am 21.07.2013 23:14, schrieb Jon Crowcroft: > it is a tiny bit cleverer than that - the work is the moral equivalent > of the Axelrod experiment in emergent cooperation, but neater because > it is quantitative rather than just qualitative selection of > strategies - what is important (imho) is that they use many many > simulation runs to evaluate a "fitness" of a given protocol...this is > heavy lifting, but pays off - so it will be nice to see empirical > follow up work but this isn't some naive "overfitting" undergrad work > - it is rather different and requires a considered response It his extremely questionable to assign a quantitative meaning to one or more of the flow- and congestion control variables in TCP. The whole VJCC kludge is a qualitative one, not a quantitative one. Although it is implemented with quantities, these are rules oft thumb. In addition, I don't see a "fitness of a protocol", I don't see even a protocol here. I see a numeric gimmick with no practical relevance. -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From mallman at icir.org Fri Sep 13 12:14:26 2013 From: mallman at icir.org (Mark Allman) Date: Fri, 13 Sep 2013 15:14:26 -0400 Subject: [e2e] yet another test ... Message-ID: <20130913191426.A3156205D475@lawyers.icir.org> An embedded and charset-unspecified text was scrubbed... Name: not available Url: http://mailman.postel.org/pipermail/end2end-interest/attachments/20130913/7cb10caf/attachment.ksh From mattmathis at google.com Fri Sep 13 15:48:50 2013 From: mattmathis at google.com (Matt Mathis) Date: Fri, 13 Sep 2013 15:48:50 -0700 Subject: [e2e] Fwd: favor - please post test to E2E list In-Reply-To: <522E365D.90608@isi.edu> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> Message-ID: The problem is that you have to be subscribed to the end2end-interest list *exactly as your mail agent/MTA sends your mail. Be aware that the end2end list is configured to silently discard mail from addresses which are not members of the list. So for example if you are subscribed as me at mydomain.com, but your mail agent send your messages from me at myhost.mydoamin.com, you will see messages from the list, but your posts will be silently discarded. *For some definition of exactly. I did not try: Me at MyDomain.com vs me at mydomain.com me+folder at mydomain.com vs me at mydomain.com If you think you are having problems, look at the "receivedfrom" lines in messages from others, and I bet your inbound subscription address doesn't match your own outbound address. I am pretty sure that groups.{yahoo, hotmail, google, ...}.com all get this right. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. Forwarded conversation Subject: favor - please post test to E2E list ------------------------ From: *Joe Touch* Date: Fri, Sep 6, 2013 at 6:45 PM To: James Sterbenz , Matt Mathis , Mark Allman Hi, all, I have a report of problems posting to the list. Can each of you post a test with the subject "testing - as requested by list admin"? This will let me know if the issue is more pervasive or limited in scope. Thanks, Joe ---------- From: *Matt Mathis* Date: Sat, Sep 7, 2013 at 6:30 PM To: Joe Touch Cc: James Sterbenz , Mark Allman I sent 2 messages, one each from mattmathis at google.com (1/2) and matt.mathis at gmail.com (2/2). I was also suspicious of delivery problems, however since Google normally hides our own messages I was not sure. In absence of any facts..... The common failure is for the list not to properly strip/update all DKIM (whatever) headers from the originator, and then email services which are enforcing signatures discard the messages as being forged. The case I commonly run into is from google to google through a broken list..... I have not yet received my own message 1/2 to either account. Have any of you received it? Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. ---------- From: *Matt Mathis* Date: Sat, Sep 7, 2013 at 8:15 PM To: Joe Touch Cc: James Sterbenz , Mark Allman Message 1 came, after a 40 minute delay. This probably means that it got greylisted. Let me know if you want to dissect the headers. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. ---------- From: *Joe Touch* Date: Mon, Sep 9, 2013 at 7:36 AM To: Matt Mathis Cc: James Sterbenz , Mark Allman Thanks - the delay is not unusual (our servers are configured so that both spam filters and mailman run only periodically rather than on-demand). Joe > ---------- From: *Joe Touch* Date: Mon, Sep 9, 2013 at 7:38 AM To: Matt Mathis Cc: James Sterbenz , Mark Allman On 9/7/2013 6:30 PM, Matt Mathis wrote: > I sent 2 messages, one each from mattmathis at google.com > (1/2) and matt.mathis at gmail.com > (2/2). > Thanks - the first got through, but not the second. That's not surprising, because only the first is subscribed to the list. Joe ---------- From: *Matt Mathis* Date: Mon, Sep 9, 2013 at 8:44 AM To: Joe Touch Cc: James Sterbenz , Mark Allman And that would be your problem: no bounce messages. In this case the name was canonicalized slightly differently - mattmathis at gmail.com is subscribed and receiving messages. Google considers the addresses to be equivalent (I know, I know...), but the list did not tell me that my posting got rejected. Note that this sort of problem is impossible for the user to diagnose themselves because without bounce messages it has no identifying symptom. Further it is compounded by modern mail agents that try real hard to hide the 822 addresses. I suggest turning on bounce messages. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. ---------- From: *Joe Touch* Date: Mon, Sep 9, 2013 at 8:57 AM To: Matt Mathis Cc: James Sterbenz , Mark Allman On 9/9/2013 8:44 AM, Matt Mathis wrote: > And that would be your problem: no bounce messages. In this case the > name was canonicalized slightly differently - mattmathis at gmail.com > is subscribed and receiving messages. > > Google considers the addresses to be equivalent (I know, I know...), > but the list did not tell me that my posting got rejected. > > Note that this sort of problem is impossible for the user to diagnose > themselves because without bounce messages it has no identifying > symptom. Further it is compounded by modern mail agents that try real > hard to hide the 822 addresses. > > I suggest turning on bounce messages. > I don't see how to do that in Mailman without responding to spam, which we don't have the resources to handle (and don't want to do). Joe > > ---------- From: *Matt Mathis* Date: Mon, Sep 9, 2013 at 10:03 AM To: Joe Touch Cc: James Sterbenz , Mark Allman I thought the proper action was to return a 500 level error to the originating MTA. Perhaps the list needs to move to newer platform? Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. ---------- From: *Joe Touch* Date: Mon, Sep 9, 2013 at 11:00 AM To: Matt Mathis Cc: James Sterbenz , Mark Allman On 9/9/2013 10:03 AM, Matt Mathis wrote: > I thought the proper action was to return a 500 level error to the > originating MTA. Perhaps the list needs to move to newer platform? > That would happen at the SMTP level; there's no way to generate that message from within SMTP on behalf of an exploder that doesn't process mail until after SMTP has already accepted it. Joe > > ---------- From: *Matt Mathis* Date: Mon, Sep 9, 2013 at 11:16 AM To: Joe Touch Cc: James Sterbenz , Mark Allman Yes, I know. Like I said, perhaps the list needs to move to a more up to date platform. I would bet that all of the big list services get this right. Thanks, --MM-- The best way to predict the future is to create it. - Alan Kay Privacy matters! We know from recent events that people are using our services to speak in defiance of unjust governments. We treat privacy and security as matters of life and death, because for some users, they are. ---------- From: *Joe Touch* Date: Mon, Sep 9, 2013 at 1:58 PM To: Matt Mathis Cc: James Sterbenz , Mark Allman Hi, Matt, I'm not aware of such systems. We use mailman here at ISI, as do a few other organizations (IEEE, etc.), and it operates on email only after it has been received by sendmail AFAICT. That appears to be true for other systems, because none appear to have their servers integrated with sendmail servers AFAICT. Joe From lars at netapp.com Sat Sep 14 00:17:04 2013 From: lars at netapp.com (Eggert, Lars) Date: Sat, 14 Sep 2013 07:17:04 +0000 Subject: [e2e] favor - please post test to E2E list In-Reply-To: References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> Message-ID: <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> Just configure mailman to reject instead of discard, so people get a notification. Lars On Sep 14, 2013, at 1:48, Matt Mathis wrote: > The problem is that you have to be subscribed to the end2end-interest list > *exactly as your mail agent/MTA sends your mail. Be aware that > the end2end list is configured to silently discard mail from addresses > which are not members of the list. So for example if you are subscribed > as me at mydomain.com, but your mail agent send your messages from > me at myhost.mydoamin.com, you will see messages from the list, but your posts > will be silently discarded. > > *For some definition of exactly. I did not try: > Me at MyDomain.com vs me at mydomain.com > me+folder at mydomain.com vs me at mydomain.com > > If you think you are having problems, look at the "receivedfrom" lines in > messages from others, and I bet your inbound subscription address doesn't > match your own outbound address. > > I am pretty sure that groups.{yahoo, hotmail, google, ...}.com all get this > right. > > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > Privacy matters! We know from recent events that people are using our > services to speak in defiance of unjust governments. We treat privacy and > security as matters of life and death, because for some users, they are. > > > Forwarded conversation > Subject: favor - please post test to E2E list > ------------------------ > > From: *Joe Touch* > Date: Fri, Sep 6, 2013 at 6:45 PM > To: James Sterbenz , Matt Mathis , > Mark Allman > > > Hi, all, > > I have a report of problems posting to the list. > > Can each of you post a test with the subject "testing - as requested by > list admin"? > > This will let me know if the issue is more pervasive or limited in scope. > > Thanks, > > Joe > > > ---------- > From: *Matt Mathis* > Date: Sat, Sep 7, 2013 at 6:30 PM > To: Joe Touch > Cc: James Sterbenz , Mark Allman > > > I sent 2 messages, one each from mattmathis at google.com (1/2) and > matt.mathis at gmail.com (2/2). > > I was also suspicious of delivery problems, however since Google normally > hides our own messages I was not sure. > > In absence of any facts..... The common failure is for the list not to > properly strip/update all DKIM (whatever) headers from the originator, and > then email services which are enforcing signatures discard the messages as > being forged. The case I commonly run into is from google to google > through a broken list..... > > I have not yet received my own message 1/2 to either account. Have any of > you received it? > > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > Privacy matters! We know from recent events that people are using our > services to speak in defiance of unjust governments. We treat privacy and > security as matters of life and death, because for some users, they are. > > ---------- > From: *Matt Mathis* > Date: Sat, Sep 7, 2013 at 8:15 PM > To: Joe Touch > Cc: James Sterbenz , Mark Allman > > > Message 1 came, after a 40 minute delay. This probably means that it got > greylisted. > > Let me know if you want to dissect the headers. > > > > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > Privacy matters! We know from recent events that people are using our > services to speak in defiance of unjust governments. We treat privacy and > security as matters of life and death, because for some users, they are. > > > > ---------- > From: *Joe Touch* > Date: Mon, Sep 9, 2013 at 7:36 AM > To: Matt Mathis > Cc: James Sterbenz , Mark Allman > > > Thanks - the delay is not unusual (our servers are configured so that both > spam filters and mailman run only periodically rather than on-demand). > > Joe > > >> ---------- > From: *Joe Touch* > Date: Mon, Sep 9, 2013 at 7:38 AM > To: Matt Mathis > Cc: James Sterbenz , Mark Allman > > > > > On 9/7/2013 6:30 PM, Matt Mathis wrote: > >> I sent 2 messages, one each from mattmathis at google.com >> (1/2) and matt.mathis at gmail.com >> (2/2). >> > > Thanks - the first got through, but not the second. That's not surprising, > because only the first is subscribed to the list. > > Joe > > ---------- > From: *Matt Mathis* > Date: Mon, Sep 9, 2013 at 8:44 AM > To: Joe Touch > Cc: James Sterbenz , Mark Allman > > > And that would be your problem: no bounce messages. In this case the name > was canonicalized slightly differently - mattmathis at gmail.com is subscribed > and receiving messages. Google considers the addresses to be equivalent (I > know, I know...), but the list did not tell me that my posting got rejected. > > Note that this sort of problem is impossible for the user to diagnose > themselves because without bounce messages it has no identifying symptom. > Further it is compounded by modern mail agents that try real hard to hide > the 822 addresses. > > I suggest turning on bounce messages. > > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > Privacy matters! We know from recent events that people are using our > services to speak in defiance of unjust governments. We treat privacy and > security as matters of life and death, because for some users, they are. > > > > ---------- > From: *Joe Touch* > Date: Mon, Sep 9, 2013 at 8:57 AM > To: Matt Mathis > Cc: James Sterbenz , Mark Allman > > > > > On 9/9/2013 8:44 AM, Matt Mathis wrote: > >> And that would be your problem: no bounce messages. In this case the >> name was canonicalized slightly differently - mattmathis at gmail.com >> is subscribed and receiving messages. >> >> Google considers the addresses to be equivalent (I know, I know...), >> but the list did not tell me that my posting got rejected. >> >> Note that this sort of problem is impossible for the user to diagnose >> themselves because without bounce messages it has no identifying >> symptom. Further it is compounded by modern mail agents that try real >> hard to hide the 822 addresses. >> >> I suggest turning on bounce messages. >> > > I don't see how to do that in Mailman without responding to spam, which we > don't have the resources to handle (and don't want to do). > > Joe > >> >> > ---------- > From: *Matt Mathis* > Date: Mon, Sep 9, 2013 at 10:03 AM > To: Joe Touch > Cc: James Sterbenz , Mark Allman > > > I thought the proper action was to return a 500 level error to the > originating MTA. Perhaps the list needs to move to newer platform? > > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > Privacy matters! We know from recent events that people are using our > services to speak in defiance of unjust governments. We treat privacy and > security as matters of life and death, because for some users, they are. > > > > ---------- > From: *Joe Touch* > Date: Mon, Sep 9, 2013 at 11:00 AM > To: Matt Mathis > Cc: James Sterbenz , Mark Allman > > > > > On 9/9/2013 10:03 AM, Matt Mathis wrote: > >> I thought the proper action was to return a 500 level error to the >> originating MTA. Perhaps the list needs to move to newer platform? >> > > That would happen at the SMTP level; there's no way to generate that > message from within SMTP on behalf of an exploder that doesn't process mail > until after SMTP has already accepted it. > > Joe > > >> >> > ---------- > From: *Matt Mathis* > Date: Mon, Sep 9, 2013 at 11:16 AM > To: Joe Touch > Cc: James Sterbenz , Mark Allman > > > Yes, I know. Like I said, perhaps the list needs to move to a more up to > date platform. I would bet that all of the big list services get this > right. > > Thanks, > --MM-- > The best way to predict the future is to create it. - Alan Kay > > Privacy matters! We know from recent events that people are using our > services to speak in defiance of unjust governments. We treat privacy and > security as matters of life and death, because for some users, they are. > > > > ---------- > From: *Joe Touch* > Date: Mon, Sep 9, 2013 at 1:58 PM > To: Matt Mathis > Cc: James Sterbenz , Mark Allman > > > Hi, Matt, > > I'm not aware of such systems. We use mailman here at ISI, as do a few > other organizations (IEEE, etc.), and it operates on email only after it > has been received by sendmail AFAICT. That appears to be true for other > systems, because none appear to have their servers integrated with sendmail > servers AFAICT. > > Joe From mail at detlef-bosau.de Sat Sep 14 11:30:04 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Sat, 14 Sep 2013 20:30:04 +0200 Subject: [e2e] Fwd: favor - please post test to E2E list In-Reply-To: References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> Message-ID: <5234AB2C.9060903@detlef-bosau.de> Am 14.09.2013 00:48, schrieb Matt Mathis: > The problem is that you have to be subscribed to the end2end-interest list > *exactly as your mail agent/MTA sends your mail. Be aware that > the end2end list is configured to silently discard mail from addresses > which are not members of the list. So for example if you are subscribed > as me at mydomain.com, but your mail agent send your messages from > me at myhost.mydoamin.com, you will see messages from the list, but your posts > will be silently discarded. At least, this is not the whole problem. When I post from detlef.bosau at web.de, I first my posts did not appear - meanwhile I get a "message could not be delivered" notification. I reported the problem to my mail operator - but I did not yet get an answer. However, when I post als "mail at detlef-bosau.de", anything works fine. And yes, I subscribed the list with both addresses. From mail at detlef-bosau.de Sat Sep 14 11:33:15 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Sat, 14 Sep 2013 20:33:15 +0200 Subject: [e2e] favor - please post test to E2E list In-Reply-To: <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> Message-ID: <5234ABEB.80602@detlef-bosau.de> Am 14.09.2013 09:17, schrieb Eggert, Lars: > Just configure mailman to reject instead of discard, so people get a notification. and it spares unnecessary finger pointing. (which happens quite often with this kind of problems.) -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From faber at isi.edu Sat Sep 14 13:53:40 2013 From: faber at isi.edu (Ted Faber) Date: Sat, 14 Sep 2013 13:53:40 -0700 Subject: [e2e] favor - please post test to E2E list In-Reply-To: <5234ABEB.80602@detlef-bosau.de> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> <5234ABEB.80602@detlef-bosau.de> Message-ID: <5234CCD4.8070908@isi.edu> On 09/14/2013 11:33, Detlef Bosau wrote: > Am 14.09.2013 09:17, schrieb Eggert, Lars: >> Just configure mailman to reject instead of discard, so people get a notification. > > and it spares unnecessary finger pointing. (which happens quite often > with this kind of problems.) > It does open a spam vector, though. (A spammer forges mail from a random person's address to end2end-interest and that person gets a spam bounce from end2end.) I suspect that's why it's off. -- Ted Faber http://www.isi.edu/~faber PGP: http://www.isi.edu/~faber/pubkeys.asc Unexpected attachment on this mail? See http://www.isi.edu/~faber/FAQ.html#SIG From mail at detlef-bosau.de Sun Sep 15 06:30:54 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Sun, 15 Sep 2013 15:30:54 +0200 Subject: [e2e] favor - please post test to E2E list In-Reply-To: <5234CCD4.8070908@isi.edu> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> <5234ABEB.80602@detlef-bosau.de> <5234CCD4.8070908@isi.edu> Message-ID: <5235B68E.6090604@detlef-bosau.de> Am 14.09.2013 22:53, schrieb Ted Faber: > On 09/14/2013 11:33, Detlef Bosau wrote: >> Am 14.09.2013 09:17, schrieb Eggert, Lars: >>> Just configure mailman to reject instead of discard, so people get a notification. >> >> and it spares unnecessary finger pointing. (which happens quite often >> with this kind of problems.) >> > It does open a spam vector, though. (A spammer forges mail from a random > person's address to end2end-interest and that person gets a spam bounce > from end2end.) I suspect that's why it's off. > Anyway. Apparently there is a problem and this should be solved. From touch at isi.edu Tue Sep 17 10:17:43 2013 From: touch at isi.edu (Joe Touch) Date: Tue, 17 Sep 2013 10:17:43 -0700 Subject: [e2e] favor - please post test to E2E list In-Reply-To: <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <7B59C1C8-AEE6-4059-9021-C582ED49A8B2@netapp.com> Message-ID: <52388EB7.9@isi.edu> On 9/14/2013 12:17 AM, Eggert, Lars wrote: > Just configure mailman to reject instead of discard, so people get a notification. > > Lars The correct behavior would have the SMTP server reject the mail, but by the time mailman sees posts they have already been accepted by SMTP. We have chose not to respond to mail from non-list subscribers because this responds to spam sources. Joe (list admin) From touch at isi.edu Tue Sep 17 10:20:02 2013 From: touch at isi.edu (Joe Touch) Date: Tue, 17 Sep 2013 10:20:02 -0700 Subject: [e2e] Fwd: favor - please post test to E2E list In-Reply-To: <5234AB2C.9060903@detlef-bosau.de> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <5234AB2C.9060903@detlef-bosau.de> Message-ID: <52388F42.1060804@isi.edu> On 9/14/2013 11:30 AM, Detlef Bosau wrote: > Am 14.09.2013 00:48, schrieb Matt Mathis: >> The problem is that you have to be subscribed to the end2end-interest list >> *exactly as your mail agent/MTA sends your mail. Be aware that >> the end2end list is configured to silently discard mail from addresses >> which are not members of the list. So for example if you are subscribed >> as me at mydomain.com, but your mail agent send your messages from >> me at myhost.mydoamin.com, you will see messages from the list, but your posts >> will be silently discarded. > > At least, this is not the whole problem. > > When I post from detlef.bosau at web.de, I first my posts did not appear - > meanwhile I get a "message could not be delivered" notification. > I reported the problem to my mail operator - but I did not yet get an > answer. We have not found any indication of those posts having arrived at our servers. We do see logs of posts that are silently rejected. At this time, we have insufficient evidence that the error is at our servers. PS - we did also see a recent shift in posts whose attachments were multipart/signed rather than multipart/text; the list had previously been configured to reject anything except plaintext or multipart/text, but that has been corrected. That did not affect posts from this address, though. Joe (list admin) From mail at detlef-bosau.de Tue Sep 17 11:57:56 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Tue, 17 Sep 2013 20:57:56 +0200 Subject: [e2e] Fwd: favor - please post test to E2E list In-Reply-To: <52388F42.1060804@isi.edu> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <5234AB2C.9060903@detlef-bosau.de> <52388F42.1060804@isi.edu> Message-ID: <5238A634.3030006@detlef-bosau.de> Am 17.09.2013 19:20, schrieb Joe Touch: > > > On 9/14/2013 11:30 AM, Detlef Bosau wrote: >> Am 14.09.2013 00:48, schrieb Matt Mathis: >>> The problem is that you have to be subscribed to the >>> end2end-interest list >>> *exactly as your mail agent/MTA sends your mail. Be aware that >>> the end2end list is configured to silently discard mail from addresses >>> which are not members of the list. So for example if you are >>> subscribed >>> as me at mydomain.com, but your mail agent send your messages from >>> me at myhost.mydoamin.com, you will see messages from the list, but >>> your posts >>> will be silently discarded. >> >> At least, this is not the whole problem. >> >> When I post from detlef.bosau at web.de, I first my posts did not appear - >> meanwhile I get a "message could not be delivered" notification. >> I reported the problem to my mail operator - but I did not yet get an >> answer. > > We have not found any indication of those posts having arrived at our > servers. We do see logs of posts that are silently rejected. At this > time, we have insufficient evidence that the error is at our servers. Unfortunately, I did not yet get a reply from my provider. This kind of problems can turn out to be extremely nasty. -- ------------------------------------------------------------------ Detlef Bosau Galileistra?e 30 70565 Stuttgart Tel.: +49 711 5208031 mobile: +49 172 6819937 skype: detlef.bosau ICQ: 566129673 detlef.bosau at web.de http://www.detlef-bosau.de From Jon.Crowcroft at cl.cam.ac.uk Tue Sep 17 14:31:29 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Tue, 17 Sep 2013 22:31:29 +0100 Subject: [e2e] Fwd: favor - please post test to E2E list In-Reply-To: <52388F42.1060804@isi.edu> References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <5234AB2C.9060903@detlef-bosau.de> <52388F42.1060804@isi.edu> Message-ID: sure - here's a test: http://www.scs.stanford.edu/~dm/home/papers/remove.pdf In missive <52388F42.1060804 at isi.edu>, Joe Touch typed: >> >> >>On 9/14/2013 11:30 AM, Detlef Bosau wrote: >>> Am 14.09.2013 00:48, schrieb Matt Mathis: >>>> The problem is that you have to be subscribed to the end2end-interest list >>>> *exactly as your mail agent/MTA sends your mail. Be aware that >>>> the end2end list is configured to silently discard mail from addresses >>>> which are not members of the list. So for example if you are subscribed >>>> as me at mydomain.com, but your mail agent send your messages from >>>> me at myhost.mydoamin.com, you will see messages from the list, but your posts >>>> will be silently discarded. >>> >>> At least, this is not the whole problem. >>> >>> When I post from detlef.bosau at web.de, I first my posts did not appear - >>> meanwhile I get a "message could not be delivered" notification. >>> I reported the problem to my mail operator - but I did not yet get an >>> answer. >> >>We have not found any indication of those posts having arrived at our >>servers. We do see logs of posts that are silently rejected. At this >>time, we have insufficient evidence that the error is at our servers. >> >>PS - we did also see a recent shift in posts whose attachments were >>multipart/signed rather than multipart/text; the list had previously >>been configured to reject anything except plaintext or multipart/text, >>but that has been corrected. That did not affect posts from this >>address, though. >> >>Joe (list admin) cheers jon From touch at isi.edu Tue Sep 17 15:04:09 2013 From: touch at isi.edu (Joe Touch) Date: Tue, 17 Sep 2013 15:04:09 -0700 Subject: [e2e] No more tests needed In-Reply-To: References: <522A8532.5090503@isi.edu> <522DDD7A.9090008@isi.edu> <522DEFDC.70800@isi.edu> <522E0CCF.1070700@isi.edu> <522E365D.90608@isi.edu> <5234AB2C.9060903@detlef-bosau.de> <52388F42.1060804@isi.edu> Message-ID: The tests have been completed. No further test posts are needed. On Sep 17, 2013, at 2:31 PM, Jon Crowcroft wrote: > sure - here's a test: > http://www.scs.stanford.edu/~dm/home/papers/remove.pdf And info on unsubscribing is in the web pages for this list. Joe (as list admin) From mail at detlef-bosau.de Thu Sep 19 13:04:44 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Thu, 19 Sep 2013 22:04:44 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: <522F28BB.20108@detlef-bosau.de> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> Message-ID: <523B58DC.2020009@detlef-bosau.de> Unfortunately, the discussion I wanted to initialte did not really start perhaps due to the difficulties with some posts? Nevertheless, I would appreciate any comment on my remarks. Thanks, Detlef From keithw at mit.edu Thu Sep 19 20:10:57 2013 From: keithw at mit.edu (Keith Winstein) Date: Thu, 19 Sep 2013 23:10:57 -0400 Subject: [e2e] TCP ex Machina In-Reply-To: <523B58DC.2020009@detlef-bosau.de> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> <523B58DC.2020009@detlef-bosau.de> Message-ID: Hello Detlef, I appreciate your thoughts. You wrote, "I'm not completely convinced that this approach actually yields a congestion control algorithm which can be generalized and works with an arbitrary scenario." Of course, neither are we. As I wrote, this is academic research, in simulation, and we hope it will represent the beginning of a longer conversation. To be clear, no current CC scheme works with arbitrary scenarios. Human-designed loss-triggered schemes do poorly if the network has big buffers or exposes stochastic loss. Delay-triggered schemes have had other difficulties that you noted. The best in-network schemes to date don't work well with the varying link speeds seen in wireless links. All of these schemes make assumptions about the network. We think it would be better if the assumptions and goals were explicit. And if you're going to do that, you may as well try to make the scheme be a function of those inputs. In our paper, we showed that computers can generate congestion-control schemes that outperform human-designed schemes over particular ranges of simulated example networks. We publish these sets of scenarios and the simulation setup for turnkey replication. (http://web.mit.edu/remy) What would be cool would be if you used the Remy tool and demonstrated concrete use cases where a computer-generated scheme could not be developed to give acceptable performance over the whole set. That would be pretty interesting! And then we can talk about, well, why does that happen, and whether there is another congestion signal or search strategy we should add that fixes the problem, or whether this represents a more fundamental difficulty. We've shown it works -- in certain ranges of scenarios. If you can show it doesn't work in other cases, that's very interesting and would motivate further discussion. We are trying to break it too! Best regards, Keith On Thu, Sep 19, 2013 at 4:04 PM, Detlef Bosau wrote: > Unfortunately, the discussion I wanted to initialte did not really start > perhaps due to the difficulties with some posts? > > Nevertheless, I would appreciate any comment on my remarks. > > > Thanks, > > Detlef > From Jon.Crowcroft at cl.cam.ac.uk Fri Sep 20 02:22:11 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Fri, 20 Sep 2013 10:22:11 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> <523B58DC.2020009@detlef-bosau.de> Message-ID: yes, i think it is hard (probably patholiogically hard) to come up with a network scenario that would make you do (significantly) worse than human designed CCs... purely hypothetically, one can imagine an adversarial network designer, designing a total extinction event for any evolutionary congestion control system - however, such a designer wou;dn't get employed for very long - i suggest that it is reasonable to consider that network designers (e.g. queuing discipline hackers in switch/router companies, buffer dimensioning scientists in ISPs and Telcos) aren't really trying to be adversarial, and are more inclined to consider the types of customer flows they are supporting thinkin about the axelrod game one can play, the scenario is one where players are competing for a share of flow, but players are members of species, for whom there are many individuals, so the strategies for each species compete, but dont want to drive flow rate (population) to zero for other individuals in the same gene pool, so the evolution of a mix of cooperation (i.e. tcp friendliness) is inevitable - designers who manually avoid this will end up being blocked by ISPs (thru deployment of TCP-bad detectors in all those middleboxes (that already mung with tcp))... imho In missive , Keith Winstein typed: >>Hello Detlef, >> >>I appreciate your thoughts. >> >>You wrote, "I'm not completely convinced that this approach actually yields >>a congestion control algorithm which can be generalized and works with an >>arbitrary scenario." >> >>Of course, neither are we. As I wrote, this is academic research, in >>simulation, and we hope it will represent the beginning of a longer >>conversation. >> >>To be clear, no current CC scheme works with arbitrary scenarios. >>Human-designed loss-triggered schemes do poorly if the network has big >>buffers or exposes stochastic loss. Delay-triggered schemes have had other >>difficulties that you noted. The best in-network schemes to date don't work >>well with the varying link speeds seen in wireless links. >> >>All of these schemes make assumptions about the network. We think it would >>be better if the assumptions and goals were explicit. And if you're going >>to do that, you may as well try to make the scheme be a function of those >>inputs. >> >>In our paper, we showed that computers can generate congestion-control >>schemes that outperform human-designed schemes over particular ranges of >>simulated example networks. We publish these sets of scenarios and the >>simulation setup for turnkey replication. (http://web.mit.edu/remy) >> >>What would be cool would be if you used the Remy tool and demonstrated >>concrete use cases where a computer-generated scheme could not be developed >>to give acceptable performance over the whole set. That would be pretty >>interesting! And then we can talk about, well, why does that happen, and >>whether there is another congestion signal or search strategy we should add >>that fixes the problem, or whether this represents a more fundamental >>difficulty. >> >>We've shown it works -- in certain ranges of scenarios. If you can show it >>doesn't work in other cases, that's very interesting and would motivate >>further discussion. We are trying to break it too! >> >>Best regards, >>Keith >> >> >>On Thu, Sep 19, 2013 at 4:04 PM, Detlef Bosau wrote: >> >>> Unfortunately, the discussion I wanted to initialte did not really start >>> perhaps due to the difficulties with some posts? >>> >>> Nevertheless, I would appreciate any comment on my remarks. >>> >>> >>> Thanks, >>> >>> Detlef >>> cheers jon From dhavey at yahoo.com Fri Sep 20 10:21:55 2013 From: dhavey at yahoo.com (Daniel Havey) Date: Fri, 20 Sep 2013 10:21:55 -0700 (PDT) Subject: [e2e] TCP ex Machina In-Reply-To: <523B58DC.2020009@detlef-bosau.de> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> <523B58DC.2020009@detlef-bosau.de> Message-ID: <1379697715.68455.YahooMailNeo@web163004.mail.bf1.yahoo.com> I think that the conversation is getting ramped up now ;^) | My thoughts on the paper are mixed. ?I believe that centralized server calculated CC is the wrong direction for the future. ?IMHO there are simply to many clients in network conditions that are too diverse, and too rapidly changing. ?One size doesn't fit all. ?I think that the client should provide the congestion control algorithm. So the paper takes a nice step in this direction. ?The client provides the network conditions to the server, then the server sends the CC algorithm to the client. ?This still seems like an overload of work for the server, but, it is necessary because of the ex Machina part of the paper. I think that the client should control the CC algorithm. ?This takes care of a host of problems. ?The client is in a better position to understand it's ever changing network conditions than the server. Also I think that relying on things like packet inter-arrival time for CC is a foolish mistake. ?There was a masters project that did this at my University. ?TCP Santa Barbara. ?It performed well in simulation with other TCP Santa Barbaras. ?However, it could not play nice with VJCCs, nor, could it survive in the wild. ?One has to be careful with parameters like inter-arrival time. ?They are interesting, but, can only be relied upon in specific network conditions. My final thought is that this new Yahoo mail interface should burn in hell. ...Daniel ________________________________ From: Detlef Bosau To: end2end-interest at postel.org Sent: Thursday, September 19, 2013 1:04 PM Subject: Re: [e2e] TCP ex Machina Unfortunately, the discussion I wanted to initialte did not really start perhaps due to the difficulties with some posts? Nevertheless, I would appreciate any comment on my remarks. Thanks, Detlef From mail at detlef-bosau.de Fri Sep 20 15:32:03 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Sat, 21 Sep 2013 00:32:03 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: <1379697715.68455.YahooMailNeo@web163004.mail.bf1.yahoo.com> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> <523B58DC.2020009@detlef-bosau.de> <1379697715.68455.YahooMailNeo@web163004.mail.bf1.yahoo.com> Message-ID: <523CCCE3.3050209@detlef-bosau.de> Am 20.09.2013 19:21, schrieb Daniel Havey: > I think that the conversation is getting ramped up now ;^) > | > My thoughts on the paper are mixed. I believe that centralized server calculated CC is the wrong direction for the future. Me too. The more I think about it, the more I'm convinced that one of the most fundamental design errors is to put CC on the end points. This has become a religion and now, it is more likely that women become priest in the roman catholic church than that this basic flaw will ever be fixed. > IMHO there are simply to many clients in network conditions that are too diverse, and too rapidly changing. One size doesn't fit all. I think that the client should provide the congestion control algorithm. Have a look for the TEAR approach. Actually, it doesn't make a difference whether you place CC at the sender side or the receiver side - congestion occurs along the path and hence must be fixed there - and not at the end points. > > So the paper takes a nice step in this direction. No. It runs into the same old vicious cycles which lead to nowhere in the last 20 years. > The client provides the network conditions to the server, Daniel, the only one to provide network conditions to anyone is the network - more precisely the node, where congestion occurs. And I really wonder, why we did not think that way the last 20 years. End points may report end point conditions. Network conditions must be provided by the network. Not by endpoints, be it an educated guess, some sophisticated conjecture or simply throwing dice. From Jon.Crowcroft at cl.cam.ac.uk Sat Sep 21 00:43:52 2013 From: Jon.Crowcroft at cl.cam.ac.uk (Jon Crowcroft) Date: Sat, 21 Sep 2013 08:43:52 +0100 Subject: [e2e] TCP ex Machina In-Reply-To: <523CCCE3.3050209@detlef-bosau.de> References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> <523B58DC.2020009@detlef-bosau.de> <1379697715.68455.YahooMailNeo@web163004.mail.bf1.yahoo.com> <523CCCE3.3050209@detlef-bosau.de> Message-ID: In missive <523CCCE3.3050209 at detlef-bosau.de>, Detlef Bosau typed: >>And I really wonder, why we did not think that way the last 20 years. we did - longer - since DECBit, ECN, re-ecn, congestion exposure - lots of people did. >>End points may report end point conditions. Network conditions must be >>provided by the network. Not by endpoints, be it an educated guess, some >>sophisticated conjecture or simply throwing dice. however, notwithstanding the failure to deploy lots of smart thinking to get explict congestion feedback out of the nodes in the net, end system inference is not an _educated guess - its a perfectly good statistical technique, and there have been lots of steps refining the inference based techniques to improve end system CC in the absence of network exposure of explict congestion information - obviously they arent perfect, but neither is ECN - without instantaneous action-at-a-distance, ANY distributed technique has to cope with latency and noise - so by the time a source received explicit information about network conditions, it is out of date already, and said explicit information may not be 100% reliably delivered - so ECN and conext are also just refininements, in the way fancier controllers and filters are just refinements....just the explicit feedback is potentially a slightly bigger refinement note that congested nodes in the net can't predict what a bunch of end node/traffic sources are _about_ to do either, so you still need all the feedback control loop paraphanalia all over the place....its just good engineering... cheers jon From mail at detlef-bosau.de Sun Sep 22 14:59:20 2013 From: mail at detlef-bosau.de (Detlef Bosau) Date: Sun, 22 Sep 2013 23:59:20 +0200 Subject: [e2e] TCP ex Machina In-Reply-To: References: <522DE87A.4020801@detlef-bosau.de> <522DFE0B.2020901@web.de> <8A198FC4-01BB-4C72-A6F4-65A03448B763@csail.mit.edu> <522F28BB.20108@detlef-bosau.de> <523B58DC.2020009@detlef-bosau.de> <1379697715.68455.YahooMailNeo@web163004.mail.bf1.yahoo.com> <523CCCE3.3050209@detlef-bosau.de> Message-ID: <523F6838.4030807@detlef-bosau.de> Am 21.09.2013 11:59, schrieb Martin Geddes: > > The converse, however, is true under some circumstances: nodes in the > network can accurately predict what the self-contention effects of > already-sent traffic will be, given knowledge of the static properties > of the onward path. If I pump packets back-to-back down a gigabit link > to a known megabit bottleneck, I can with certainty say they will > contend. That opens up the possibility for much smarter ways of > trading contention and spacing traffic. However, receiving the > consequent benefits requires letting go of some cherished beliefs > about how data networks ought to operate. > I'm still to understand "static properties" here. From garmitage at swin.edu.au Thu Sep 26 13:22:13 2013 From: garmitage at swin.edu.au (grenville armitage) Date: Fri, 27 Sep 2013 06:22:13 +1000 Subject: [e2e] New: FreeBSD TCP stack inside NS3 simulations Message-ID: <52449775.6010906@swin.edu.au> All, Hopefully of interest to this list (and apologies in advanced if you see this on multiple mailing lists). I'm please to announce the v0.1 release of our NS-3/NSC environment for doing network traffic simulations, using the actual FreeBSD TCP stack inside the simulations. The tool itself has been released as a VirtualBox virtual machine (VM) appliance. This VM provides a FreeBSD 9-based turn-key environment to experiment with the CAIA NS-3/NSC modifications. Patched versions of the NS-3 and NSC code are provided, along with ready-to-compile-and-run example simulation code (multi-leaf dumbbell topology and incast topology using FreeBSD 9's TCP stack in the simulated end hosts). Simple scripts to produce initial plots of simulation output are also provided as a starting point. The VirtualBox ova can be found at http://caia.swin.edu.au/urp/incast/tools.html Many more details can be found in the README at http://caia.swin.edu.au/urp/incast/tools/README.caia-freebsd9-amd64-caia-ns3-release-0.1.txt This work is the culmination of significant effort by Lawrence Stewart (lstewart at freebsd.org) as part of his PhD work, and was supported in part by a gift from the Cisco University Research Program Fund. cheers, gja -- Professor Grenville Armitage Director, Centre for Advanced Internet Architectures Faculty of Information and Communication Technologies Swinburne University of Technology, Australia http://caia.swin.edu.au