[e2e] Need help: setting winsock receive low watermark while using completion port and TCP

[David P. Reed]

Lars Eggert wrote:

> On Aug 24, 2005, at 0:21, Alex Krivonosov (alexkr) wrote:
>
>> I have a TCP connection handled by the completion port IO model.  
>> What is happening is in case I specify a large buffer for receiving  
>> (WSARecv), the operation completes only after the buffer is full,  
>> not after receiving about 500 bytes (a packet), so a significant  
>> delay is introduced. In case of small buffers, performance  degrades. 
>> Any advice on this? Completion port model is a must.
>
>
> Please understand that TCP doesn't deliver "packets" to the  
> application, it provides a byte stream. You may want to look into  
> using non-blocking I/O for the receive call. (I don't know what you  
> mean by "completion port model.")

The definition of I/O completion in Winsock *is* buffer full.   Size of 
buffer on receive is not a major performance problem (system calls 
aren't slow compared to processing), so if you want notification on 500 
bytes, use 500 byte buffers.

A thought you might not have considered: Perhaps you are sending your 
500 byte messages, one per call, on the sender with TCP_NODELAY set?  
This could cause some performance problems if the source end has a fast 
link, but the receiving node has a slow absorption rate (the packets on 
the source will not combine into larger frames until the window fills 
up.)   Of course that is exactly what TCP_NODELAY is for (minimizing 
message latency, but increasing network overhead) - if you don't care so 
much about latency, don't set TCP_NODELAY.

(or you can get very complex by using I/O completion based app-level 
output management on the send side to control the latency/efficiency 
tradeoff, using WSASendMessage to gather mutliple frames adaptively and 
"delaying" sends at the app level until precise conditions hold related 
to your desired latency goal and trying to gather 1-3 of your messages 
into single sends).