Socket support for TCP_QUICKACK

[felipepessoto]

AB#1117049
System.Net.Sockets.Socket currently implements cross platform support for a large number of manually configurable socket options. We allow many TCP specific options, but do not support the standard TCP_QUICKACK option that is supported in both Windows and Unix.

Rationale and Usage

Nagles algorithm and TCP delayed ack are both used to solve similar problems around network congestion. Both are often enabled by default, but the interactions between them can cause extreme and unnecessary latency. For more information see this blog on MSDN, or this SD comment by John Nagle.

By enabling support for TCP_QUICKACK, we allow performance oriented developers to choose how to balance network congestion and latency.

The new option would be set via the standard SetSocketOption API. As the option is already supported natively on all supported platforms, the changes required would be very minimal.

Proposed API

namespace System.Net.Sockets
{
    // Defines socket option names for the <see cref='System.Net.Sockets.Socket'/> class.
    public enum SocketOptionName
    {
        #region SocketOptionLevel.Tcp
        TcpQuickAck = 12,
        #endregion
    }
}

[[API proposal added above by @rmkerr. Original issue below]]

Original Proposal

I'm not sure if Windows support this at socket level as Linux does. But delayed ack does not works well when the remote endpoint is using Nagle Algorithm (TCP_NODELAY = false).

Reference: https://blogs.technet.microsoft.com/nettracer/2013/01/05/tcp-delayed-ack-combined-with-nagle-algorithm-can-badly-impact-communication-performance/

[karelz]

I couldn't find existing option on Windows sockets APIs either.
@davidsh should we ping Windows folks to confirm?

cc @geoffkizer

[davidsh]

@davidsh should we ping Windows folks to confirm?

Yes @rmkerr could follow up with the usual TCP team that he already has worked with.

[rmkerr]

I'll reach out to the TCP team to confirm. In the meantime, I recommend checking out the documentation here on managing some of the issues with the Nagle algorithm and delayed ack.

[felipepessoto]

@rmkerr, I read that article, one of the issues I intend to prevent is the problem with Nagle algorithm

[rmkerr]

It looks like WinSock does support this! We can either set TCP_QUICKACK via setsocketopt or SIO_TCP_SET_ACK_FREQUENCY via WSAIoctl. In the interest of cross platform consistency we should probably use TCP_QUICKACK.

I think that this is definitely worth taking through API review, as it is a very useful tool for reducing latency when the Nagle algorithm is enabled.

[rmkerr]

System.Net.Sockets.Socket currently implements cross platform support for a large number of manually configurable socket options. We allow many TCP specific options, but do not support the standard TCP_QUICKACK option that is supported in both Windows and Unix.

Rationale and Usage

Nagles algorithm and TCP delayed ack are both used to solve similar problems around network congestion. Both are often enabled by default, but the interactions between them can cause extreme and unnecessary latency. For more information see this blog on MSDN, or this SD comment by John Nagle.

By enabling support for TCP_QUICKACK, we allow performance oriented developers to choose how to balance network congestion and latency.

The new option would be set via the standard SetSocketOption API. As the option is already supported natively on all supported platforms, the changes required would be very minimal.

Proposed API

namespace System.Net.Sockets
{
    // Defines socket option names for the <see cref='System.Net.Sockets.Socket'/> class.
    public enum SocketOptionName
    {
        #region SocketOptionLevel.Tcp
        TcpQuickAck = 12,
        #endregion
    }
}

[karelz]

Should be fairly straightforward - model it by other options
We need to make sure the TCP_QUICKACK is part of public Windows headers (or we need to wait for public MSDN docs). The number (12) should match the value in them (on Windows).

[felipepessoto]

@rmkerr, These TCP constant options are the same in all SO? http://www.cse.scu.edu/~dclark/am_256_graph_theory/linux_2_6_stack/linux_2tcp_8h-source.html.
How did you find the constant 12 in Windows?

[rmkerr]

This issue actually needs to be updated -- the API is slightly different than expected on Windows. Rather than using the standard setsocketopt interface, Windows allows for more granular control over ack frequency via the WSAIoctl control code SIO_TCP_SET_ACK_FREQUENCY. That means that we can add the ability to disable delayed ack on all platforms, but the implementation may need to be platform specific. I think the implementation details need more discussion. Depending on the results of that discussion we may need to go through API review again.

[alfredmyers]

While researching on the topic, I found an issue over on the Kestrel repo.

The folks over there had an issue about it but it was closed aspnet/KestrelHttpServer#480

[juliusfriedman]

I have some methods related to this however by other names, e.g

I believe the name "CongestionAlgorithm" is appropriate and the api allows a broader range of options:

      #region CongestionAlgorithm

        const int TcpCongestionAlgorithm = 12;

        static readonly System.Net.Sockets.SocketOptionName TcpCongestionAlgorithmOption = (System.Net.Sockets.SocketOptionName)TcpCongestionAlgorithm;

        public static void SetTcpCongestionAlgorithm(System.Net.Sockets.Socket socket, int value = 1)
        {
            SetTcpOption(socket, TcpCongestionAlgorithmOption, value);
        }

        public static void DisableTcpCongestionAlgorithm(System.Net.Sockets.Socket socket)
        {
            SetTcpCongestionAlgorithm(socket, 0);
        }

        public static void EnableTcpCongestionAlgorithm(System.Net.Sockets.Socket socket, int algorithmType = 1)
        {
            SetTcpCongestionAlgorithm(socket, algorithmType);
        }

        #endregion

Also for Retransmission:

    #region Retransmission

        static System.Net.Sockets.SocketOptionName TcpMaximumRetransmissionOptionName = (System.Net.Sockets.SocketOptionName)(Common.Extensions.OperatingSystemExtensions.IsWindows ? 5 : 18);

        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public static void SetMaximumTcpRetransmissionTime(System.Net.Sockets.Socket socket, int amountInSeconds = 3)
        {
            //On windows this is TCP_MAXRT elsewhere USER_TIMEOUT

            //Mono checks the options and will not call setsocketopt if the name is not known to the Mono Runtime.
            //A work around would be to either define the call for get and set socketopt in this library or call the SetSocketOption_internal method for mono.
            SetTcpOption(socket, TcpMaximumRetransmissionOptionName, amountInSeconds);
        }

        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public static void DisableTcpRetransmissions(System.Net.Sockets.Socket socket)
        {
            SetMaximumTcpRetransmissionTime(socket, 0);
        }

        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public static void EnableTcpRetransmissions(System.Net.Sockets.Socket socket)
        {
            SetMaximumTcpRetransmissionTime(socket);
        }

        public static int GetMaximumTcpRetransmissionTime(System.Net.Sockets.Socket socket)
        {
            int len = Common.Binary.BytesPerInteger;

            //Todo, static local allocation
            byte[] value = socket.GetSocketOption(System.Net.Sockets.SocketOptionLevel.Tcp, TcpMaximumRetransmissionOptionName, len);

            if (Common.Extensions.Array.ArrayExtensions.IsNullOrEmpty(value, out len) || len < Common.Binary.BytesPerInteger) return -1;

            return Common.Binary.Read32(value, Common.Binary.BytesPerInteger, BitConverter.IsLittleEndian);

        }

        #endregion

And for others (most already with baked in checked for OS) see(https://github.com/juliusfriedman/net7mma/blob/master/Common/Extensions/SocketExtensions.cs):

NoSynRetries, Timestamp, OffloadPreference, DelayFinAck, GetMaximumTransmittableUnit, Urgency and last but not least MaximumSegmentSize.

[stephentoub]

Depending on the results of that discussion we may need to go through API review again.

Removing api-approved until this is resolved.

[scalablecory]

I think we need someone to dive into these options and do some prototyping on Linux and Windows to understand if we can make this portable.

[github-cygwin]

Given that socket option 12 is called TCP_CONGESTION_ALGORITHM, taking values from 0..7, wouldn't it make more
sense to assume that TCP_DELAY_FIN_ACK, value 13, a boolean option, is the WinSock equivalent to TCP_QUICKACK?
The name seems to indicate a relationship, but the documentation is inexistent as well :(

[juliusfriedman]

I think we need someone to dive into these options and do some prototyping on Linux and Windows to understand if we can make this portable.

This would be done by either runtime checks of ifdef on the rid but I highly suggest you keep it runtime since technically kernel modules can be loaded and unloaded while processes are running... not to mentioned patched. Perhaps static readonly for the process.

[antonfirsov]

I did carry out an experiment based on Case Study 1 in this article. On Windows I was able to disable delayed acks and benchmark significant improve in throughput using the SIO_TCP_SET_ACK_FREQUENCY option:

private const int SIO_TCP_SET_ACK_FREQUENCY = unchecked((int)0x98000017);
private static readonly byte[] Dummy = new byte[0];

private static void DisableDelayedAck(Socket socket)
{
    socket.IOControl(SIO_TCP_SET_ACK_FREQUENCY, BitConverter.GetBytes(1), Dummy);
}

I tried setting TCP_QUICKACK by calling socket.SetSocketOption(SocketOptionLevel.Tcp, (SocketOptionName)12, 1) on both Windows and Ubuntu 18.04, but got a SocketException with SocketError.OperationNotSupported on both platforms.

In case of Linux, this is quite surprising since the kernel 2.4.4+ is expected to support the option, and the value should be 12. Maybe I'm doing something wrong, will keep experimenting.

[stephentoub]

I tried setting TCP_QUICKACK by calling socket.SetSocketOption(SocketOptionLevel.Tcp, (SocketOptionName)12, 1) on both Windows and Ubuntu 18.04, but got a SocketException with SocketError.OperationNotSupported on both platforms. In case of Linux, this is quite surprising since the kernel 2.4.4+ is expected to support the option, and the value should be 12. Maybe I'm doing something wrong, will keep experimenting.

SetSocketOption takes a SocketOptionName, with values that don't necessarily map to the corresponding values on each platform (since each platform often has different values from each other). As such, the values go through a translation layer in the PAL, so unless you updated that as well, it's going to try to translate "12" into whatever the PAL says that maps to, or fail if there's no mapping.

If you just want to try to pass "12" directly for experimentation purposes, try using SetRawSocketOption instead.

[antonfirsov]

I did some investigations, sharing my results:

• TCP_QUICKACK is Linux-specific. It doesn't look like any other Unix implements it.

• According to the manual, and the analysis in this article (point 3) the options is not permanent, therefore it needs to be reset after each recv call. In my very basic experiment it persisted though.

• In .NET 5 it can be set with SetRawSocketOption, and it worked (improved throughput on Linux) in my experiments:

socket.SetRawSocketOption((int) SocketOptionLevel.Tcp, 12, BitConverter.GetBytes(1));

• Setting TCP_QUICKACK == 12 doesn't work on Windows and leads to an error. (@github-cygwin neither does TCP_DELAY_FIN_ACK == 13 work -- it seems to be something completely unrelated)
• On Windows it's possible to disable delayed ACKs by changing SIO_TCP_SET_ACK_FREQUENCY. See my previous Socket support for TCP_QUICKACK #798 (comment) for more details. Unlike TCP_QUICKACK, this option has to be set only once.
• On macOS, there is a sysctl option net.inet.tcp.delayed_ack but I wasn't able to find a way to control it for individual sockets.

Conclusion:

I think it's really questionable whether we want to provide a cross platform option for disabling delayed acks. The main issue is the difference between the non-persistent nature of TCP_QUICKACK on Linux vs. the persistent behavior of SIO_TCP_SET_ACK_FREQUENCY on Windows. Since we likely don't want to do an expensive syscall after each receive on Linux just to guarantee consistence for a rare use-case, the only way is to implement a setting that behaves inconsistently accross platforms.

I wonder if someone with deeper Linux knowledge can confirm or disprove my conclusions on TCP_QUICKACK. @tmds @wfurt?

EDIT: possible API

If we decide to add an API anyways, I think it should be named differently to make it clear, it's not 100% equivalent of the Linux-only TCP_QUICKACK.

public enum SocketOptionName
{
+   DisableTcpDelayedAck = 12,
}

Alternatively, we may decide to add a property on Socket instead:

public class Socket
{
+   public bool DisableTcpDelayedAck { get; set; }
}

[github-cygwin]

* Setting `TCP_QUICKACK == 12` doesn't work on Windows and leads to an error. (@github-cygwin neither does `TCP_DELAY_FIN_ACK == 13` work -- it seems to be something completely unrelated)

@antonfirsov, can you please outline which error that is? Setting this option works, afaics, so are you talking about a later error in read/write? Which error code is returned? And on which OS are you testing?

Edit: Per my testing setting values from 0..7 works, but I have no idea what they are supposed to do...

Thanks,
Corinna

[antonfirsov]

@github-cygwin trying to threat option 12 as TCP_QUICKACK resulted in OperationNotSupported (10045) for me (I know this is probably unexpected, I may try again and post code, but don't think it will change the big picture, it's a different switch anyways.). Setting values 0 or 1 on TCP_DELAY_FIN_ACK (13) does not lead to an error, but does not have an impact on delayed ack behavior.

[juliusfriedman]

Whats wrong with my proposal for CongestionAlgorithmn et al? it allows for parameters also... see the KeepAlive example

[antonfirsov]

@juliusfriedman TCP_CONGESTION_ALGORITHM (12) seems to be an undocumented Windows feature. We have no idea on what does it exactly, and how it is related to TCP_QUICKACK (Linux) or SIO_TCP_SET_ACK_FREQUENCY (Windows).

I suggest to keep focus of this issue on disabling delayed acks.

[juliusfriedman]

@antonfirsov

On some Windows Server versions I believe it controls various attributes of the TCP transport known as congestion control see Here. I believe you should have a look since it controls things such as ack timeout and duplicate ack etc and is supported on linux and mac.

Why are we disabling something specific like delayed ack while not having options for retransmission etc al so that people can use their own algorithmic means when required for their applications, they can already use interop or set socket options to do so right?

Typing of such we should really have methods for keep alive as depending on what is in transmission over TCP the application layer might require it. (Just like Delayed Ack, Retransmission etc)

https://serverfault.com/questions/236913/windows-tcp-congestion-control
https://superuser.com/questions/992919/how-to-check-the-tcp-congestion-control-algorithm-flavour-in-ubuntu
https://apple.stackexchange.com/questions/12062/how-do-you-set-the-tcp-congestion-control-algorithm-on-os-x

My proposed would be similar to the API I created for TCP ReTransmissions indicating that 0 is disabled and any other value is the value corresponding in the spec required e.g. time in msec etc.

https://github.com/juliusfriedman/net7mma_core/blob/master/Common/Extensions/SocketExtensions.cs#L685

Here is the API which I had for QuickAck

https://github.com/juliusfriedman/net7mma_core/blob/master/Common/Extensions/SocketExtensions.cs#L1094 (It does not include the custom time option but is otherwise the same)

So you would have at least 2 methods but probably 3 or more 2 which calls the other:

 #region QuickAck

        static System.Net.Sockets.SocketOptionName TcpDelayedAckOptionName = (System.Net.Sockets.SocketOptionName)(Common.Extensions.OperatingSystemExtensions.IsWindows ? 13 : 12); //Double check this

       //Uncommon case
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public static void SetDelayedAckTimeout(System.Net.Sockets.Socket socket, TimeSpan time)
        {
            SetTcpOption(socket, TcpDelayedAckOptionName, (int)time.TotalSeconds);
        }

       //Common case
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public static void DisableTcpDelayedAck(System.Net.Sockets.Socket socket)
        {
            SetDelayedAckTimeout(socket, 0);
        }

        //Common case
        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        public static void EnableTcpDelayedAck(System.Net.Sockets.Socket socket)
        {
            SetDelayedAckTimeout(socket);
        }

        public static TimeSpan GetTcpDelayedAckTime(System.Net.Sockets.Socket socket)
        {
            int len = sizeof(int);

            byte[] value = socket.GetSocketOption(System.Net.Sockets.SocketOptionLevel.Tcp, TcpDelayedAckOptionName , len);

           //Should be a framework method anyway....
            if (Common.Extensions.Array.ArrayExtensions.IsNullOrEmpty(value, out len) || len < sizeof(int)) return -1;

            return TimeSpan.FromSeconds(BinaryPrimitives.ReadInt32BigEndian(value));

        }

        #endregion

Since we are there may as well add:

  [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        internal static void SetTcpOption(System.Net.Sockets.Socket socket, System.Net.Sockets.SocketOptionName name, int value)
        {
          socket.SetRawSocketOption(System.Net.Sockets.SocketOptionLevel.Tcp, name, value);
        }

        [System.Runtime.CompilerServices.MethodImpl(System.Runtime.CompilerServices.MethodImplOptions.AggressiveInlining)]
        internal static void GetTcpOption(System.Net.Sockets.Socket socket, System.Net.Sockets.SocketOptionName name, byte[] buffer)
        {
          socket.GetRawSocketOption(System.Net.Sockets.SocketOptionLevel.Tcp, name, buffer);
        }

I would also highly suggest MaximumSegmentSize etc as well.

[antonfirsov]

Why are we disabling something specific like delayed ack while not having options for retransmission etc al so that people can use their own algorithmic means when required for their applications.

A few arguments:

• The number of users requesting the former seems to be quite high, while I don't see strong need for the latter.
• It's not a good idea to expose undocumented Windows features, unless we have a strong proof they actually work, and at least some confirmation from the Windows TCP team. These are not trivial things to implement and maintain. (Although much simpler, it looks like we even won't be able to provide a cross-platform option for quickack.)
• I don't see options to control these for individual sockets on other platforms. The posts you shared are working with global sysctl options.

(System.Net.Sockets.SocketOptionName)(Common.Extensions.OperatingSystemExtensions.IsWindows ? 13 : 12); //Double check this

According to my experiments, these options are not equivalent. TCP_DELAY_FIN_ACK is undocumented stuff, we have no information nor empiric proof on what it does.

[juliusfriedman]

Hence why I recommended what I did, if you give people the toggles to control the Retransmission time they can sort of implement these algorithms on their own. They are not asking for it because they don't know what they need because TCP is complex.

E.g. until you read the RFC text you probably don't know your application is supposed to disable re-transmission for example RFC 2326

On platform where sysctl is not available there is definitely another suitable way to accomplish the same thing but how becomes a question for that distro. What are you trying to accomplish with sysctl?

That would be because congestion control algorithms toggle a bunch of switches which is why something like sysctl is used to perform the change system wide.

On the individual sockets this would be equal to setting the options manually at the socket level...

Thats because Windows uses the TCP ACK Frequency (I believe) to achieve the same thing: See

Hence again why I recommended giving users all the knobs.

[antonfirsov]

@juliusfriedman users have the knobs now to control platform specific options with SetRawSocketOption if they want and really know what they are doing.

However, if we expose a high-level, cross-platform switch on the public API, that should be stable, proven, well documented, and consistent across platforms.

I barely see a way to achieve this for disabling delayed acks. How can we manage it for some undocumented option where we can only guess the behavior based on names in Windows headers and StackOverflow folklore?

[juliusfriedman]

I have had them in use in my library for some time and have never had any issues, I would expect the greatest level of compatibility to be able to be achieved on Linux simply due to the open source nature.

The same way we probe for CPU information we should be doing for socket information. Offloading is just as important on the network side in certain cases as it is on the GPU. (Special network links or protocols)

Since you admit that SetRawSocketOption is a place for the advanced users what exactly was the purpose of this API proposal, review and approval?

Again you also say a lot of users are asking for it so why not given them a good API around TCP and that will satisfy them especially with easy to use methods which you can control and achieve the same thing as your delayed ack which is system wide AFAIK and not Socket level in the first place.

[antonfirsov]

Since you admit that SetRawSocketOption is a place for the advanced users what exactly was the purpose of this API proposal, review and approval?

This request came before SetRawSocketOption existed, the first approval was a mistake (result of incorrect assumptions), the api approved label has been removed since then.

the same thing as your delayed ack which is system wide AFAIK and not Socket level in the first place.

This is not true, delayed ack can be controlled on Socket level in both Linux and Windows (but with very different switches). The goal of this issue is to try to provide a unified API for that specific switch. If you think it's valuable to expose other knobs, I suggest to open a separate issue so the proposal can be discussed and assessed separately.

[juliusfriedman]

Since you admit that SetRawSocketOption is a place for the advanced users what exactly was the purpose of this API proposal, review and approval?

This request came before SetRawSocketOption existed, the first approval was a mistake (result of incorrect assumptions), the api approved label has been removed since then.

the same thing as your delayed ack which is system wide AFAIK and not Socket level in the first place.

This is not true, delayed ack can be controlled on Socket level in both Linux and Windows (but with very different switches). The goal of this issue is to try to provide a unified API for that specific switch. If you think it's valuable to expose other knobs, I suggest to open a separate issue so the proposal can be discussed and assessed separately.

That myst be sum switchboard...

[karelz]

Moving to Future based on above analysis by @antonfirsov: #798 (comment)

[ericsampson]

This would be great to see in some form in .NET 6

[antonfirsov]

It's possible to configure this on .NET 5 on Linux and Windows:

Linux (per call):

socket.SetRawSocketOption((int) SocketOptionLevel.Tcp, 12, BitConverter.GetBytes(1));

Windows (once):

socket.IOControl(SIO_TCP_SET_ACK_FREQUENCY, BitConverter.GetBytes(1), Dummy);

I can't see a good cross-platform abstraction for the option though. @ericsampson if you have a real-life use-case for this, is there any problem with the platform-specific solutions I'm suggesting?

[wfurt]

triage: workaround exist through platform specific. Since there is no consistent cross-platform it seems like bad fit for general Socket API. We can revisit this in the future.