Subject: Solaris 2.x - tuning your TCP/IP stack Type: FAQ Creation Date: 31-MAY-2000 FREQUENTLY ASKED QUESTIONS -------------------------- 30-05-2000 QUESTIONS & ANSWERS ------------------- 1.2 Quick intro into ndd Solaris allows you to tune, tweak, set and reset various parameters related to the TCP/IP stack while the system is running. Back in the SunOS 4.x days, one had to change various C files in the kernel source tree, generate a new kernel, reboot the machine and try out the changes. The Solaris feature of changing the important parameters on the fly is very convenient. Many of the parameters I mention in the rest of the document you are reading are time intervals. All intervals are measured in milliseconds. Other parameters are usually bytecounts, but a few times different units of measurements are used and documented. A few items appear totally unrelated to TCP/IP, but due to the lack of a better framework, they materialized on this page. Most tunings can be achieved using the program ndd. Any user may execute this program to read the current settings, depending on the readability of the respective device files. But only the super user is allowed to execute ndd -set to change values. This makes sense considering the sensitive parameters you are tuning. Details on the use of ndd can be obtained from the respective manual page. ndd will become your friend, as it is the major tool to tweak most of the parameters described in this document. Therefore you better make yourself familiar with it. A quick overview will be given in this section, too. ndd is not limited to tweaking TCP/IP related parameters. Many other devices, which have a device file underneath /dev and a kernel module can be configured with the help of ndd. For instance, any networking driver which supports the Data Link Provider Interface (DLPI) can be configured. The parameters supplied to ndd are symbolic keys indexing either a single usually numerically value, or a table. Please note that the keys usually (but not always) start out with the module or device name. For instance, changing values of the IP driver, you have to use the device file /dev/ip and all parameters start out with ip_. The question mark is the most notable exception to this rule. interactive mode The interactive mode allows you to inspect and modify a device, driver or module interactively. In order to inspect the available keyword names associated with a parameter, just type the question mark. The next item will explain about the output format of the parameter list. # ndd /dev/tcp name to get/set ? tcp_slow_start_initial value ? length ? 2 name to get/set ? ^D The example above queries the TCP driver for the value of the slow start feature in an interactive fashion. The typed input is shown boldface. show all available parameters If you are interested in the parameters you can tweak for a given module, query for the question mark. This special parameter name is part of all ndd configurable material. It tells the names of all parameters available - including itself - and the access mode of the parameter. # ndd /dev/icmp \? ? (read only) icmp_wroff_extra (read and write) icmp_def_ttl (read and write) icmp_bsd_compat (read and write) icmp_xmit_hiwat (read and write) icmp_xmit_lowat (read and write) icmp_recv_hiwat (read and write) icmp_max_buf (read and write) Please mind that you have to escape the question mark with a backslash from the shell, if you are querying in the non-interactive fashion as shown above. query the value of one or more parameters (read access) At the command line, you often need to check on settings of your TCP/IP stack or other parameters. By supplying the parameter name, you can examine the current setting. It is permissible to mention several parameters to check on at once. # ndd /dev/udp udp_smallest_anon_port 32768 # ndd /dev/hme link_status link_speed link_mode 1 1 1 The first example checks on the smallest anonymous port UDP may use when sending a PDU. Please refer to the appropriate section later in this document on the recommended settings for this parameter. The second example checks the three important link report values of a 100 Mbit ethernet interface. The results are separated by an empty line, because some parameters may refer to tabular values instead of a single number. Modify the value of one parameter (write access) This mode of interaction with ndd will frequently be found in scripts or when changing value at the command line in a non-interactive fashion. Please note that you may only set one value at a time. The scripts section below contains examples in how to make changes permanent using a startup script. # ndd -set /dev/ip ip_forwarding 0 The example will stop the forwarding of IP PDUs, even if more than one non-local interface is active and up. Of course, you can only change parameters which are marked for both, reading and writing. Andres Kroonmaa kindly supplied a nifty script to check all existing values for a network component (tcp, udp, ip, icmp, etc.). Usually I do the same thing using a small Perl script. 2. TCP connection initiation This section is dedicated exclusively to the various queues and tunable variable(s) used during connection instantiation. The socket API maintains some control over the queues. But in order to tune anything, you have to understand how listen and accept interact with the queues. For details, see the various Stevens books mentioned in the literature section. When the server calls listen, the kernel moves the socket from the TCP state CLOSED into the state LISTEN, thus doing a passive open. All TCP servers work like this. Also, the kernel creates and initializes various data structures, among them the socket buffers and two queues: incomplete connection queue This queue contains an entry for every SYN that has arrived. BSD sources assign so_q0len entries to this queue. The server sends off the ACK of the client's SYN and the server side SYN. The connection get queued and the kernel now awaits the completion of the TCP three way handshake to open a connection. The socket is in the SYN_RCVD state. On the reception of the client's ACK to the server's SYN, the connection stays one round trip time (RTT) in this queue before the kernel moves the entry into the completed connection queue This queue contains an entry for each connection for which the three way handshake is completed. The socket is in the ESTABLISHED state. Each call to accept() removes the front entry of the queue. If there are no entries in the queue, the call to accept usually blocks. BSD source assign a length of so_qlen to this queue. Both queues are limited regarding their number of entries. By calling listen(), the server is allowed to specify the size of the second queue for completed connections. If the server is for whatever reason unable to remove entries from the completed connection queue, the kernel is not supposed to queue any more connections. A timeout is associated with each received and queued SYN segment. If the server never receives an acknowledgment for a queued SYN segment, TCP state SYN_RCVD, the time will run out and the connection thrown away. The timeout is an important resistance against SYN flood attacks. Historically, the argument to the listen function specified the maximum number of entries for the sum of both queues. Many BSD derived implementations multiply the argument with a fudge factor of 3/2. Solaris <= 2.5.1 do not use the fudge factor, but adds 1, while Solaris 2.6 does use the fudge factor, though with a slightly different rounding mechanism than the one BSD uses. With a backlog argument of 14, Solaris 2.5.1 servers can queue 15 connections. Solaris 2.6 server can queue 22 connections. Stevens shows that the incomplete connection queue does need more entries for busy servers than the completed connection queue. The only reason for specifying a large backlog value is to enable the incomplete connection queue to grow as SYN arrive from clients. Stevens shows that moderately busy webserver has an empty completed connection queue during 99 % of the time, but the incomplete connection queue needed 15 or less entries in 98 % of the time! Just try to imagine what this would mean for a really busy webcache like Squid. Data for an established connection which arrives before the connection is accept()ed, should be stored into the socket buffer. If the queues are full when a SYN arrived, it is dropped in the hope that the client will resend it, hopefully finding room in the queues then. According to Cockroft [2], there was only one listen queue for unpatched Solari <= 2.5.1. Solari >= 2.6 or an applied TCP patch 103582-12 or above splits the single queue in the two shown in figure 1. The system administrator is allowed to tweak and tune the various maxima of the queue or queues with Solaris. Depending on whether there are one or two queues, there are different sets of tweakable parameters. The old semantics contained just one tunable parameter tcp_conn_req_max which specified the maximum argument for the listen(). The patched versions and Solaris 2.6 replaced this parameter with the two new parameters tcp_conn_req_max_q0 and tcp_conn_req_max_q. A SunWorld article on 2.6 by Adrian Cockroft tells the following about the new parameters: tcp_conn_req_max [is] replaced. This value is well-known as it normally needs to be increased for Web servers in older releases of Solaris 2. It no longer exists in Solaris 2.6, and patch 103582-12 adds this feature to Solaris 2.5.1. The change is part of a fix that prevents denial of service from SYN flood attacks. There are now two separate queues of partially complete connections instead of one. tcp_conn_req_max_q0 is the maximum number of connections with handshake incomplete. A SYN flood attack could only affect this queue, and a special algorithm makes sure that valid connections can still get through. tcp_conn_req_max_q is the maximum number of completed connections waiting to return from an accept call as soon as the right process gets some CPU time. In other words, the first specifies the size of the incomplete connection queue while the second parameters assigns the maximum length of the completed connection queue. All three parameters are covered below. You can determine if you need to tweak this set of parameters by watching the output of netstat -sP tcp. Look for the value of tcpListenDrop, if available on your version of Solaris. Older versions don't have this counter. Any value showing up might indicate something wrong with your server, but then, killing a busy server (like squid) shuts down its listening socket, and might increase this counter (and others). If you get many drops, you might need to increase the appropriate parameter. Since connections can also be dropped, because listen() specifies a too small argument, you have to be careful interpreting the counter value. On old versions, a SYN flood attack might also increase this counter. Newer or patched versions of Solaris, with both queues available, will also have the additional counters tcpListenDropQ0 and tcpHalfOpenDrop. Now the original counter tcpListenDrop counts only connections dropped from the completed connection queue, and the counter ending in Q0 the drops from the incomplete connection queue. Killing a busy server application might increase either or both counters. If the tcpHalfOpenDrop shows up values, your server was likely to be the victim of a SYN flood. The counter is only incremented for dropping noxious connection attempts. I have no idea, if those will also show up in the Q0 counter, too. tcp_conn_req_max default 8 (max. 32), since 2.5 32 (max. 1024), recommended 128 <= x <= 1024 since 2.6 or 2.5.1 with patches 103630-09 and 103582-12 or above applied: see tcp_conn_req_max_q and tcp_conn_req_max_q0 The current parameter describes the maximum number of pending connection requests queued for a listening endpoint in the completed connection queue. The queue can only save the specified finite number of requests. If a queue overflows, nothing is sent back. The client will time out and (hopefully) retransmit. The size of the completed connection queue does not influence the maximum number of simultaneous established connections after they were accepted nor does it have any influence on the maximum number of clients a server can serve. With Solaris, the maximum number of file descriptors is the limiting factor for simultaneous connections, which just happened to coincide with the maximum backlog queue size. From the viewpoint of TCP those connections placed in the completed connection queue are in the TCP state ESTABLISHED, even though the application has not reaped the connection with a call to accept. That is the number limited by the size of the queue, which you tune with this parameter. If the application, for some reason, does not release entries from the queue by calling accept, the queue might overflow, and the connection is dropped. The client's TCP will hopefully retransmit, and might find a place in the queue. Solaris offers the possibility to place connections into the backlog queue as soon as the first SYN arrives, called eager listening. The three way handshake will be completed as soon as the application accept()s the connection. The use of eager listening is not recommended for production systems. Solari < 2.5 have a maximum queue length of 32 pending connections. The length of the completed connection queue can also be used to decrease the load on an overloaded server: If the queue is completely filled, remote clients will be denied further connections. Sometimes this will lead to a connection timed out error message. Naively, I assumed that a very huge length might lead to a long service time on a loaded server. Stevens showed that the incomplete connection queue needs much more attention than the completed connection queue. But with tcp_conn_req_max you have no option to tweak that particular length. When tuning tcp_conn_req_max, always do it with regards to the values of rlim_fd_max and rlim_fd_cur. This is just a rule of thumb. Setting your listen backlog queue larger than the number of file descriptors available to you won't do you any good if your service time is long. A server shouldn't accept any further connections, if it has run out of descriptors. Even though new connection won't be thrown away with a long backlog, a server might want to reduce the size to as many connections as can be serviced simultaneously. Again, you have to consider your average service time, too. There is a trick to overcome the hardcoded limit of 1024 with a patch. SunSolve shows this trick in connection with SYN flood attacks. A greatly increased listen backlog queue may offer some small increased protection against this vulnerability. On this topic also look at the tcp_ip_abort_cinterval parameter. Better, use the mentioned TCP patches, and increase the q0 length. echo "tcp_param_arr+14/W 0t10240" | adb -kw /dev/ksyms /dev/mem This patch is only effective on the currently active kernel, limiting its extend to the next boot. Usually you want to append the line above on the startup script /etd/init.d/inetinit. The shown patch increases hard limit of the listen backlog queue to 10240. Only after applying this patch you may use values above 1024 for the tcp_conn_req_max parameter. A further warning: Changes to the value of tcp_conn_req_max parameter in a running system will not take effect until each listening application is restarted. The backlog queue length is evaluated whenever an application calls listen(3N), usually once during startup. Sending a HUP signal may or may not work; personally I prefer to TERM the application and restart them manually or, even better, use a startup script. tcp_conn_req_max_q0 since 2.5.1 with patches 103630-09 and 103582-12 or above applied: default 1024; since 2.6: default 1024, recommended 1024 <= x <= 10240 After installing the mentioned TCP patches, alternatively after installing Solaris 2.6, the parameter tcp_conn_req_max is no longer available. In its stead the new parameters tcp_conn_req_max_q and tcp_conn_req_max_q0 emerged. tcp_conn_req_max_q0 is the maximum number of connections with handshake incomplete, basically the length of the incomplete connection queue. In other words, the connections in this queue are just being instantiated. A SYN was just received from the client, thus the connection is in the TCP SYN_RCVD state. The connection cannot be accept()ed until the handshake is complete, even if the eager listening is active. To protect against SYN flooding, you can increase this parameter. Also refer to the parameter tcp_conn_req_max_q above. I believe that changes won't take effect unless the applications are restarted. tcp_conn_req_max_q since 2.5.1 with patches 103630-09 and 103582-12 or above applied: default 128; since 2.6: default 128, recommended 128 <= x <= tcp_conn_req_max_q0 After installing the mentioned TCP patches, alternatively after installing Solaris 2.6, the parameter tcp_conn_req_max is no longer available. In its stead the new parameters tcp_conn_req_max_q and tcp_conn_req_max_q0 emerged. tcp_conn_req_max_q is the length of the completed connection queue. In other words, connections in this queue of length tcp_conn_req_max_q have completed the three way handshake of a TCP open. The connection is in the state ESTABLISHED. Connections in this queue have not been accept()ed by the server process (yet). Also refer to the parameter tcp_conn_req_max_q0. Remember that changes won't take effect unless the applications are restarted. tcp_conn_req_min Since 2.6: default 1, recommended: don't touch This parameter specifies the minimum number of available connections in the completed connection queue for select() or poll() to return "readable" for a listening (server) socket descriptor. Programmers should note that Stevens [7] describes a timing problem, if the connection is RST between the select() or poll() call and the subsequent accept() call. If the listening socket is blocking, the default for sockets, it will block in accept() until a valid connection is received. While this seems no tragedy with a webserver or cache receiving several connection requests per second, the application is not free to do other things in the meantime, which might constitute a problem. 3. Retransmission related parameters The retransmission timeout values used by Solaris are way too aggressive for wide area networks, although they can be considered appropriate for local area networks. SUN thus did not follow the suggestions mentioned in RFC 1122. Newer releases of the Solaris kernel are correcting the values in question: The recommended upper and lower bounds on the RTO are known to be inadequate on large internets. The lower bound SHOULD be measured in fractions of a second (to accommodate high speed LANs) and the upper bound should be 2*MSL, i.e., 240 seconds. Besides the retransmit timeout (RTO) value two further parameters R1 and R2 may be of interest. These don't seem to be tunable via any Solaris' offered interface that I know of. The value of R1 SHOULD correspond to at least 3 retransmissions, at the current RTO. The value of R2 SHOULD correspond to at least 100 seconds. [...] However, the values of R1 and R2 may be different for SYN and data segments. In particular, R2 for a SYN segment MUST be set large enough to provide retransmission of the segment for at least 3 minutes. The application can close the connection (i.e., give up on the open attempt) sooner, of course. Great many internet servers which are running Solaris do retransmit segments unnecessarily often. The current condition of European networks indicate that a connection to the US may take up to 2 seconds. All parameters mentioned in the first part of this section relate to each other! As a starter take this little example. Consider a picture, size 1440 byte, LZW compressed, which is to be transferred over a serial linkup with 14400 bps and using a MTU of 1500. In the ideal case only one PDU gets transmitted. The ACK segment can only be sent after the complete PDU is received. The transmission takes about 1 second. These values seem low, but they are meant as 'food for thought'. Now consider something going awry... Solaris 2.5.1 is behaving strange, if the initial SYN segment from the host doing the active open is lost. The initial SYN gets retransmitted only after a period of 4 * tcp_rexmit_interval_initial plus a constant C. The time is 12 seconds with the default settings. More information is being prepared on the retransmission test page. The initial lost SYN may or may not be of importance in your environment. For instance, if you are connected via ATM SVCs, the initial PDU might initiate a logical connection (ATM works point to point) in less than 0.3 seconds, but will still be lost in the process. It is rather annoying for a user of 2.5.1 to wait 12 seconds until something happens. tcp_rexmit_interval_initial default 500, since 2.5.1 3000, recommended >= 2000 (500 for special purposes) This interval is waited before the last data sent is retransmitted due to a missing acknowledgment. Mind that this interval is used only for the first retransmission. The more international your server is, the larger you should chose this interval. Special laboratory environments working in LAN-only environments might be better off with 500 ms or even less. If you are doing measurements involving TCP (which is almost always a bad idea), you should consider lowering this parameter. tcp_rexmit_interval_min default 200, recommended >= 1000 (200 for special purposes) After the initial retransmission further retransmissions will start after the tcp_rexmit_interval_min interval. BSD usually specifies 1500 milliseconds. This interval should be tuned to the value of tcp_rexmit_interval_initial, e.g. some value between 50 % up to 200 %. The parameter has no effect on retransmissions during an active open, see my accompanying document on retransmissions. The tcp_rexmit_interval_min doesn't display any influence on connection establishment with Solaris 2.5.1. It does with 2.6, though. The influence on regular data retransmissions, or FIN retransmissions I have yet to research. tcp_ip_abort_interval default 120000, since 2.5 480000, recommended 600000 This interval specifies how long retransmissions for a connection in the ESTABLISHED state should be tried before a RESET segment is sent. BSD systems default to 9 minutes. tcp_ip_abort_cinterval default 240000, since 2.5 180000, recommended ? This interval specifies how long retransmissions for a remote host are repeated until the RESET segment is sent. The difference to the tcp_ip_abort_interval parameter is that this connection is about to be established - it has not yet reached the state ESTABLISHED. This value is interesting considering SYN flood attacks on your server. Proxy server are doubly handicapped because of their Janus behavior (like a server towards the downstream cache, like a client towards the upstream server). According to Stevens this interval is connected to the active open, e.g. the connect(3N) call. But according to SunSolve the interval has an impetus on both directions. A remote client can refuse to acknowledge an opening connection up to this interval. After the interval a RESET is sent. The other way around works out, too. If the three-way handshake to open a connection is not finished within this interval, the RESET Segment will be sent. This can only happen, if the final ACK went astray, which is a difficult test case to simulate. To improve your SYN flood resistance, SUN suggests to use an interval as small as 10000 milliseconds. This value has only been tested for the "fast" networks of SUN. The more international your connection is, the slower it will be, and the more time you should grant in this interval. Proxy server should never lower this value (and should let Squid terminate the connection). Webservers are usually not affected, as they seldom actively open connections beyond the LAN. tcp_rexmit_interval_max default 60000, RFC 1122 recommends 240000 (2MSL), recommended 1...2 * tcp_close_wait_interval or tcp_time_wait_interval since 2.6: default 240000 All previously mentioned retransmissions related interval use an exponential backoff algorithm. The wait interval between two consecutive retransmissions for the same PDU is doubled starting with the minimum. The tcp_rexmit_interval_max interval specifies the maximum wait interval between two retransmissions. If changing this value, you should also give the abort interval an inspection. The maximum wait interval should only be reached shortly before the abort interval timer expires. Additionally, you should coordinate your interval with the value of tcp_close_wait_interval or tcp_time_wait_interval. tcp_deferred_ack_interval default 50, BSD 200, recommended 200 (regular), 50 (benchmarking), or 500 (WAN server) This parameter specifies the timeout before sending a delayed ACK. The value should not be increased above 500, as required by RFC 1122. This value is of great interest for interactive services. A small number will increase the "responsiveness" of a remote service (telnet, X11), while a larger value can decrease the number of segments exchanged. The parameter might also interest to HTTP servers which transmit small amounts of data after a very short retrieval time. With a heavy-duty servers or in laboratory banging environment, you might encounter service times answering a request which are well above 50 ms. An increase to 500 might lead to less PDUs transferred over the network, because TCP is able to merge the ACK with data. Increases beyond 500 should not be even considered. SUN claims that Solaris recognizes the initial data phase of a connection. An initial ACK (not SYN) is not delayed. As opposed to the simplis