Networking 0x300 - TCP and Its Three-Way Handshake

Every time you open a website, send an email, or download a file, something happens before a single byte of actual data is transferred. Two machines — your computer and a server somewhere on the internet — go through a quick negotiation. They introduce themselves, agree on some ground rules, and only then start exchanging data.

This negotiation is the TCP three-way handshake, and it's one of the most fundamental processes in computer networking. If you've ever captured packets in Wireshark and seen those SYN, SYN-ACK, ACK packets at the start of every connection — that's what we're going to break down today.

But the handshake is just the front door. Behind it sits a genuinely remarkable piece of engineering: a protocol that takes an unreliable, packet-dropping, packet-reordering, wildly variable network and presents your application with the illusion of a clean, ordered, infinite pipe of bytes. This post is the long version. We'll cover the handshake in detail, then go through the machinery that keeps the illusion running — retransmission, flow control, congestion control — and finish with the places where that machinery can be attacked.

Let's dive in.

What is TCP?

TCP (Transmission Control Protocol) is one of the core protocols in the Internet Protocol Suite. It lives at the Transport Layer (Layer 4) of the OSI model, sitting between the application layer (HTTP, SMTP, FTP) and the network layer (IP).

The key word with TCP is reliable. Unlike its sibling UDP (User Datagram Protocol), which just fires packets and hopes for the best, TCP guarantees:

• Reliable delivery — Every byte you send will arrive at the other end, or you'll know about it
• Ordered transmission — Data arrives in the exact sequence it was sent
• Error detection — Corrupted packets are detected and retransmitted
• Flow control — The sender won't overwhelm the receiver with more data than it can handle
• Congestion control — The sender won't overwhelm the network either

This makes TCP the protocol of choice for applications where data integrity matters:

The Byte Stream Abstraction

Here's the idea that everything else in TCP follows from, and it's the one most tutorials skip.

TCP does not send packets. TCP sends a stream of bytes.

When your application calls write() with 4000 bytes, TCP does not promise to send one 4000-byte thing. It might send three segments. It might merge your 4000 bytes with the 200 bytes from your next write() and send them together. On the other end, the receiving application might get 1460 bytes from its first read(), then 2740 from the next.

This has a very practical consequence that bites people constantly: there are no message boundaries in TCP. If you're writing a network protocol, you must define your own framing — a length prefix, a delimiter, something. Every "my JSON parser sometimes gets half a message" bug traces back to somebody assuming that one write() equals one read(). It doesn't, and TCP never promised it would.

What TCP does promise is that the bytes come out in the order they went in, with none missing and none duplicated. Every mechanism in this article exists to keep that promise.

All of this reliability comes at a cost — overhead. Before TCP can deliver any data, it needs to set up a connection. And that setup process is the three-way handshake.

The TCP Header — What's Inside a TCP Packet?

Before we dive into the handshake, it helps to understand what a TCP segment actually looks like. Every segment carries a 20-byte fixed header, optionally followed by up to 40 bytes of options:

packet-beta
0-15: "Source Port"
16-31: "Destination Port"
32-63: "Sequence Number"
64-95: "Acknowledgment Number"
96-99: "Data Offset"
100-103: "Reserved"
104-111: "Flags"
112-127: "Window Size"
128-143: "Checksum"
144-159: "Urgent Pointer"
160-191: "Options (0 - 40 bytes), then Data"

Let's go field by field, because nearly every one of them shows up later in this article.

Source Port / Destination Port (16 bits each). Which application on each end. Combined with the source and destination IP from the IP header, these four values form the four-tuple that uniquely identifies a connection. Two connections can share three of the four; they can never share all four.

Sequence Number (32 bits). The position of this segment's first byte within the overall byte stream. This is the field that makes ordering and retransmission possible.

Acknowledgment Number (32 bits). "I have received every byte up to, but not including, this number, and this is what I expect next." Note the phrasing — TCP acknowledgments are cumulative. An ACK of 5000 means everything below 5000 arrived, full stop.

Data Offset (4 bits). The header length in 32-bit words. Minimum value 5 (a 20-byte header with no options), maximum 15 (60 bytes). That ceiling is why TCP options are limited to 40 bytes — a constraint that has shaped protocol evolution for decades and is a large part of why QUIC eventually got built on UDP instead.

Flags (8 bits). The control bits:

Window Size (16 bits). How many more bytes the sender of this segment is willing to receive. This is flow control, and we'll spend a whole section on it.

Checksum (16 bits). Covers the TCP header, the payload, and a "pseudo-header" containing the source IP, destination IP, protocol number, and TCP length. Pulling IP fields into the TCP checksum is a deliberate layering violation — it lets TCP detect a segment that was misdelivered to the wrong host. It's a weak 16-bit ones-complement sum, so it catches transmission noise, not tampering. Anyone who can modify packets can trivially fix the checksum. The checksum is an integrity check, not a security control.

Urgent Pointer (16 bits). Largely historical.

Options. Where the interesting negotiation happens — MSS, window scaling, SACK, timestamps. We'll cover each as it becomes relevant.

The Three-Way Handshake — Step by Step

The purpose of the handshake is to:

1. Synchronize sequence numbers — so both sides can track every byte
2. Confirm both sides are ready — the client wants to send, the server is willing to receive
3. Negotiate options — MSS, window scaling, SACK support, timestamps

That third point is underrated. The handshake isn't only a greeting; it's a capability negotiation, and several options can only be set during it. Get the handshake wrong and you're stuck with bad parameters for the life of the connection.

Let's walk through it with concrete numbers.

Step 1: SYN — Client to Server

The client sends a SYN packet with a random Initial Sequence Number (ISN) — let's say 1000.

The client enters the SYN_SENT state and waits.

Why a random ISN instead of starting at 0? Two reasons, and both matter.

The first is correctness. If a connection on the same four-tuple existed recently, delayed packets from it could still be wandering the network. A fresh random ISN makes it very unlikely that an old stray segment falls inside the new connection's valid sequence window.

The second is security. If ISNs were predictable, an off-path attacker could forge segments that land inside the window without ever seeing your traffic. This is not theoretical — it's the basis of the 1994 Mitnick attack, which we'll come back to. RFC 6528 specifies the modern approach: the ISN is a fine-grained timer plus a cryptographic hash of the four-tuple and a boot-time secret. That keeps ISNs unpredictable to outsiders while still increasing monotonically for any given connection pair.

Step 2: SYN-ACK — Server to Client

The server, if willing to accept, responds with SYN-ACK, doing two things at once:

1. Acknowledging the client's ISN with ACK = 1001 (client ISN + 1)
2. Sending its own ISN — say 5000

Why ISN + 1 when no data was sent? Because SYN consumes one sequence number. It has to: SYN itself must be reliably delivered, so it needs a number to be acknowledged by. FIN works the same way. This is why you always see the off-by-one.

The server enters SYN_RECEIVED.

Step 3: ACK — Client to Server

The client acknowledges the server's ISN. Both sides reach ESTABLISHED, and data can flow.

sequenceDiagram
    autonumber
    participant C as Client
    participant S as Server
    Note over C: CLOSED
    Note over S: LISTEN
    C->>S: SYN, seq=1000, MSS=1460, WS=7, SACK-OK
    Note over C: SYN_SENT
    S->>C: SYN-ACK, seq=5000, ack=1001, MSS=1460, WS=7, SACK-OK
    Note over S: SYN_RECEIVED
    C->>S: ACK, seq=1001, ack=5001
    Note over C,S: ESTABLISHED — data can flow
    C->>S: PSH-ACK, seq=1001, 500 bytes
    S->>C: ACK, ack=1501

Three packets, and note what rides along in them: MSS, WS (window scale), and SACK-OK. Those are the options being negotiated, and they only get one chance.

What Actually Gets Negotiated

MSS (Maximum Segment Size) — the largest payload each side is willing to receive in one segment. On standard Ethernet this is 1460 bytes: a 1500-byte MTU minus 20 bytes of IP header and 20 bytes of TCP header. MSS is announced independently by each side, and it is not a negotiation to a common value — each side simply respects what the other advertised. If the option is absent entirely, the default is a very conservative 536 bytes.

Window Scale (WS) — the Window Size field is only 16 bits, capping the receive window at 65535 bytes. On a fast, high-latency link that's crippling. A 1 Gbps path with 80 ms round-trip time has a bandwidth-delay product of about 10 MB, meaning you'd need roughly 10 MB in flight to keep the pipe full. Limited to 64 KB, you'd top out near 6 Mbps regardless of available bandwidth.

Window scaling (RFC 7323) fixes this with a shift count: the real window is the advertised value left-shifted by that many bits. A shift of 7 multiplies by 128, and the maximum shift of 14 allows windows up to 1 GB.

The critical detail: window scale is only exchanged in SYN packets. It cannot be enabled later. And it's only used if both sides offer it. This is why a middlebox that strips unknown TCP options can silently destroy throughput on long-haul links — the connection works fine, it's just mysteriously slow forever.

SACK Permitted — enables Selective Acknowledgment. More on this shortly.

Timestamps — two roles. They give a clean round-trip-time measurement on every segment, and they enable PAWS (Protection Against Wrapped Sequence numbers). The sequence space is 32 bits, about 4 GB; on a 10 Gbps link you can transmit 4 GB in a few seconds, so sequence numbers wrap while old packets could still be in flight. Timestamps let the receiver reject the stale ones.

The Accept Queue: Where Connections Actually Wait

There's an implementation detail here that matters enormously in production and is almost never mentioned in protocol tutorials. The kernel maintains two queues for a listening socket:

flowchart LR
    A["SYN arrives"] --> B["SYN queue<br/>half-open connections<br/>tcp_max_syn_backlog"]
    B -->|"final ACK arrives"| C["Accept queue<br/>fully established<br/>min of backlog and somaxconn"]
    C -->|"app calls accept()"| D["Your application"]
    B -.->|"queue full"| E["SYN dropped<br/>or SYN cookies engage"]
    C -.->|"queue full"| F["ACK dropped<br/>client thinks it is connected"]
    classDef bad fill:#c0392b,stroke:#922b21,color:#ffffff
    classDef good fill:#008987,stroke:#00605f,color:#ffffff
    class E,F bad
    class D good

The SYN queue holds connections in SYN_RECEIVED — the handshake started but isn't finished. The accept queue holds completed connections waiting for your application to call accept().

The failure mode on the right-hand side is nasty. If your application is slow to accept(), the accept queue fills, and the kernel drops the client's final ACK. The client has already transitioned to ESTABLISHED and cheerfully starts sending its request. The server never sees a connection at all. The client eventually times out or gets a reset. From the application's perspective nothing is wrong; from the user's perspective the site randomly hangs.

thilan@ubuntu:~$ ss -lnt
State   Recv-Q  Send-Q  Local Address:Port  Peer Address:Port
LISTEN  0       511           0.0.0.0:80         0.0.0.0:*

On a listening socket those columns are special: Recv-Q is the current accept queue depth and Send-Q is its maximum. A persistently non-zero Recv-Q means your application isn't accepting fast enough. Also worth checking:

thilan@ubuntu:~$ nstat -az TcpExtListenOverflows TcpExtListenDrops
TcpExtListenOverflows           0
TcpExtListenDrops               0

Any number climbing there is dropped connections, full stop.

Simultaneous Open

A footnote, but it explains a state you'll see in the state diagram. If both hosts send SYN to each other at the same time, there's no client and no server — both go SYN_SENT, both receive a SYN, both reply SYN-ACK, and both reach ESTABLISHED. It takes four packets and no three-way handshake ever occurs. It's rare in practice but it's why the state machine has a transition you'd otherwise never explain.

Why Sequence Numbers Matter

Sequence numbers are the backbone of TCP's reliability.

Tracking Data

Every byte in a TCP stream has a sequence number. If the client sends 500 bytes starting at 1001, those bytes are numbered 1001 through 1500. The receiver acknowledges 1501, meaning "I have everything below 1501, send that next."

Client sends:  SEQ=1001, 500 bytes of data
Server replies: ACK=1501  ("I got bytes 1001-1500, send 1501 next")

Client sends:  SEQ=1501, 300 bytes of data
Server replies: ACK=1801  ("I got bytes 1501-1800, send 1801 next")

The Cumulative ACK Problem

Because ACKs are cumulative, they can only describe a contiguous prefix of the stream. Consider four segments where the second is lost:

sequenceDiagram
    autonumber
    participant C as Sender
    participant S as Receiver
    C->>S: seq=1000, 1000 bytes
    S->>C: ack=2000
    C-XS: seq=2000, 1000 bytes (LOST)
    C->>S: seq=3000, 1000 bytes
    S->>C: ack=2000 (duplicate — still want 2000)
    C->>S: seq=4000, 1000 bytes
    S->>C: ack=2000 (duplicate #2)
    C->>S: seq=5000, 1000 bytes
    S->>C: ack=2000 (duplicate #3)
    Note over C: 3 duplicate ACKs → fast retransmit
    C->>S: seq=2000, 1000 bytes (RETRANSMIT)
    S->>C: ack=6000 (now everything is contiguous)

The receiver actually has bytes 3000-5999 sitting in its buffer, but with a plain cumulative ACK it has no way to say so. All it can repeat is "I still need 2000."

Those repeated ACKs are called duplicate ACKs, and TCP turns the problem into a signal. Three duplicate ACKs is strong evidence a single segment was lost while later ones got through — so the sender retransmits immediately rather than waiting for a timeout. This is fast retransmit, and it's the difference between recovering in one round-trip versus one full retransmission timeout.

SACK: Saying What You Actually Have

Selective Acknowledgment (RFC 2018) fixes the underlying limitation. With SACK enabled, the receiver appends blocks describing the non-contiguous ranges it holds:

ACK=2000, SACK=[3000-6000]
   "I still need byte 2000, but I already have 3000 through 5999.
    Don't bother resending those."

Without SACK, a sender that loses several segments in one window often has to retransmit everything from the first loss onward. With SACK, it retransmits exactly the missing pieces. On lossy links this is a large win, and it's why net.ipv4.tcp_sack should stay enabled.

Duplicate Detection and Wraparound

If a segment arrives with a sequence number already acknowledged, it's a duplicate and gets discarded. The subtlety is the 32-bit sequence space wrapping, which PAWS handles using timestamps, as described earlier.

Real-World Analogy

Imagine mailing chapters of a book to a friend, one envelope at a time:

• You number each envelope (sequence numbers)
• Your friend sends a postcard after each: "Got chapter 5, send chapter 6" (acknowledgments)
• If chapter 3 is lost, your friend keeps saying "still need chapter 3" (duplicate ACKs)
• With SACK, the postcard adds "but I do have 4 through 7 already, don't resend those"
• If chapter 4 arrives twice, the extra is discarded (duplicate detection)

That's TCP, at the byte level, billions of times faster.

Retransmission: How TCP Handles Loss

Fast retransmit only works when later segments arrive to trigger duplicate ACKs. If the tail of your data is lost, or the network drops everything, nothing comes back at all. That's what the retransmission timeout (RTO) is for.

Picking a Timeout

Setting the RTO is harder than it sounds. Too short and you flood the network with needless retransmissions, which is exactly the wrong response to congestion. Too long and every loss stalls the connection.

TCP measures the round-trip time continuously and maintains two smoothed values (RFC 6298): the smoothed RTT (SRTT) and the RTT variation (RTTVAR).

SRTT   = (1 - 1/8) * SRTT   + (1/8) * newRTT
RTTVAR = (1 - 1/4) * RTTVAR + (1/4) * |SRTT - newRTT|

RTO    = SRTT + 4 * RTTVAR

Including the variance is the clever part. On a stable link with consistent RTT, RTTVAR is near zero and the RTO sits just above the actual RTT, so losses are detected fast. On a jittery link, RTTVAR grows and the RTO backs off automatically, avoiding spurious retransmissions. TCP adapts to the path it's actually on.

Two important refinements:

Karn's algorithm — never sample RTT from a retransmitted segment. When an ACK arrives for a segment sent twice, you can't tell whether it's acknowledging the original or the retransmission, and guessing wrong corrupts your RTT estimate in whichever direction hurts most.

Exponential backoff — each consecutive timeout for the same segment doubles the RTO. 1s, 2s, 4s, 8s. If the network is genuinely broken, TCP backs off rather than adding to the problem. This is why a connection to a machine that has vanished takes so long to die: Linux retries net.ipv4.tcp_retries2 times (default 15), which works out to roughly 15 minutes of doubling.

Seeing Retransmissions

thilan@ubuntu:~$ nstat -az TcpRetransSegs TcpExtTCPLostRetransmit TcpExtTCPFastRetrans
TcpRetransSegs                  1843
TcpExtTCPLostRetransmit         12
TcpExtTCPFastRetrans            1602

Compare TcpRetransSegs against total segments sent for a retransmission rate. Anything above about 1% deserves investigation. Note that most of these were fast retransmits, which is the healthy case — a high ratio of timeout-driven retransmits instead would suggest the loss is bursty enough to take out whole windows.

In Wireshark, the filter is tcp.analysis.retransmission.

Flow Control: The Sliding Window

Reliability solves "did it arrive?" Flow control solves a different question: "can the receiver keep up?"

A fast server talking to a small embedded device can easily send data faster than the device can process it. Without flow control, the receiver's buffer overflows and data is lost — not because the network failed, but because the receiver did.

TCP's answer is the receive window (rwnd), advertised in the Window Size field of every segment. It says: "I have this many bytes of buffer space free right now." The sender may have at most rwnd unacknowledged bytes in flight.

block-beta
  columns 4
  a["1 - 1000<br/>sent + ACKed"] b["1001 - 3000<br/>sent, awaiting ACK"] c["3001 - 5000<br/>may send now"] d["5001+<br/>must wait"]
  classDef done fill:#e6f5f5,stroke:#008987,color:#2e2f3e
  classDef flight fill:#008987,stroke:#00605f,color:#ffffff
  classDef usable fill:#e6f5f5,stroke:#008987,color:#2e2f3e
  classDef blocked fill:#c0392b,stroke:#922b21,color:#ffffff
  class a done
  class b flight
  class c usable
  class d blocked

The window "slides" rightward as ACKs arrive. Bytes 1001-3000 are in flight; 3001-5000 can be sent immediately; anything beyond 5000 must wait for the window to advance. The window is the sender's permission slip, reissued continuously.

Zero Window and the Persist Timer

What if the receiver's buffer fills completely? It advertises a window of zero, and the sender must stop.

Now there's a deadlock risk. The sender is waiting for a window update. The receiver will send one when its application drains the buffer — but that update is a bare ACK, and ACKs are not retransmitted. If it's lost, both sides wait forever.

TCP breaks this with the persist timer. The sender periodically transmits a window probe — a segment carrying one byte — forcing the receiver to respond with its current window. If it's still zero, the sender backs off and probes again later.

thilan@ubuntu:~$ nstat -az TcpExtTCPZeroWindowDrop

Persistent zero windows almost always mean the receiving application isn't reading fast enough. The network is fine; the app is the bottleneck.

Silly Window Syndrome

An adjacent pathology: if the receiving application consumes one byte at a time, the receiver could advertise a one-byte window, prompting the sender to transmit a 41-byte packet (20 IP + 20 TCP + 1 data) to carry a single byte. Throughput collapses into overhead.

The fix is on both sides. Receivers refuse to advertise tiny windows, waiting until a worthwhile chunk is free. Senders wait until they can send a full MSS — which is Nagle's algorithm.

Nagle's Algorithm and Delayed ACKs

Two independent optimizations that interact badly. Worth knowing because the symptom is baffling if you haven't seen it.

Nagle's algorithm reduces small-packet overhead: if there's unacknowledged data outstanding, buffer small writes instead of sending immediately. Send when you have a full MSS, or when everything outstanding has been acknowledged. It was designed for telnet, where sending a 41-byte packet per keystroke was genuinely wasteful.

Delayed ACK reduces pure-ACK traffic: don't acknowledge immediately, wait up to 500 ms (Linux uses ~40 ms) in case you have data to piggyback on, or until a second full segment arrives.

Individually sensible. Together:

sequenceDiagram
    autonumber
    participant A as Sender (Nagle on)
    participant B as Receiver (delayed ACK)
    A->>B: small write, part 1 — sent immediately
    Note over A: part 2 is small and part 1 is unacked → Nagle buffers it
    Note over B: has no data to send back → delays the ACK
    Note over A,B: deadlock — each waits on the other
    B->>A: ACK (finally, after the delayed-ACK timer fires)
    A->>B: part 2 — released by Nagle

The sender won't send because Nagle is waiting for an ACK. The receiver won't ACK because delayed-ACK is waiting for data. Nothing moves until the delayed-ACK timer expires. The result is a request-response protocol that mysteriously takes ~40 ms per exchange instead of the sub-millisecond RTT the network supports.

The classic trigger is a write-write-read pattern: header in one write(), body in another, then wait for the response. The fix is either to combine the writes (writev() or a single buffer, which is better anyway) or to disable Nagle:

int flag = 1;
setsockopt(sockfd, IPPROTO_TCP, TCP_NODELAY, &flag, sizeof(flag));

TCP_NODELAY is set by default in most modern RPC frameworks and databases for exactly this reason. If you're writing a latency-sensitive protocol, set it — and fix your write pattern too.

Congestion Control: Protecting the Network

Flow control protects the receiver. Congestion control protects the network between you, and it's the reason the internet works at all.

The Congestion Collapse of 1986

In October 1986, the link between Lawrence Berkeley Laboratory and UC Berkeley — 400 yards apart — dropped from 32 kbps to 40 bps. A factor of a thousand.

The cause: senders had no idea the network was overloaded. Routers dropped packets, senders timed out and retransmitted, which added more traffic to an already-saturated network, causing more drops and more retransmissions. The network spent essentially all its capacity carrying duplicate packets. This is congestion collapse, and Van Jacobson's response — the congestion control algorithms still in use today — is arguably what saved the internet.

The Congestion Window

The sender maintains a second limit alongside the receive window: the congestion window (cwnd). Unlike rwnd, which the receiver tells you, cwnd is a guess. Nobody reports network capacity, so TCP has to infer it by experiment.

bytes in flight <= min(cwnd, rwnd)

The receiver's limit and the network's limit, whichever is smaller. The core assumption, which was reasonable in 1988 and is shakier on modern wireless: packet loss means congestion.

Slow Start and Congestion Avoidance

xychart-beta
    title "Congestion window over time"
    x-axis "Round trips" [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
    y-axis "cwnd (segments)" 0 --> 90
    line [10, 20, 40, 80, 40, 41, 42, 43, 44, 45, 23, 24, 25, 26, 27, 28]

Slow start (the steep part) — despite the name, this is exponential growth. cwnd starts at the initial window, 10 segments on modern Linux (RFC 6928), and doubles every round trip. The goal is to find the available capacity quickly rather than creeping up to it.

Congestion avoidance (the gentle slopes) — once cwnd passes a threshold called ssthresh, growth becomes linear: roughly one extra segment per round trip. TCP is now near capacity and probes carefully.

On loss (the cliffs) — ssthresh is set to half the current cwnd, and cwnd drops to that. This is AIMD: Additive Increase, Multiplicative Decrease. Grow slowly, back off sharply.

AIMD is why the graph is a sawtooth, and the sawtooth isn't a flaw — it's TCP continuously probing for more bandwidth and retreating when it overshoots. AIMD also has a property that makes the whole system work: competing flows sharing a bottleneck converge toward an equal share, regardless of when they started.

A loss detected by timeout is treated as much worse news than one detected by duplicate ACKs. Three dup ACKs means packets are still flowing, so the sender halves cwnd and continues (fast recovery). A timeout means nothing is getting through, so cwnd collapses all the way back to the initial window and slow start begins again.

The Algorithms You'll Actually Meet

Reno / NewReno — the classic described above. Still the reference point.

CUBIC — Linux's default since 2.6.19. Growth follows a cubic function of time since the last loss: it climbs aggressively toward the previous maximum, flattens near it (the probable capacity), then probes beyond. Being time-based rather than ACK-based makes it much fairer between connections with very different RTTs, where Reno badly favours short ones.

BBR — Google, 2016. A genuine departure: instead of treating loss as the congestion signal, BBR models the path's bottleneck bandwidth and minimum RTT directly, then paces packets to match. This matters because the loss-equals-congestion assumption has two modern failure modes. On wireless links, loss is often corruption, not congestion — and Reno needlessly halves its window. And with bufferbloat, oversized router buffers absorb the excess so loss doesn't occur until latency has already ballooned into seconds. BBR aims for high throughput and low latency by not filling those buffers in the first place.

thilan@ubuntu:~$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

thilan@ubuntu:~$ cat /proc/sys/net/ipv4/tcp_available_congestion_control
reno cubic bbr

You can inspect a live connection's congestion state:

thilan@ubuntu:~$ ss -ti dst 93.184.216.34
ESTAB  0  0   192.168.1.10:54321   93.184.216.34:443
     cubic wscale:7,7 rto:204 rtt:3.821/0.912 mss:1448
     cwnd:42 ssthresh:31 bytes_sent:184320 bytes_acked:184320
     segs_out:129 segs_in:64 send 12.7Mbps pacing_rate 15.2Mbps
     retrans:0/2 rcv_space:14480

Everything from this article is in that output: the algorithm (cubic), negotiated window scaling (wscale:7,7), the computed retransmission timeout (rto:204), smoothed RTT and variance (rtt:3.821/0.912), the negotiated segment size (mss:1448), and both windows (cwnd:42, ssthresh:31). ss -ti is the single most useful command for debugging TCP performance.

Connection Teardown — The Four-Way Handshake

Connections are established in three packets but torn down in four, because a TCP connection is really two independent byte streams and each direction closes separately.

sequenceDiagram
    autonumber
    participant C as Client
    participant S as Server
    Note over C,S: ESTABLISHED
    C->>S: FIN, seq=5000
    Note over C: FIN_WAIT_1
    S->>C: ACK, ack=5001
    Note over S: CLOSE_WAIT
    Note over C: FIN_WAIT_2
    Note over S: server may still send data here
    S->>C: FIN, seq=8000
    Note over S: LAST_ACK
    C->>S: ACK, ack=8001
    Note over C: TIME_WAIT (2 x MSL)
    Note over S: CLOSED
    Note over C: CLOSED

When the client sends FIN it's saying "I have no more data to send" — not "this conversation is over." The server acknowledges, may keep sending for as long as it likes, and sends its own FIN when finished. That intermediate condition is a half-closed connection, and it's a real, usable state: shutdown(fd, SHUT_WR) is how tools like nc signal end-of-input while still reading the response.

CLOSE_WAIT Is Always Your Bug

A practical note worth internalising. CLOSE_WAIT means the kernel received a FIN, acknowledged it, and is waiting for your application to call close(). The kernel cannot proceed on its own.

So piles of CLOSE_WAIT sockets are never a network problem and never a tuning problem — they are a file-descriptor leak in the application. Some code path isn't closing its sockets. There's no sysctl for this.

thilan@ubuntu:~$ ss -tan state close-wait

If that list grows and never drains, go find the leak.

TIME_WAIT: Why the Wait Is Necessary

The side that closes first ends in TIME_WAIT for twice the Maximum Segment Lifetime. Linux fixes this at 60 seconds regardless of tcp_fin_timeout (which governs FIN_WAIT_2). TIME_WAIT does two jobs:

Preventing old segments from corrupting a new connection. A delayed segment from this connection could otherwise arrive during a new connection reusing the same four-tuple and be accepted as valid data. Waiting 2×MSL guarantees every straggler has expired.

Ensuring the peer can close cleanly. If the final ACK is lost, the peer retransmits its FIN. Something must remain to answer. A socket in TIME_WAIT re-sends the ACK; a fully closed one replies RST, and the peer's connection dies with an error.

TIME_WAIT gets blamed for a lot, usually unfairly. On a busy reverse proxy making many short outbound connections you can genuinely exhaust the ephemeral port range, since each four-tuple is unavailable for 60 seconds:

thilan@ubuntu:~$ ss -tan state time-wait | wc -l
28451

The right fixes, in order: use connection pooling and keep-alive so you're not making so many connections; widen net.ipv4.ip_local_port_range; and if needed enable net.ipv4.tcp_tw_reuse, which safely reuses TIME_WAIT sockets for outbound connections using timestamps to reject old segments.

The wrong fix is net.ipv4.tcp_tw_recycle. It broke clients behind NAT — connections from different hosts sharing an IP have unrelated timestamp clocks, so legitimate SYNs got dropped. It was removed from Linux in 4.12. If you find it in a tuning guide, that guide is obsolete.

RST: The Abrupt Alternative

Not every connection closes politely. A RST tears it down immediately: no acknowledgment, no waiting, unsent buffered data discarded. You'll see one when connecting to a port with nothing listening (the basis of port scanning), when an application crashes and the kernel closes its sockets, when data arrives on an already-closed socket, or when a firewall is configured to REJECT rather than DROP.

ECONNRESET in application logs means the peer sent RST. It is not a network error; something on the other end actively refused or abandoned the connection.

TCP States — The Full Picture

stateDiagram-v2
    [*] --> CLOSED
    CLOSED --> LISTEN: passive open
    CLOSED --> SYN_SENT: active open, send SYN
    LISTEN --> SYN_RECEIVED: recv SYN, send SYN-ACK
    SYN_SENT --> SYN_RECEIVED: recv SYN, send SYN-ACK
    SYN_SENT --> ESTABLISHED: recv SYN-ACK, send ACK
    SYN_RECEIVED --> ESTABLISHED: recv ACK
    ESTABLISHED --> FIN_WAIT_1: close, send FIN
    ESTABLISHED --> CLOSE_WAIT: recv FIN, send ACK
    FIN_WAIT_1 --> FIN_WAIT_2: recv ACK
    FIN_WAIT_1 --> CLOSING: recv FIN, send ACK
    FIN_WAIT_2 --> TIME_WAIT: recv FIN, send ACK
    CLOSING --> TIME_WAIT: recv ACK
    CLOSE_WAIT --> LAST_ACK: app closes, send FIN
    LAST_ACK --> CLOSED: recv ACK
    TIME_WAIT --> CLOSED: 2 x MSL elapsed

thilan@ubuntu:~$ ss -tan | awk 'NR>1 {print $1}' | sort | uniq -c | sort -rn
   1247 ESTAB
    412 TIME-WAIT
     23 LISTEN
      8 CLOSE-WAIT

That one-liner is a good first move on any server that's behaving strangely.

MSS, MTU, and the Black Hole Problem

We touched on MSS during the handshake. It deserves a little more, because a specific failure mode here produces one of networking's most maddening symptoms.

MTU is the largest frame a link can carry — 1500 bytes on standard Ethernet. MSS is the largest TCP payload, normally MTU minus 40 bytes of IP and TCP headers, so 1460.

The problem arises when a path contains a link with a smaller MTU — a VPN or tunnel, typically, since encapsulation adds overhead. Path MTU Discovery handles this: IP packets are sent with the Don't Fragment bit set, and a router that can't forward one returns an ICMP Fragmentation Needed message reporting its MTU. The sender shrinks its segments accordingly.

This works right up until somebody blocks ICMP. A well-intentioned firewall rule dropping "all ICMP" kills PMTUD, and you get a PMTU black hole:

• The handshake works perfectly — SYN, SYN-ACK, ACK are all tiny
• Small requests work perfectly
• Anything large hangs forever

Big packets are silently discarded, the ICMP message that would explain it never arrives, and TCP retransmits the same oversized segment until it gives up. "SSH connects but hangs when I run ls in a big directory" is the canonical symptom.

The proper fix is to allow ICMP type 3 code 4. The common workaround is MSS clamping, where a router rewrites the MSS option in passing SYN packets to a value that fits:

iptables -t mangle -A FORWARD -p tcp --tcp-flags SYN,RST SYN \
    -j TCPMSS --clamp-mss-to-pmtu

This is standard on VPN gateways, and it's another reason the handshake matters: it's the only chance to set MSS.

Keepalives and Half-Open Connections

An idle TCP connection sends nothing. That's efficient, but it means if the peer is unplugged, crashes, or is silently dropped by a NAT device, your side has no idea. It sits in ESTABLISHED forever, connected to nothing. This is a half-open connection.

TCP keepalive probes an idle connection periodically. Linux defaults:

thilan@ubuntu:~$ sysctl net.ipv4.tcp_keepalive_time net.ipv4.tcp_keepalive_intvl net.ipv4.tcp_keepalive_probes
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9

Two hours idle before the first probe, then nine probes 75 seconds apart. Note that keepalive is off by default and must be enabled per-socket with SO_KEEPALIVE — and the two-hour default is far too long for most purposes. Stateful firewalls and NAT devices commonly drop idle mappings after 5 to 30 minutes, so a connection can be dead for an hour and a half before TCP investigates. Set the per-socket options (TCP_KEEPIDLE, TCP_KEEPINTVL, TCP_KEEPCNT) to something matched to your environment, or implement heartbeats at the application layer, which is usually better because it also proves the application is alive rather than just the kernel.

Seeing It in Action — Wireshark and tcpdump

Theory is great, but nothing beats watching it happen.

Using tcpdump

thilan@ubuntu:~$ sudo tcpdump -i eth0 -n 'tcp port 443' -c 3

Then in another terminal:

thilan@ubuntu:~$ curl https://example.com

You'll see:

10:32:01.123456 IP 192.168.1.10.54321 > 93.184.216.34.443: Flags [S], seq 1847291356, win 64240, options [mss 1460,sackOK,TS val int 3921 ecr 0,nop,wscale 7], length 0
10:32:01.145678 IP 93.184.216.34.443 > 192.168.1.10.54321: Flags [S.], seq 2981473625, ack 1847291357, win 65535, options [mss 1460,sackOK,TS val 8812 ecr 3921,nop,wscale 7], length 0
10:32:01.145789 IP 192.168.1.10.54321 > 93.184.216.34.443: Flags [.], ack 2981473626, win 502, length 0

The three-way handshake in three lines:

1. [S] — SYN from client, offering mss 1460, sackOK, wscale 7
2. [S.] — SYN-ACK, matching those options
3. [.] — bare ACK

Two things worth noticing. The options are visible right there in the first two packets and absent from the third — that's the one-shot negotiation we discussed. And look at the window on line 3: win 502. That's not a shrunken window, it's the scaled representation. With wscale 7, the real window is 502 × 128 = 64256 bytes. tcpdump can only apply the scale factor if it captured the handshake; start a capture mid-connection and the window numbers will be nonsense.

Useful flag notation: [S] SYN, [S.] SYN-ACK, [.] ACK, [P.] PSH-ACK, [F.] FIN-ACK, [R] RST.

To watch only connection setup and teardown across the box:

sudo tcpdump -i any 'tcp[tcpflags] & (tcp-syn|tcp-fin|tcp-rst) != 0'

Using Wireshark

Start a capture and try these filters:

tcp.flags.syn == 1 && tcp.flags.ack == 0     # connection attempts only
tcp.analysis.retransmission                   # retransmissions
tcp.analysis.zero_window                      # receiver out of buffer
tcp.analysis.duplicate_ack                    # loss signals
tcp.flags.reset == 1                          # resets

Right-click any packet and choose Follow → TCP Stream to see the whole conversation reassembled. Statistics → TCP Stream Graphs → Time Sequence (tcptrace) plots sequence numbers over time and makes the sawtooth, retransmissions, and window limits visible at a glance.

In a Security Context

Understanding the handshake isn't academic — nearly every property we've described has been attacked.

SYN Flood

The oldest handshake attack. The attacker sends a flood of SYNs with spoofed source addresses. The server allocates a SYN-queue entry for each, sends SYN-ACK, and waits for an ACK that will never come — the spoofed hosts never initiated anything.

sequenceDiagram
    autonumber
    participant A as Attacker
    participant S as Server
    participant V as Spoofed IPs
    A->>S: SYN, src=1.2.3.4 (spoofed)
    A->>S: SYN, src=5.6.7.8 (spoofed)
    A->>S: SYN, src=9.10.11.12 (spoofed)
    Note over A,S: ...thousands more...
    S->>V: SYN-ACK to 1.2.3.4
    S->>V: SYN-ACK to 5.6.7.8
    S->>V: SYN-ACK to 9.10.11.12
    Note over S: SYN queue fills with half-open connections
    Note over V: no ACK ever returns
    Note over S: legitimate SYNs are now dropped

The elegance of the attack, from the attacker's side, is its asymmetry: sending a SYN is nearly free, while the server must allocate and hold state.

SYN cookies are the standard defense, and the mechanism is genuinely clever. When the SYN queue fills, the server stops allocating state entirely. Instead it encodes the connection details — a coarse timestamp, the negotiated MSS, and a cryptographic hash of the four-tuple plus a server secret — into the ISN of its SYN-ACK.

Then it forgets the connection completely.

If a legitimate ACK arrives, it carries that ISN + 1. The server recomputes the hash, confirms it issued that number, and reconstructs the connection from scratch. A flood costs the server no memory at all, because state is offloaded into a value the client is obliged to echo back.

thilan@ubuntu:~$ sysctl net.ipv4.tcp_syncookies
net.ipv4.tcp_syncookies = 1

The trade-off: cookies have limited room to encode options, so connections established this way may lose window scaling, SACK, or timestamps (Linux recovers some of this using the timestamp field when available). Cookies engage only under pressure, so normal connections are unaffected.

Complementary defenses are rate limiting, sizing tcp_max_syn_backlog appropriately, and upstream filtering.

Sequence Prediction and the Mitnick Attack

If ISNs are predictable, an attacker can forge an entire connection blind — without ever seeing a packet from the victim.

This is what Kevin Mitnick did to Tsutomu Shimomura's machines on Christmas Day 1994, and it's a clean illustration of why ISN randomization matters:

1. Probe the target to learn its ISN generation pattern (early BSD simply incremented by a fixed amount each second)
2. SYN-flood the trusted host the target relies on, so it can't respond
3. Send a SYN to the target spoofed as the trusted host
4. The target's SYN-ACK goes to the silenced trusted host, and the attacker never sees it
5. Predict the ISN, send a correctly-numbered ACK, and the connection is established
6. Issue commands that arrive with the trusted host's IP address

The attack depended on predictable ISNs and on IP-address-based trust. RFC 6528 randomization killed the first. The second was always a bad idea.

RST Injection

If an attacker knows the four-tuple and can land a sequence number inside the receive window, a forged RST kills the connection. On-path attackers get this for free. Off-path attackers must guess — and a large receive window makes guessing dramatically easier, since any value in the window will do.

RFC 5961 hardened this with challenge ACKs: an in-window RST that isn't an exact sequence match doesn't tear down the connection. Instead the receiver sends a challenge ACK, and only a correctly-numbered response counts.

There's a lovely irony here. Linux implemented the challenge-ACK rate limit as a global counter shared across all connections, which turned it into a side channel — an off-path attacker could infer whether their guess landed by observing whether their own connection's challenge ACKs were consumed. That became CVE-2016-5696, an off-path TCP hijacking attack made possible by a mitigation. The fix was to randomize the limit.

RST injection is also deployed at scale for censorship: several national firewalls forge RSTs to both endpoints when a connection matches a filter, so it looks like a network problem rather than a block.

Port Scanning

The handshake is what makes port scanning possible. A SYN scan sends a SYN and interprets the reply:

• SYN-ACK → the port is open
• RST → the port is closed
• nothing → filtered, likely a firewall dropping packets

thilan@macbook:~$ sudo nmap -sS -p 22,80,443 target.com

PORT    STATE    SERVICE
22/tcp  open     ssh
80/tcp  open     http
443/tcp open     https

The scanner never sends the final ACK, so the connection never completes and never reaches the application — meaning the target's application logs stay empty. That's why SYN scans are called half-open, and it's the whole reason they're stealthier than full connect scans. The mechanics, along with building a scanner from scratch, are covered in port scanning from scratch.

Other Things Worth Knowing

Idle scanning (nmap -sI) abuses predictable IP fragmentation IDs on a third-party "zombie" host to scan a target without ever sending it a packet from your own address.

OS fingerprinting works because RFC 793 leaves plenty undefined. Initial window size, default TTL, option ordering, and responses to malformed flag combinations all vary by stack — enough to identify the OS from a handful of packets. It's how nmap -O works.

Connection exhaustion attacks like Slowloris don't flood at all. They open many legitimate connections and keep them barely alive, exhausting connection slots rather than bandwidth. TCP-level defenses don't help; you need per-IP connection limits and aggressive timeouts.

TCP vs UDP — And What Comes After

Head-of-Line Blocking

TCP's ordering guarantee has a cost that becomes serious when you multiplex.

Because TCP delivers a strictly ordered stream, one lost segment blocks delivery of everything behind it — even data that already arrived safely and is sitting in the receive buffer. The application can't have it, because handing it over would break the ordering promise.

For a single file transfer that's fine. For HTTP/2, which multiplexes many independent requests over one TCP connection, it's a real problem: a single lost packet belonging to one image stalls delivery of every other concurrent response. HTTP/2 solved application-layer head-of-line blocking and then ran straight into the transport-layer version.

QUIC and HTTP/3

This is the reason QUIC exists, and it explains several choices that look strange otherwise. QUIC runs over UDP and reimplements reliability, ordering, flow control, and congestion control — but per-stream, so a loss affecting one stream doesn't block the others.

Building on UDP wasn't a rejection of TCP's design so much as an escape from its deployment constraints:

• Ossification. Middleboxes worldwide inspect and rewrite TCP headers. New TCP options are frequently stripped or mangled, and the 40-byte option ceiling leaves little room anyway. TCP Fast Open, which allows data in the SYN, took a decade to see meaningful deployment for exactly this reason.
• Kernel coupling. TCP lives in the kernel, so changes require OS updates across billions of devices. QUIC lives in userspace and ships with the browser.
• Handshake latency. TCP's handshake plus TLS's handshake costs 2-3 round trips before any data moves. QUIC merges them into one, with zero round trips on resumption.

QUIC is HTTP/3's transport, and a large fraction of traffic to major sites now uses it. TCP isn't going anywhere — but for the first time in decades, it has a serious competitor, and the reasons for that competitor are exactly the constraints we've walked through here.

A Practical Tuning Reference

Worth knowing what these do, and worth resisting the urge to change them without measuring first. Defaults on modern Linux are sensible, and most "TCP tuning guides" on the internet are copied from 2005.

# Connection queues — raise if you see ListenOverflows
net.core.somaxconn = 4096
net.ipv4.tcp_max_syn_backlog = 8192

# SYN flood protection — leave this on
net.ipv4.tcp_syncookies = 1

# Socket buffers: min, default, max. Autotuned between min and max.
net.ipv4.tcp_rmem = 4096 131072 6291456
net.ipv4.tcp_wmem = 4096 16384 4194304

# Features negotiated in the handshake — leave these on
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_timestamps = 1

# Ephemeral ports — widen if you exhaust them making outbound connections
net.ipv4.ip_local_port_range = 32768 60999

# Safe for outbound connections. Do NOT use tcp_tw_recycle (removed in 4.12).
net.ipv4.tcp_tw_reuse = 1

# Congestion control
net.ipv4.tcp_congestion_control = cubic

The diagnostic commands worth committing to memory:

ss -ti                    # live per-connection TCP internals: cwnd, rtt, retrans
ss -lnt                   # listening sockets with accept-queue depth
ss -tan state close-wait  # find socket leaks in your application
nstat -az | grep -i tcp   # cumulative TCP counters since boot

Final Thoughts

The three-way handshake might look like a small detail, but it's the entrance to one of the most consequential pieces of engineering ever deployed. Three packets establish a connection — and then a whole hidden apparatus takes over: sequence numbers tracking every byte, an RTO adapting to the path's measured jitter, a sliding window pacing the receiver, a congestion window probing the network's capacity and retreating when it overshoots.

What makes TCP remarkable isn't any single mechanism. It's that they compose into something that works across a 10 Gbps datacenter fabric and a lossy mobile link, on hardware separated by decades, without any central coordination. AIMD in particular is doing something quietly extraordinary — billions of independent flows, each greedily probing for bandwidth, converging on a roughly fair share of every bottleneck on Earth, with no negotiation between them at all.

It's also worth remembering that every property here was designed for a smaller and more trusting network. Predictable sequence numbers were fine until they weren't. Loss-as-congestion was correct until wireless and bufferbloat. The 40-byte option limit was generous until it ossified the protocol. Understanding why each mechanism exists — and what it assumed — is what lets you recognise both the attacks and the failure modes.

Next time you fire up Wireshark, don't just filter for SYN. Pull up a stream graph and watch the congestion window sawtooth its way through a download. There's something genuinely satisfying about seeing an algorithm from 1988 negotiating with the internet in real time.

If you want to see the handshake used offensively, the natural next step is port scanning from scratch, which builds a scanner out of exactly the SYN/RST behaviour described here.

Thanks for reading!