Skip to content
Talk — 30 min
Menu

Networking8 min read

TCP Deep Dive

Reliable, ordered, connection-oriented transport

On this page · 8 sections

TCP Segment Header

Every TCP segment carries a 20-byte minimum header (up to 60 bytes with options). This header is the control plane of the entire protocol — sequence tracking, acknowledgments, flow control, and connection management all live here.

Source Port 16 bits
Destination Port 16 bits
Sequence Number 32 bits
Acknowledgment Number 32 bits
Data Offset 4 bits
Reserved 4 bits
Flags: CWR ECE URG ACK PSH RST SYN FIN 8 bits
Window Size 16 bits
Checksum 16 bits
Urgent Pointer 16 bits
Options + Padding variable (0-320 bits)

Header Field Roles

FieldPurpose
Sequence NumberByte offset of the first data byte in this segment within the stream
Acknowledgment NumberNext byte the sender expects to receive (cumulative ACK)
Data OffsetHeader length in 32-bit words (min 5 = 20 bytes, max 15 = 60 bytes)
Window SizeReceive buffer space available (flow control)
ChecksumCovers header + data + pseudo-header (source/dest IP, protocol, length)
Urgent PointerOffset to end of urgent data (rarely used, mostly legacy)

TCP Flags Breakdown

FlagMeaningWhen Used
SYNSynchronize sequence numbersConnection establishment
ACKAcknowledgment field is validAlmost every segment after SYN
FINNo more data from senderConnection teardown
RSTReset the connectionAbort, rejected connection, half-open detection
PSHPush data to application immediatelyInteractive traffic (SSH keystrokes)
URGUrgent pointer field is validOut-of-band data (rarely used today)
ECEECN-EchoExplicit Congestion Notification
CWRCongestion Window ReducedSender acknowledges ECE

3-Way Handshake

TCP connections are full-duplex. The 3-way handshake synchronizes sequence numbers in both directions and exchanges supported options (MSS, window scaling, SACK, timestamps).

Client Server
SYN   seq=1000
SYN+ACK   seq=5000, ack=1001
ACK   seq=1001, ack=5001

What Happens at Each Step

  1. SYN: Client picks a random Initial Sequence Number (ISN = 1000) and sends it. The SYN itself consumes one sequence number.
  2. SYN+ACK: Server picks its own ISN (5000), acknowledges the client's ISN+1 (ack=1001), and sends both flags in one segment.
  3. ACK: Client acknowledges the server's ISN+1 (ack=5001). Connection is ESTABLISHED. Data can now flow in both directions.

Note

Why random ISNs? Predictable ISNs allow TCP sequence prediction attacks (blind spoofing). Modern kernels use cryptographically random ISNs (RFC 6528).

Sequence & Acknowledgment Numbers

A critical distinction: TCP sequence numbers count bytes, not packets. TCP is a byte-stream protocol — the receiver reconstructs an ordered byte stream, not a sequence of messages.

Byte-Stream Semantics

  • If seq=1001 and the segment carries 500 bytes of data, the bytes span positions 1001-1500
  • The receiver's ACK will be ack=1501 — "I have all bytes up to 1500, send me byte 1501 next"
  • ACKs are cumulative: ack=3000 means "I have everything up to byte 2999"
  • Out-of-order segments are buffered by the receiver and ACKed only when the gap is filled (unless SACK is in use)
Sender Receiver
seq=1001, len=500 (bytes 1001-1500)
seq=1501, len=500 (bytes 1501-2000)
ack=2001 (got everything up to 2000)

Tip

Wireshark note: Wireshark shows "relative sequence numbers" by default (starting from 0). The actual ISN is random and 32-bit. Toggle this in Edit > Preferences > Protocols > TCP.

Flow Control: Sliding Window

Flow control prevents a fast sender from overwhelming a slow receiver. The receiver advertises how much buffer space it has available — the receive window (rwnd) — in every ACK segment's Window Size field.

Sliding Window Mechanism

  • The sender can have at most rwnd bytes of unacknowledged data in flight
  • As the receiver processes data and frees buffer space, it increases rwnd in subsequent ACKs
  • If rwnd=0, the sender must stop and periodically send window probes (1-byte segments) to check if space has opened up
  • The Window Size field is 16 bits = max 65,535 bytes. On modern networks, this is far too small

Window Scaling Option (RFC 7323)

Negotiated during the handshake via TCP options. The scale factor is a power of 2 (0-14), allowing windows up to 65,535 x 2^14 = ~1 GB.

console
$ ss -i dst 10.0.0.5
ESTAB  0  0  10.0.0.1:44832  10.0.0.5:443
  cubic wscale:7,7 rto:204 rtt:1.5/0.5 ato:40 mss:1448
  rcvmss:1448 advmss:1448 cwnd:10 ssthresh:7
  bytes_sent:1024 bytes_acked:1025 bytes_received:8192
  send 77.2Mbps rcv_space:14480 rcv_ssthresh:64088

Note

wscale:7,7 means both sides use a scale factor of 7 (multiply the Window Size field by 128). Effective max window = 65,535 x 128 = ~8 MB.

Congestion Control

Flow control protects the receiver. Congestion control protects the network. The sender maintains a congestion window (cwnd) that limits how much unacknowledged data can be in flight. The effective window is min(cwnd, rwnd).

Classic Congestion Control (Reno/NewReno)

The following is the classic loss-based model, not a universal description of Linux TCP: the active congestion-control algorithm is configurable.

  1. Slow Start

    cwnd doubles every RTT (exponential growth)

  2. Congestion Avoidance

    cwnd += 1 MSS per RTT (linear growth)

  3. Loss Detected

    Timeout or 3 duplicate ACKs

  4. Recovery

    Adjust cwnd, retransmit

Phases in Detail

PhaseTriggercwnd Behaviorssthresh Change
Slow Start Connection start, or timeout recovery cwnd starts at the initial window and grows approximately exponentially while ACKs arrive N/A (continues until cwnd >= ssthresh)
Congestion Avoidance cwnd >= ssthresh cwnd grows by ~1 MSS per RTT (additive increase) N/A
Fast Retransmit 3 duplicate ACKs received Retransmit the lost segment immediately (don't wait for timeout) ssthresh is reduced based on the data in flight
Fast Recovery After fast retransmit cwnd = ssthresh + 3 MSS, then congestion avoidance (skip slow start!) Already set by fast retransmit
Timeout Retransmission timer expires cwnd is reduced to the loss window (1 MSS in RFC 5681), then slow start resumes ssthresh is reduced based on the data in flight

Warning

Why fast recovery matters: In classic Reno, duplicate ACKs can signal loss before a retransmission timeout. Fast retransmit repairs the apparent gap; fast recovery keeps the ACK clock running with a reduced window. A timeout instead restarts with a much smaller window. Modern loss recovery and congestion-control algorithms can behave differently.

Modern Congestion Control Algorithms

CUBIC (Linux Default)
  • Loss-based — reacts to packet loss
  • Uses a cubic function for window growth (not linear)
  • After loss: cwnd drops, then rapidly recovers to the pre-loss level, then grows cautiously beyond
  • Better than Reno for high-BDP (bandwidth-delay product) networks
  • Default on Linux since kernel 2.6.19 (2006)
  • Problem: fills buffers before detecting loss (bufferbloat)
BBR (Google)
  • Model-based — estimates bottleneck bandwidth and RTT
  • Tries to send at the rate the bottleneck can handle, no more
  • Does NOT use loss as the primary signal
  • Significantly better on lossy links (wireless, intercontinental)
  • Available on Linux since kernel 4.9 (2016)
  • BBRv2 addresses fairness issues with CUBIC coexistence
console
# Check current congestion control algorithm
$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

# Switch to BBR (requires kernel >= 4.9)
$ sysctl -w net.ipv4.tcp_congestion_control=bbr
$ sysctl -w net.core.default_qdisc=fq

Tip

DevOps tip: If you run services over high-latency links (cross-region, CDN origins), switching to BBR can dramatically improve throughput. Many cloud providers (GCP, some AWS instances) already default to BBR.

Nagle's Algorithm

Nagle's algorithm (RFC 896) coalesces small writes into larger segments to reduce the overhead of tiny packets on the network. The rule is simple:

Nagle's Rule

If there is unacknowledged data in flight AND the new data is smaller than MSS, buffer it — wait for the ACK (or enough data to fill an MSS) before sending.

If there is no unacknowledged data in flight, send immediately regardless of size.

Nagle Enabled (default)
  • Reduces small-packet overhead (fewer headers per payload byte)
  • Good for bulk transfers, general-purpose traffic
  • Reduces network congestion from "tinygram" floods
Nagle Disabled (TCP_NODELAY)
  • Every write() sends immediately
  • Essential for latency-sensitive apps: SSH, gaming, real-time APIs
  • Required when application does its own buffering (e.g., HTTP/2 framing)

Delayed ACKs

Rather than ACKing every segment immediately, a receiver can delay an ACK to reduce pure-ACK traffic or piggyback it on a response. RFC 1122 requires the delay to stay below 500 ms; Linux defines a minimum timer of about 40 ms and a maximum of about 200 ms, not one fixed default delay.

Warning

Nagle and delayed ACKs: If a sender makes successive small writes while earlier data is unacknowledged, Nagle may hold the next segment. A receiver that delays its ACK can then add latency until that ACK is sent. The effect depends on the write pattern and TCP implementation; it is not a delay on every small write.

Solutions

  • TCP_NODELAY on the sender — disables Nagle, send immediately. Most common fix.
  • TCP_QUICKACK on the receiver (Linux-specific) — disables delayed ACKs temporarily. Less common.
  • Application-level buffering: batch small writes into a single write() call, then let it send as one segment.

Tip

When debugging small-write latency: check the application's write pattern and whether TCP_NODELAY is set before changing socket behavior.

TCP Options

TCP options are negotiated during the handshake (in SYN and SYN-ACK segments) and carried in the variable-length Options field of the header.

OptionSizePurposeNegotiated In
MSS (Maximum Segment Size) 4 bytes Largest payload each side can receive (typically 1460 bytes for Ethernet: 1500 MTU - 20 IP - 20 TCP) SYN, SYN-ACK
Window Scale 3 bytes Multiplier for the 16-bit Window Size field (scale factor 0-14, allows windows up to ~1 GB) SYN, SYN-ACK
SACK Permitted 2 bytes Enables Selective Acknowledgment — receiver can report non-contiguous blocks it has received SYN, SYN-ACK
SACK Blocks Variable Actual SACK data: up to 4 block ranges of received-but-not-yet-acked bytes Any segment
Timestamps 10 bytes TSval/TSecr: enables precise RTT measurement and PAWS (Protection Against Wrapped Sequence numbers) SYN, SYN-ACK (then every segment)

Note

SACK in practice: Cumulative ACKs alone do not describe out-of-order blocks held by the receiver. SACK reports some of those blocks so the sender can focus retransmission on missing data. Without SACK, recovery can cause unnecessary retransmissions, especially after multiple losses.

console
# Check SACK and timestamps settings on Linux
$ sysctl net.ipv4.tcp_sack
net.ipv4.tcp_sack = 1
$ sysctl net.ipv4.tcp_timestamps
net.ipv4.tcp_timestamps = 1
$ sysctl net.ipv4.tcp_window_scaling
net.ipv4.tcp_window_scaling = 1

Tip

Troubleshooting tip: If you see poor TCP performance through a middlebox (load balancer, firewall), check whether it strips TCP options. Some older devices strip timestamps or window scaling, silently crippling performance.

References

Solidnines — solidnines.com