Demonstrations of tcpdrop, the Linux BPF/bcc version.


tcpdrop prints details of TCP packets or segments that were dropped by the
kernel, including the kernel stack trace that led to the drop:

# ./tcpdrop.py
TIME     PID    IP SADDR:SPORT          > DADDR:DPORT          STATE (FLAGS)
20:49:06 0      4  10.32.119.56:443     > 10.66.65.252:22912   CLOSE (ACK)
	tcp_drop+0x1
	tcp_v4_do_rcv+0x135
	tcp_v4_rcv+0x9c7
	ip_local_deliver_finish+0x62
	ip_local_deliver+0x6f
	ip_rcv_finish+0x129
	ip_rcv+0x28f
	__netif_receive_skb_core+0x432
	__netif_receive_skb+0x18
	netif_receive_skb_internal+0x37
	napi_gro_receive+0xc5
	ena_clean_rx_irq+0x3c3
	ena_io_poll+0x33f
	net_rx_action+0x140
	__softirqentry_text_start+0xdf
	irq_exit+0xb6
	do_IRQ+0x82
	ret_from_intr+0x0
	native_safe_halt+0x6
	default_idle+0x20
	arch_cpu_idle+0x15
	default_idle_call+0x23
	do_idle+0x17f
	cpu_startup_entry+0x73
	rest_init+0xae
	start_kernel+0x4dc
	x86_64_start_reservations+0x24
	x86_64_start_kernel+0x74
	secondary_startup_64+0xa5

20:49:50 12431  4  127.0.0.1:8198       > 127.0.0.1:48280      CLOSE (RST|ACK)
	tcp_drop+0x1
	tcp_v4_do_rcv+0x135
	__release_sock+0x88
	release_sock+0x30
	inet_stream_connect+0x47
	SYSC_connect+0x9e
	sys_connect+0xe
	do_syscall_64+0x73
	entry_SYSCALL_64_after_hwframe+0x3d

[...]

The last two columns show the state of the TCP session, and the TCP flags.
These two examples show packets arriving for a session in the closed state,
that were dropped by the kernel.

This tool is useful for debugging high rates of drops, which can cause the
remote end to do timer-based retransmits, hurting performance.


USAGE:

# ./tcpdrop.py -h
usage: tcpdrop.py [4 | -6] [-h]

Trace TCP drops by the kernel

optional arguments:
  -4, --ipv4  trace IPv4 family only
  -6, --ipv6  trace IPv6 family only
  -h, --help  show this help message and exit

examples:
    ./tcpdrop           # trace kernel TCP drops
    ./tcpdrop -4        # trace IPv4 family only
    ./tcpdrop -6        # trace IPv6 family only