Demonstrations of tcpretrans, the Linux eBPF/bcc version. This tool traces the kernel TCP retransmit function to show details of these retransmits. For example: # ./tcpretrans TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE 01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED 01:55:05 0 4 10.153.223.157:22 R> 69.53.245.40:34619 ESTABLISHED 01:55:17 0 4 10.153.223.157:22 R> 69.53.245.40:22957 ESTABLISHED [...] This output shows three TCP retransmits, the first two were for an IPv4 connection from 10.153.223.157 port 22 to 69.53.245.40 port 34619. The TCP state was "ESTABLISHED" at the time of the retransmit. The on-CPU PID at the time of the retransmit is printed, in this case 0 (the kernel, which will be the case most of the time). Retransmits are usually a sign of poor network health, and this tool is useful for their investigation. Unlike using tcpdump, this tool has very low overhead, as it only traces the retransmit function. It also prints additional kernel details: the state of the TCP session at the time of the retransmit. A -l option will include TCP tail loss probe attempts: # ./tcpretrans -l TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE 01:55:45 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED 01:55:46 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED 01:55:46 0 4 10.153.223.157:22 R> 69.53.245.40:51601 ESTABLISHED 01:55:53 0 4 10.153.223.157:22 L> 69.53.245.40:46444 ESTABLISHED 01:56:06 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 01:56:06 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 01:56:08 1938 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED 01:56:08 0 4 10.153.223.157:22 R> 69.53.245.40:46444 ESTABLISHED [...] See the "L>" in the "T>" column. These are attempts: the kernel probably sent a TLP, but in some cases it might not have been ultimately sent. To spot heavily retransmitting flows quickly one can use the -c flag. It will count occurring retransmits per flow. # ./tcpretrans.py -c Tracing retransmits ... Hit Ctrl-C to end ^C LADDR:LPORT RADDR:RPORT RETRANSMITS 192.168.10.50:60366 <-> 172.217.21.194:443 700 192.168.10.50:666 <-> 172.213.11.195:443 345 192.168.10.50:366 <-> 172.212.22.194:443 211 [...] This can ease to quickly isolate congested or otherwise awry network paths responsible for clamping tcp performance. TCP sequence numbers can be included via -s, except in count mode. These numbers are useful for identifying specific retransmissions in large packet caputes. Note, lossprobe -l output will display 0 for the sequence number for L type. # ./tcpretrans.py -s TIME PID IP LADDR:LPORT T> RADDR:RPORT STATE SEQ 18:03:46 0 4 192.168.10.50:41976 R> 172.217.21.194:443 SYN_SENT 2879306108 18:03:49 0 4 192.168.10.50:41976 R> 172.217.21.194:443 SYN_SENT 2879306108 USAGE message: # ./tcpretrans -h usage: tcpretrans.py [-h] [-s] [-l] [-c] [-4 | -6] Trace TCP retransmits optional arguments: -h, --help show this help message and exit -s, --sequence display TCP sequence numbers -l, --lossprobe include tail loss probe attempts -c, --count count occurred retransmits per flow -4, --ipv4 trace IPv4 family only -6, --ipv6 trace IPv6 family only examples: ./tcpretrans # trace TCP retransmits ./tcpretrans -l # include TLP attempts ./tcpretrans -4 # trace IPv4 family only ./tcpretrans -6 # trace IPv6 family only