Published on

tcpdump Troubleshooting

Authors
  • Name
    Jackson Chen

tcpdump

tpcdump is a packet analyzer that provides good troubleshooting for netowrk traffic issues

https://www.tcpdump.org/manpages/tcpdump.1.html

tcpdump cheat sheet

https://www.comparitech.com/net-admin/tcpdump-cheat-sheet/

https://gist.github.com/jforge/27962c52223ea9b8003b22b8189d93fb

# Some useful commands
switch      syntax              description
-----------------------------------------------------------------------------------------------
-i any      tcpdump -i any      # Capture from all interfaces
-D          tcpdump -D          # Show all available interfaces
-i eth0     tcpdump -i eth0     # Capture from specific interface ( Ex Eth0)

-nn         tcpdum  -nn -i 0.0  # disable name and service port resolution

tcp         tcpdump -i eth0 tcp # Capture TCP packets only
port        tcpdump -i eth0 port 80 # capture defind port
host        tcpdump host 192.168.1.10   # capture packet from specific host
net         tcpdump net 10.1.1.0/24     # capture packet from network subnet
src         tcpdump src 10.1.1.10       # capture from specific source address
dst         tcpdump dst 10.1.1.20       # capture from specific destination address
portrand    tcpdump portrange 21-123    # filter based on port range

-S          tcpdump -S http             # display entire pcacket
-p          tcpdump -p -i eth0          # Not capturing in promiscuous mode

-w          tcpdump -i any -w tcpdump.txt  # Save capture to a file 

# Logical Operators
Operator    Syntax      Example                                         Description
-----------------------------------------------------------------------------------------------
AND         and, &&     tcpdump -n src 192.168.1.2 and dst port 21      
OR          or, ||      tcmdump dst 10.1.1.10 || dst 10.1.1.20
EXCEPT      not, !      tcpdump dst 10.1.1.10 and not icmp 
LESS        <           tcpdump <32   # show packet size less than 32
GREATER     >           tcpdump >=32  

# Protocol
Ether, fddi, icmp ,ip, ip6 , ppp, radio, rarp, slip, tcp , udp, wlan

Display tcpdump result for real time network analysis

tcpdump -nnpi any net 10.1.1.0/24 and port 80
tcpdump -nnpi any net 10.1.1.0/24 && (port 8530 || port 8531)   # filter for WSUS packets for specific netowrk
tcpdump -nnpi 0.0 net 10.1.1.0/24       # filter any packet for specific network

F5 tcpdump

Overview of packet tracing with the tcpdump utility

https://www.tcpdump.org/manpages/tcpdump.1.html

https://support.f5.com/csp/article/K411

# Selecting an Interface or VLAN
The tcpdump utility's interface or -i option accepts only one option. This option may be a numbered interface or a named VLAN.

tcpdump -i <option>     # To view traffic, use the -i flag as follows
tcpdump -i 2.1          #To view the traffic on a single specific interface
tcpdump -i internal     # To view the traffic on a specific VLAN called internal
tcpdump -i eth0         # To view the traffic on the management interface

tcpdump -i 0.0          # To view the traffic on all TMM interfaces
    Note: This does not capture traffic on the management interface

# Disabling name resolution
By default, tcpdump attempts to look up IP addresses and use names, rather than numbers, in the output.
tcpdump -n
tcpdump -ni internal
tcpdump -nni internal   # no name resolution for host names and port numbers

#### Filter on a tcp flag
# To view all packets that are traveling through the BIG-IP system that contain the SYN flag
tcpdump 'tcp[tcpflags] & (tcp-syn) != 0'
# To view all packets that are traveling through the BIG-IP system that contain the RST flag
tcpdump 'tcp[tcpflags] & (tcp-rst) != 0'    

# Capturing packet data
The tcpdump utility provides an option that allows you to specify the amount of each packet to capture.
You can use the -s (snarf/snaplen) option to specify the amount of each packet to capture.
To capture the entire packet, use a value of 0 (zero).

# To capture the entire packet, use a value of 0 (zero)
tcpdump -s0 src host 172.16.101.20 and dst port 80  

# Saving tcpdump output to a file
You can save the tcpdump data to one of the following file formats:
a. A binary file that contains all the information collected by the tcpdump and is readable by the tcpdump utility,
as well as many other traffic analysis packages.
b. A text file that contains a subset of the full tcpdump data but is readable only as plain text.

tcpdump -w <filename>   # save the tcpdump output to a binary file

F5 tcpdump and Wireshark

https://support.f5.com/csp/article/K00463541

The BIG-IP is a 'full' proxy. This means there are 2 separate and independent connections that are managed by the BIG-IP. We refer to those as the Clientside (incoming traffic) and Serverside (outgoing traffic). Whether the traffic originates on your external or internal side is irrelevant to a BIG-IP. It is where the connection originates and hits that virtual server (clientside) and exits the BIG-IP (serverside).

In most cases to track an issue or resolve a question you need the traffic for both connections and then be able to align that traffic. To do that there are 2 things that will help.

a. One is referred to as the 'p' flag. This will instruct the BIG-IP to catch the flow on both sides of the BIG-IP.
b. The other is to drop the capture into Wireshark and look at the F5 Ethernet headers. To gather those you need the interface modifier :nnn.

# capture to wireshark file
tcpdump -s0 -nni 0.0:nnnp host 192.168.1.1 and port 443 -vw /var/tmp/hostname.pcap

-s0     # Unlimited Snaplen, this will provide the most data
-nn     # Don’t convert host addresses to names. This is used to avoid DNS lookups.
i 0.0   # Capture the traffic on interface 0.0 which tells the BIG-IP to use 'any' interface to gather this traffic on
nnnp    # Here you see the 'p' flag and what we call "full noise" by the use of the 'nnn'. 
        # This will create the information for the F5 Ethernet Trailers and the ‘p’ the traffic on both sides of the proxy.
-v      # will add verbosity and provide and screen counter so you see if packets are being caught, how many, and how fast
w       # this will send it to the file location
/var/tmp/hostname   # the path to the location and the file name
.pcap   # the file type, .cap is still used but is not quite as effective and pcapng is the newest form

Capture traffic for required host on all F5 network interface

When troubleshooting system connection or activity issues, this will show all traffic packets, very useful to watch all packets and identify any issues.

tcpdump -nn -i 0.0 host <system-ip>
    # -i    interface
    # 0.0   all interfaces
    Example
        tcpdump -nni 0.0 host 10.0.0.8  

ESXi and NSX tcpdump

tcpdump -i eth0:1 | grep <vCenter-hostname>     # Verify vCenter network flow

Quick tcpdump commands

# When doing the quick tcpdump checking
ip r    # check routing
tcpdump -i any port 25      # check tcp 25 on any network interface
tcpdump -n host 1.2.3.4     # check traffic in relates to host
tcpdump -n udp port 52      # check udp traffic
tcpdump -n not arp and not port 123 and not port 22