- Published on
tcpdump troubleshooting
- Authors
- Name
- Jackson Chen
tcpdump Examples
https://danielmiessler.com/study/tcpdump/
https://danielmiessler.com/study/
Commonly used tcpdump flags
Flag Description
--------------------------------------------------------
-i <interface> Listen on <interface>, .e.g. -i ens192
-n Do not perform reverse DNS resolution on IP addresses
-w <filename> Save capture in pcap format to <filename>, e.g. -w /tmp/wan.pcap
-s <bytes> Snap length: Amount of data to be captured from each frame
-c <packets> Exit after receiving a specific number of packets
-p Do not put the interface in promiscuous mode
-v Verbose output
-e Print link-layer header on each line
-s flag
By default tcpdump only saves the first 64 bytes of each frame when capturing to a file. This is enough to contain the IP and protocol header for most protocols, but limits the usability of capture files. By using the -s flag, tcpdump can be told how much of the frame to capture, in bytes. This is called the snap length.
Flag Description
---------------------------------
-s 500 Capture the first 500 bytes of each frame
-s 0 Capture each frame in its entirety
In most cases, using -s 0 is the best practice when capturing to a file for analysis on another system. The only exception to this is scenarios where a significant amount of traffic must be captured over a longer period of time. If the information being sought is known to be in the header, the default 64 bytes of each frame may be used to get the required information while significantly reducing the size of the resulting capture file.
-c flag
To capture a certain number of frames and then exit, use the -c flag. Example usage: tcpdump will exit after capturing 100 frames by specifying -c 100.
-p flag
Normally when capturing traffic with tcpdump, it puts the network interface into promiscuous mode. When not running in promiscuous mode, the interface only receives frames destined for its own MAC address as well as broadcast and multicast addresses.
When switched into promiscuous mode, the interface shows every frame on the wire that arrives at the network interface. In a switched network, this generally has little impact on the capture. In networks where the device is connected to a vswitch also in promiscuous mode, or a hub, using -p can significantly limit noise in the capture when the only traffic of interest is to and from the system performing the capture.
-v flag
The -v flag controls the detail, or verbosity, of the output. Using more v options yields more detail, so use -v, -vv, or -vvv to view even more detail in the output printed to the console. This option does not affect the detail stored in a capture file when using the -w switch, but will instead cause the process to report the number of packets captured every 10 seconds.
-e flag
Normally tcpdump does not show any link layer information. Specify -e to display the source and destination MAC addresses, and VLAN tag information for any traffic tagged with 802.1q VLANs.
# Example
tcpdump -ni any -e -c 5
Network filters
Network filters narrow the capture to a specific subnet using the net expression. Following net, specify a CIDR-masked network (192.168.1.0/24), dotted quad ( 192.168.1.1), dotted triple (192.168.1), dotted pair ( 192.168) or simply a number ( 192). A dotted quad is equivalent to specifying host, a dotted triple uses a subnet mask of 255.255.255.0, a dotted pair uses 255.255.0.0, and a number alone uses 255.0.0.0
# Example
tcpdump -ni igb1 src net 172.16.0.0/12
# displays traffic to or from any host with a 192.168.1.x IP address
tcpdump -ni igb1 net 192.168.1
Protocol filters
Specific protocols can be filtered using the proto directive or by using the protocol name directly. Parameters passed to the proto directive can be specified using the IP protocol number or one of the names icmp, igmp, igrp, pim, ah, esp, carp, vrrp, udp, or tcp. Because the normal protocol names are reserved words, they must be escaped with one or two backslashes when used with the proto directive, depending on the shell. The default shell available in pfSense software requires two backslashes to escape these protocol names. If the command returns a syntax error, check that the protocol name is properly escaped.
# The following capture will show all ICMP traffic on the igb1 interface:
tcpdump -ni igb1 proto \\icmp
Specifying carp for the protocol will capture CARP traffic but it also needs -T carp in order to interpret the CARP packets correctly when viewing the output using tcpdump. The GUI makes this adjustment automatically when capturing CARP
tcpdump -i igb1 -T carp carp
Display all HTTP traffic to and from multiple hosts
# Display all HTTP traffic from either 192.168.1.11 or 192.168.1.15:
tcpdump -ni igb1 host 192.168.1.11 or host 192.168.1.15 and tcp port 80
Filter expression usage
Filter expressions must come after every command line flag used. Adding any flags after a filter expression will result in a syntax error.
# tcpdump -ni igb1 -T carp carp -c 2
tcpdump: syntax error
# Correct ordering
tcpdump -ni igb1 -T carp -c 2 carp
Install tcpdump
yum install tcpdump or dnf install tcpdump
Analysis HTTPS traffic
# Show some https traffic with a hex display visible on the right portion of the outpu
tcpdump -nnSX port 443
Traffic on interface
# Get all interfaces
tcpdump -i any
tcpdump -nnpi any
tcpdump -nnpvi any
tcpdump -nnpvei any
# Get traffic on specific interface
tcpdump -i ens192
tcpdump -nnpvi ens192
tcpdump -nnpvei ens192
Traffic on IP or host
# Traffic can be filter on type
host
net
port
tcpdump host 1.2.3.4
tcpdump net 1.2.3.0/24
tcpdump port 8080
# Traffic directions
src
dst
tcpdump src 1.2.3.4
tcpdump dst 1.2.3.4
tcpdump src port 1025
tcpdump dst port 9443 # UAG console port
# Protocols
tcp
udp
icmp
tcpdump icmp
tcpdump tcp
tcpdump udp
Get packet contents with HEX output
tcpdump -c 1 -X icmp
Find traffic using port range
tcpdump portrange 21-23 # FTP port
Find traffic based on packet size
tcpdump less 32
tcpdump greater 64
tcpdump <= 128
Reading / writing captures to a file (PCAP)
# Capture traffic to file
tcpdump port 80 -w /tmp/capture_file
# Reading pcap file
tcpdump -r capture_file
More options
# Here are some additional ways to tweak how you call tcpdump.
Option Comment
------------------------------------------------------
-X Show the packet’s contents in both hex and ASCII.
-XX Same as -X, but also shows the ethernet header.
-D Show the list of available interfaces
-l Line-readable output (for viewing as you save, or sending to other commands)
-q Be less verbose (more quiet) with your output.
-t Give human-readable timestamp output.
-tttt Give maximally human-readable timestamp output.
-i eth0 Listen on the eth0 interface.
-vv Verbose output (more v’s gives more output).
-c Only get x number of packets and then stop.
-s Define the snaplength (size) of the capture in bytes.
Use -s0 to get everything,
unless you are intentionally capturing less.
-S Print absolute sequence numbers.
-e Get the ethernet header as well.
-q Show less protocol information.
-E Decrypt IPSEC traffic by providing an encryption key.
Combinations
# And
and
&&
# OR
or
||
# Except (Not)
not
!
Raw Output View
# See verbose oupt, with no resolutions of hostname, or port numbers
# Using absolute sequence numbers
# Showing human readable timestamps
tcpdump -ttnnvvs
Other examples
# From specific IP and destined for a specific port
tcpdump -nnvvS src 10.5.2.3 and dst port 3389
# From one network to another
tcpdump -nvX src net 192.168.0.0/16 and dst net 10.0.0.0/8 or 172.16.0.0/16
# Non-icmp traffic going to a specific port
tcpdump dst 192.168.0.2 and src net and not icmp
# All traffic from a host, that isn't on a specfic port
tcpdump -vv src SourceHost and not dst port 22
tcpdump -vv src 1.2.3.4 and not dst port 22
Keep in mind that when you’re building complex queries you might have to group your options using single quotes. Single quotes are used in order to tell tcpdump to ignore certain special characters—in this case below the “( )” brackets. This same technique can be used to group using other expressions such as host, port, net, etc
tcpdump 'src 10.0.2.4 and (dst port 3389 or 22)'
Isolate TCP flags
Isolate TCP RST flags, RST is causing traffic failed.
# The filters below find these various packets because tcp[13] looks at offset 13 in the TCP header,
the number represents the location within the byte,
and the !=0 means that the flag in question is set to 1, i.e. it’s on
tcpdump 'tcp[13] & 4!=0'
tcpdump 'tcp[tcpflags] == tcp-rst'
Isolate TCP SYN flags
tcpdump 'tcp[13] & 2!=0'
tcpdump 'tcp[tcpflags] == tcp-syn'
Isolate packets that have both the SYN and ACK flags set
tcpdump 'tcp[13]=18'
Only the PSH, RST, SYN, and FIN flags are displayed in tcpdump‘s flag field output. URGs and ACKs are displayed, but they are shown elsewhere in the output rather than in the flags field.
# Isolate TCP URG flags
tcpdump 'tcp[13] & 32!=0'
tcpdump 'tcp[tcpflags] == tcp-urg'
# Isolate TCP ACK flags
tcpdump 'tcp[13] & 16!=0'
tcpdump 'tcp[tcpflags] == tcp-ack'
# Isolate TCP PSH flags
tcpdump 'tcp[13] & 1!=0'
tcpdump 'tcp[tcpflags] == tcp-fin'
Because tcpdump can output content in ASCII, you can use it to search for cleartext content using other command-line tools like grep. The -l switch lets you see the traffic as you’re capturing it, and helps when sending to commands like grep.
# Both SYN and RST Set
tcpdump 'tcp[13] = 6'
# HTTP User Agents
tcpdump -vvAls0 | grep 'User-Agent:'
# leartext GET Requests
tcpdump -vvAls0 | grep 'GET'
# Find HTTP Host Headers
tcpdump -vvAls0 | grep 'Set-Cookie|Host:|Cookie:'
# Find SSH Connections
This one works regardless of what port the connection comes in on, because it’s getting the banner response
tcpdump 'tcp[(tcp[12]>>2):4] = 0x5353482D'
# Find DNS Traffic
tcpdump -vvAs0 port 53
# Find FTP Traffic
tcpdump -vvAs0 port ftp or ftp-data
# Find NTP Traffic
tcpdump -vvAs0 port 123
Troubleshooting analysis
https://docs.netgate.com/pfsense/en/latest/diagnostics/packetcapture/tcpdump.html
Port forward not working
In this example, a new port forward is failing to respond to a request from a host on the Internet. The troubleshooting steps outlined in Troubleshooting NAT Port Forwards offers one way to approach this, but sometimes packet capturing is the only or easiest way to find the source of the problem.
Start from WAN
First, make sure the traffic is getting to the WAN interface. Start a tcpdump session on the WAN interface, and watch for the traffic:
tcpdump -ni igb1 tcp port 5900
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb1, link-type EN10MB (Ethernet), capture size 96 bytes
11:14:02.444006 IP 172.17.11.9.37219 > 10.0.73.5.5900: S 3863112259:3863112259(0)
win 65535 <mss 1260,nop,nop,sackOK>
In this case, a packet comes in from the WAN, so it is making it that far. Note that the first part of the TCP handshake, a packet with only SYN set (the S shown), is reaching the firewall. If the port forward was working, a SYN ACK (S.) packet would be shown in reply to the SYN. With no return traffic visible, it could be a firewall rule or the target system may be unreachable – turned off, not listening on the specified port, host firewall blocking the traffic, etc.
Check Internal Interface
The next step would be to run a tcpdump session on the internal interface associated with the port forward:
tcpdump -ni igb0 tcp port 5900
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on igb0, link-type EN10MB (Ethernet), capture size 96 bytes
11:14:38.339926 IP 172.17.11.9.2302 > 192.168.30.5.5900: S 1481321921:1481321921(0)
win 65535 <mss 1260,nop,nop,sackOK>
Looking at the internal traffic, the connection left the inside interface and the local IP address was translated correctly. If this local address matches what was expected, then both the port forward and the firewall rule are working properly, and connectivity to the local PC must be confirmed by other means. If no output was displayed, then there is a problem with the firewall rule or the port forward may have been incorrectly defined. For this example, the target system was unplugged.