This post explains how to use tcpdump on Linux to detect and investigate DNS requests. One of our Red Hat client systems was making requests to an old DNS server, even though it had been adjusted, through a change to/etc/resolv.conf, to point to a new one.

Requests to the old server were identified as follows.

[root@pluto root]# tcpdump -i eth0 -l -vvv dst host 192.168.1.103 and dst port 53
(...waited 15 mins or so...)
tcpdump: listening on eth0
16:38:18.019703 pluto.mycompany.com.41783 > olddnsbox.mycompany.com: [bad udp cksum 48f5!]  21331+ A? somebox.mycompany.com. (42) (DF) (ttl 64, id 48623, len 70)
16:38:18.033461 pluto.mycompany.com.41783 > olddnsbox.mycompany.com: [bad udp cksum 5919!]  12099+ A? somebox.mycompany.com. (42) (DF) (ttl 64, id 48625, len 70)

192.168.1.103 is the ip address of the old DNS server. Tcpdump shows network packets sent to the standard DNS port (53) at that IP address. Requests were few so I had to wait 15 or 20 minutes to capture the above.

The client was last rebooted a year ago, many months before /etc/resolv.conf was last edited. Tcpdump shows that some application is still querying the old server. The fix was to reboot the client, restarting the erroneous application and stopping the outdated requests.