A few of our mail gateway servers running with postfix/policyd-weight/amavis/spamassaisin generate a lot of DNS queries to our DNS servers at times.
I’m not particularly concerned about that myself but there were some discussions about whether we should or how we could decrease the volume of DNS queries.
One suggestion for an easy and convenient solution was to just install the Name Service Cache Daemon (nscd) to cache responses locally on the mail server. They implemented this quickly on one of the servers but it didn’t really seem to work, it still generated loads of queries and the nscd statistics didn’t indicate that caching was working. Also, the statistics output of nscd -g always displayed 0% cache hit ratio and 0 cache hits on positive entries.
So they just as quickly abandoned the idea without digging into it deeper and more or less forgot about the whole plan in general, as it wasn’t like we had any real issues in the first place.
Other options discussed were setting up dedicated caching-only resolvers (onto the hosts themselves) which wouldn’t have been difficult either.
Fast forward a few months and the so called “issue” of too many DNS queries came up again recently and I decided to check nscd myself and why it supposedly wouldn’t work.
After installing and starting nscd and without really knowing anything about it, it’s easy to verify with stuff like ping and tcpdump whether DNS queries are actually leaving your host. Note that bind-tools like nslookup, dig or host always query DNS servers directly so you won’t see any effect of nscd when running these.
There’s not really anything to configure to make it work, the standard /etc/nscd.conf should work fine: (a change to my default RHEL/CentOS /etc/nsswitch.conf is not necessary either)
$ grep hosts /etc/nscd.conf enable-cache hosts yes positive-time-to-live hosts 3600 negative-time-to-live hosts 20 suggested-size hosts 211 check-files hosts yes persistent hosts yes shared hosts yes max-db-size hosts 33554432 $ grep hosts /etc/nsswitch.conf hosts: files dns
The cached values are stored in a binary file in /var/db/nscd/hosts, you can run strings on it to see which names are stored within:
# strings /var/db/nscd/hosts | grep -P '[\w-]+\.\w+' | sort -u google.com localhost4.localdomain4 localhost.localdomain www.google.com
However, I was puzzled as to why the nscd statistics (nscd -g) kept indicating 0 cache hits while it obviously did properly cache and return hosts:
# nscd -g | grep 'hosts cache' -A 22 hosts cache: yes cache is enabled yes cache is persistent no cache is shared 211 suggested size 216064 total data pool size 2256 used data pool size 3600 seconds time to live for positive entries 20 seconds time to live for negative entries 0 cache hits on positive entries 0 cache hits on negative entries 55 cache misses on positive entries 10 cache misses on negative entries 0% cache hit rate 17 current number of cached values 18 maximum number of cached values 2 maximum chain length searched 0 number of delays on rdlock 0 number of delays on wrlock 0 memory allocations failed yes check /etc/hosts for changes
After a bit of googling I found this important explanation in the comments explaining why this happened:
John on August 13th, 2008Adam, you cache is (probably) working, but nscd shows a 0% cache hit rate because you have the “shared” option turned on. That allows clients to directly search the nscd cache themselves instead of asking the nscd daemon; as a side effect, nscd can’t collect statistics about these search.
To verify that nscd is working, temporarily set “shared no”, restart nscd, and then cause some lookups. You should start to see a high cache hit rate %. Don’t forget to set “shared yes” again because it is much faster.
After setting the “shared” option to “no” to /etc/nscd.conf and restarting nscd I was now seeing statistics on cache hits too! Confusing bummer.
Having verified nscd indeed worked I turned to postfix, which still showed no signs of taking advantage of nscd. Turns out the postfix default directives always query DNS servers directly:
$ postconf | grep host_lookup
lmtp_host_lookup = dns
smtp_host_lookup = dns
What mechanisms the Postfix SMTP client uses to look up a host’s IP address. This parameter is ignored when DNS lookups are disabled (see: disable_dns_lookups).
dns Hosts can be found in the DNS (preferred).
native Use the native naming service only (nsswitch.conf, or equivalent mechanism).
(dns also means /etc/hosts entries are ignored by postfix too.)
So setting these options from “dns” to “native” and restarting postfix fixed this, nscd was now queried by postfix and returned cached values accordingly.
It should be noted however, that besides postfix other components like policyd-weight/amavis/spamassaisin still generate a lot of direct DNS queries without using nscd.
I haven’t found a way to disable this in a similar manner yet, if you have any idea please share it.