nscd DNS caching and postfix

A few of our mail gateway servers running with postfix/policyd-weight/amavis/spamassaisin generate a lot of DNS queries to our DNS servers at times.
I’m not particularly concerned about that myself but there were some discussions about whether we should or how we could decrease the volume of DNS queries.

One suggestion for an easy and convenient solution was to just install the Name Service Cache Daemon (nscd) to cache responses locally on the mail server. They implemented this quickly on one of the servers but it didn’t really seem to work, it still generated loads of queries and the nscd statistics didn’t indicate that caching was working. Also, the statistics output of nscd -g always displayed 0% cache hit ratio and 0 cache hits on positive entries.
So they just as quickly abandoned the idea without digging into it deeper and more or less forgot about the whole plan in general, as it wasn’t like we had any real issues in the first place.
Other options discussed were setting up dedicated caching-only resolvers (onto the hosts themselves) which wouldn’t have been difficult either.

Fast forward a few months and the so called “issue” of too many DNS queries came up again recently and I decided to check nscd myself and why it supposedly wouldn’t work.

After installing and starting nscd and without really knowing anything about it, it’s easy to verify with stuff like ping and tcpdump whether DNS queries are actually leaving your host. Note that bind-tools like nslookup, dig or host always query DNS servers directly so you won’t see any effect of nscd when running these.
There’s not really anything to configure to make it work, the standard /etc/nscd.conf should work fine: (a change to my default RHEL/CentOS /etc/nsswitch.conf is not necessary either)

$ grep hosts /etc/nscd.conf
        enable-cache            hosts           yes
        positive-time-to-live   hosts           3600
        negative-time-to-live   hosts           20
        suggested-size          hosts           211
        check-files             hosts           yes
        persistent              hosts           yes
        shared                  hosts           yes
        max-db-size             hosts           33554432

$ grep hosts /etc/nsswitch.conf
     hosts:      files dns

The cached values are stored in a binary file in /var/db/nscd/hosts, you can run strings on it to see which names are stored within:

# strings /var/db/nscd/hosts | grep -P '[\w-]+\.\w+' | sort -u

However, I was puzzled as to why the nscd statistics (nscd -g) kept indicating 0 cache hits while it obviously did properly cache and return hosts:

# nscd -g | grep 'hosts cache' -A 22
 hosts cache:
 yes  cache is enabled
 yes  cache is persistent
 no  cache is shared
 211  suggested size
 216064  total data pool size
 2256  used data pool size
 3600  seconds time to live for positive entries
 20  seconds time to live for negative entries
 0  cache hits on positive entries
 0  cache hits on negative entries
 55  cache misses on positive entries
 10  cache misses on negative entries
 0% cache hit rate
 17  current number of cached values
 18  maximum number of cached values
 2  maximum chain length searched
 0  number of delays on rdlock
 0  number of delays on wrlock
 0  memory allocations failed
 yes  check /etc/hosts for changes

After a bit of googling I found this important explanation in the comments explaining why this happened:
John  on August 13th, 2008Adam, you cache is (probably) working, but nscd shows a 0% cache hit rate because you have the “shared” option turned on. That allows clients to directly search the nscd cache themselves instead of asking the nscd daemon; as a side effect, nscd can’t collect statistics about these search.
To verify that nscd is working, temporarily set “shared no”, restart nscd, and then cause some lookups. You should start to see a high cache hit rate %. Don’t forget to set “shared yes” again because it is much faster.

After setting the “shared” option to “no” to /etc/nscd.conf and restarting nscd I was now seeing statistics on cache hits too! Confusing bummer.

Having verified nscd indeed worked I turned to postfix, which still showed no signs of taking advantage of nscd. Turns out the postfix default directives always query DNS servers directly:
$ postconf | grep host_lookup
lmtp_host_lookup = dns
smtp_host_lookup = dns
What mechanisms the Postfix SMTP client uses to look up a host’s IP address. This parameter is ignored when DNS lookups are disabled (see: disable_dns_lookups).

dns Hosts can be found in the DNS (preferred).
native Use the native naming service only (nsswitch.conf, or equivalent mechanism).

(dns also means /etc/hosts entries are ignored by postfix too.)

So setting these options from “dns” to “native” and restarting postfix fixed this, nscd was now queried by postfix and returned cached values accordingly.

It should be noted however, that besides postfix other components like policyd-weight/amavis/spamassaisin still generate a lot of direct DNS queries without using nscd.
I haven’t found a way to disable this in a similar manner yet, if you have any idea please share it.

4 thoughts on “nscd DNS caching and postfix

  1. Pingback: Configurar la caché DNS de sistema en Linux | Hello, IT.

  2. I don’t think you want amavis or spamassassin to use the cache, because with even a moderate amount of spam you will fill the cache with dns hits (especially spamassassin which uses dns queries of message hashes to various spam RBLs and cloudmark etc. Save the cache for things that affect interactive users, web browsing, etc. (Especially nscd is used on your firewall).

    BTW, thanks for this article.

  3. Pingback: Unix:How can I improve nscd's cache hit rate? – Unix Questions

  4. Thanks! That solved the mystery for me. I wasn’t able to figure out why the cache hit rate was sticking at 0.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s