Hosting many domains as a secondary: parallelizing "nsdc update"

[Stephane Bortzmeyer]

Our ns2.nic.fr is a secondary name server for 458 domains.

Many of them are broken and axfr times out.

Running "nsdc update" takes a *long* time, partly because it runs
sequentially. This is a real issue with nsd.

Also, some domains are more important than others (".fr" :-) and we
would like to process them first.

Therefore, I suggest the following solution:

1) from nsd.zones, produce a Makefile
2) run this Makefile, either with "make fr" to update only one domain,
or with "make -j N" if you want to run N axfr in parallel (yes, it
requires GNU make).

Advices? I attached the script get-one-zone, used by the Makefile to
retrieve one zone, and the script which converts nsd.zones to a
Makefile. 

It seems it works.
-------------- next part --------------
#!/usr/bin/python

# Converts a NSD zones file to a Makefile, to allow parallel updates.

# The program to get *one* zone
update_program = "/home/bortzmeyer/tmp/get-one-zone"
# The location of the zones file
zones_file = "/usr/local/nsd/etc/nsd.zones"
# The produced Makefile
makefile_name = "/tmp/Makefile"
#
verbose = True

import re, os, time

comment = re.compile("^\s*;")
zone_def = re.compile("^\s*zone\s+([a-z0-9\.-]+)\s+([^\s]+)\s+masters\s+(.+)$",
                      re.IGNORECASE)
temporary_makefile = makefile_name + ".tmp"

zones = open(zones_file)
makefile = open(temporary_makefile, "w")
makefile_content = ""
first = True
num = 0
for line in zones.xreadlines():
    if comment.search(line):
        continue
    match = zone_def.search(line)
    if match:
        zone = match.group(1)
        file = match.group(2)
        masters = match.group(3)
        makefile_content = "%s\n%s:\n\t %s %s %s %s\n" % (makefile_content,
                                                          zone, update_program,
                                                          zone, file, masters)
        if first:
            all = "all: %s" % (zone)
        else:
            all = "%s %s" % (all, zone)
        first = False
        num = num +1
        
makefile.write(
    "# AUTOMATICALLY GENERATED from %s on %s.\n# DO NOT EDIT!!!\n# %i domains\n%s\n\n" % \
    (zones_file,
     time.strftime ("%Y-%B-%d %H:%M", time.localtime(time.time())),
     num,
     all))
makefile.write(makefile_content)
makefile.close()
os.rename(temporary_makefile, makefile_name)
zones.close()
-------------- next part --------------
#!/bin/sh
#
# get-one-zone: runs nsdxfer for *one* zone
#
# Copyright (c) 2001-2004, NLnet Labs. All rights reserved.
#
# See LICENSE for the license.
#
#

# Optional configuration file for nsdc
configfile="/usr/local/nsd/etc/nsdc.conf"

#
# Default values in absense of ${configfile} (Usually ``/etc/nsd/nsdc.conf'')
#
prefix="/usr/local/nsd"
exec_prefix="/usr/local/nsd"

sbindir="/usr/local/nsd/sbin"
#zonesdir="/usr/local/nsd/zones"
zonesdir="/usr/local/nsd/zones/bidon"
flags=""
dbfile="/usr/local/nsd/var/nsd.db"
zonesfile="/usr/local/nsd/etc/nsd.zones"
#keysdir="/usr/local/nsd/etc/keys"
keysdir="/home/bortzmeyer/tmp/keys"
notify="/usr/local/nsd/sbin/nsd-notify"
nsdxfer="/usr/local/nsd/sbin/nsd-xfer"
pidfile="/var/run/nsd.pid"

ZONEC_VERBOSE=-v
NSD_XFER_VERBOSE=-v

#
# Read in configuration file if any
#
if [ -f ${configfile} ]
then
	. ${configfile}
fi

if [ "$3" = "" ]
then
     echo "Usage: $0 zone filename masters"
     exit 1
fi
zone=$1
file=$2
masters=$3

#
# You sure heard this many times before: NO USER SERVICEABLE PARTS BELOW
#
if [ ! -x "${nsdxfer}" ]
    then
    echo "${nsdxfer} program is not available, aborting..."
    exit 1
fi


# now get the serial number
serial_opt=''
if [ -e ${zonesdir}/$file ]; then
    serial=`awk '/.*IN[ \t]+SOA.*\($/ { getline; print $1; exit }' ${zonesdir}/$file`
    serial_opt="-s $serial"
fi

# take care of tsig info file if any
# See bug #91 - move to ${zone} but be backward
# compatible
unset tsiginfoarg
if [ -f "${keysdir}/${masters}.tsiginfo" ]
    then
    ln "${keysdir}/${masters}.tsiginfo" "${keysdir}/${masters}.tsiginfo.$$"
    tsiginfoarg="-T ${keysdir}/${masters}.tsiginfo.$$"
else 
    if [ -f "${keysdir}/${zone}.tsiginfo" ]
	then
	# the new way of doing things
	ln "${keysdir}/${zone}.tsiginfo" "${keysdir}/${zone}.tsiginfo.$$"
	tsiginfoarg="-T ${keysdir}/${zone}.tsiginfo.$$"
    fi
fi

# AXFR to a temp file $file.axfr
$nsdxfer ${NSDXFER_VERBOSE} -z $zone -f ${zonesdir}/$file.axfr ${tsiginfoarg} $serial_opt $masters
if [ $? -eq 1 ]
then
    if [ -f ${zonesdir}/$file.axfr ]
    then
	# axfr succeeded
	# test compile the zone to see what happens
	${sbindir}/zonec -o ${zone} -f ${zonesdir}/$file.axfr.db - < ${zonesdir}/$file.axfr 2>/dev/null
	if [ $? -eq 1 ]
	then
	    echo "Warning: AXFR of $zone did not compile"
	    rm -f ${zonesdir}/$file.axfr
	else
	    # we succeed
	    mv -f ${zonesdir}/$file.axfr ${zonesdir}/$file
	fi
	rm -f ${zonesdir}/$file.axfr.db
    else
	echo "Warning: AXFR for $zone failed"
    fi
fi

# clean up
rm -f -- "${keysdir}"/*.tsiginfo.*