I had a problem the other day where I did an IP migration on a server which was shipping slave zones to another server. Unfortunately, the IP migration script provided with plesk either failed to update the zone serials, or BIND9/rndc just didn't care for whatever reason. As such, the other nameserver kept returning responses with the old IPs, despite:

1. Restarting bind, reloading rndc on both
2. incrementing every zone serial on the zones we were authoritative for using sed
3. Manually wasting the zonefile on the slave, then running rndc retransfer and rndc resync on the master
4. rndc delzone of said slave, and then try retransfer/resync again

Nothing actually brought over the new zonefile. Obviously, more rigorous means of forcing the issue were required.

First, I needed to see if the zones on the slave were not updating, or if I was nuts. Unfortunately rndc showzone tells you next to nothing other than that a zone exists and where it comes from. This means I need to convert them to text, as the slave zones by default are shipped in bind's Binary format. To make a more generic alias, I've made a means to detect whether or not it's binary and "do the right thing".

showzone.source

showzone()
{
    echo ${1?Error domain name is not defined e.g. showzone test.test }
    # Update the following as appropriate to your environment, this works on plesk
    ZONE_LOC=/var/named/chroot/var

    file $ZONE_LOC/$1 | grep text &> /dev/null
    RESULT=$?
    if [ $RESULT == 0 ]
    then
        cat $ZONE_LOC/$1
    else
        /usr/sbin/named-compilezone -f raw -F text -o - $1 $ZONE_LOC/$1
    fi
}

The missing link here was doing rndc addzone in the same way it would happen automatically, to fool the system into thinking it was a brand new slave configured:

fix_borked_slaves.sh

#!/bin/bash

ZONE_LOC=/var/named/chroot/var

# Run this On slave:
for zone in $(ls $ZONE_LOC)
do
    if [ -z `rndc showzone $zone 2>/dev/null` ]
    then
        # Grab rndc's current definition
        TO_ADD=$(rndc showzone $zone 2>/dev/null | grep slave | awk '{ $1=$2=$3=""; print $0}')
        # Dump the current list for use on master
        echo "$zone" >> for_master_update.txt
        # Delete and re-add the zones
        rndc delzone $(rndc showzone $zone 2>/dev/null | grep slave | awk '{ print $2 }')
        rndc addzone $zone $TO_ADD
    fi
done

# Preamble to script you need to run on master, based on for_master_update.txt

# ADAPTER=eth0
# LOCAL_IP=$(ip addr show $ADAPTER | grep 'inet ' | awk '{print $2}' | sed -e 's/\/.*$//g')
# REMOTE_IP="REPLACEME"

# On REMOTE_IP, do a mass search/replace to make for_master_update.txt look like this and run it

#rndc -b $LOCAL_IP -s $REMOTE_IP -p 953 -y rndc-key retransfer $zone

This thorough application of the LART did the trick. I'm still not 100% sure what was BIND and rndc's problem here; bumping zone serials should always result in a transfer.

What would be better?

Nonsense like this is probably why cPanel's "DNS Clustering" never used afxrs or master/slave DNS in the first place. Eventually people liked it's enabling of a multi-master setup where you could edit the zones from anywhere in your cluster. It was of course a pretty funky system in that it had no consensus protocol whatsoever. Edit loops and Sibyl attacks were entirely possible and straightforward for customers to mistakenly produce for years because of confusing documentation. The documentation is better now, but the design of the system haven't fundamentally changed.

I was part of the team tasked with considering redesign of that system when we had to add DNSSEC support to it. Even the supermaster mode in pdns doesn't have the same strength as the RAFT protocol. As such I think the ideal setup for that is probably pdns in sqlite mode, utilizing RQlite. This has the added benefit of being simple for people to just setup themselves.