Unplanned Interruption: Intermittent DNS Connectivity

Aus Net Servers Australia was made aware of an issue regarding intermittent connectivity between our DNS Cluster at around 11:54PM Friday night. Engineers where dispatched to investigate the issue.

The problem turned out to be a glitch with BIND (the software that runs the root DNS software). This was corrected and the servers where put back into service.

At 6:28AM Saturday morning dns-server-1.vic.ausnetservers.net.au housing the records for ns1.ausnetservers.net.au & ns2.ausnetservers.net.au crashed and went offline. For some reason the cluster did not fall over as it should have and kept routing dns queries onto dns-server-1.vic.ausnetservers.net.au. Instead it should have fallen over onto the remaining DNS clustered servers and stopped sending queries onto the effected servers.

At 3:45PM Saturday afternoon, dns-server-2.vic.ausnetservers.net.au housing the records for ns3.ausnetservers.net.au & ns4.ausnetservers.net.au did a kernel stack crash and went offline

This left the entire ANSA Websites unreachable for about 4 hours.

At 7:31PM Saturday, our engineers brought dns-server-1.vic.ausnetservers.net.au back online and followed by dns-server-2.vic.ausnetservers.net.au about 3 hours later.

All servers are now back online and all services are looking stable. All traffic is flowing through the network nicely.

Our engineers have been looking into what caused the original issue in the first place and why the servers crashed as a result. If you are still having issues please contact NOC[at]ausnetservers[dot]net[au]