[[!meta author="Peter Palfrader"]]
# Abstract
I recently moved our primary nameserver from `orff.debian.org`, which is
an aging blade in Greece, to a VM on one of our ganeti clusters. In the
process, I rediscovered a lot about our DNS infrastructure. In this post,
I will describe the many sources of information and how they all come
together.
# Introduction
The [Domain Name System][DNS] is the hierarchical database and query
protocol that is in use on the Internet today to map hostnames to IP
addresses, to map the reverse thereof, to lookup relevant servers for
certain services such as mail, and a gazillion other things. Management
and authority in the DNS is split into different zones, subtrees of the
global tree of domain names.
Debian currently has a bit over a score of zones. The two most
prominents clearly are `debian.org` and
`debian.net`. The rest is made up of `debian`
domains in various other top level domains and reverse zones, which
are utilized in IP address to hostname mappings.
# Types and sources of information
The data we put into DNS comes from a wide range of different systems:
* [Classical zonefiles maintained in git][domains]. This represents
the core of our domain data. It maps services like
`blends.debian.org` to `static.debian.org` or
specifies the servers responsible for accepting mail to
`@debian.org` addresses. It also is where all the
`ftp.CC.debian.org` entries are kept and maintained together with the
mirror team.
* Information about `debian.org` hosts, such as `master`, is maintained
in Debian's [userdir LDAP][db], queryable using
LDAP[^ldap].
* This includes first and foremost the host's IP addresses (v4 and v6).
* Additionally, we store the server responsible for receiving a host's
mail in LDAP (`mXRecord` LDAP attribute, DNS `MX` record type).
* LDAP also has some specs on computers, which we put into each host's
`HINFO` record, mainly because we can and we are old-school.
* Last but not least, LDAP also has each host's public ssh key, which we
extract into [SSHFP][rfc4255] records for DNS.
* LDAP also has per-user information. Users of debian infrastructure
can attach limited DNS elements as `dnsZoneEntry` attributes to their
user[^ldap2].
* The auto-dns system (more on that below).
* Our [puppet] also is a source of DNS information. Currently it
generates only the [`TLSA`][rfc6698] records that enable clients to
securely authenticate certificates used for mail and HTTPS, similar
to how `SSHFP` works for authenticating ssh host keys.
# Debian's auto-dns and geo setup
We try to provide the best service we can. As such, our goal is that,
for instance, user access to [`www`][www] or [`bugs`][bugs] should always
work. These services are, thus, provided by more than one machine on
the Internet.
However, HTTP did not specify a requirement for clients to re-try a
different server if one of those in a set is unavailable. This means
for us that when a host goes down, it needs to be removed from the
corresponding DNS entry.
Ideally, the world wouldn't have to wait for one of us to notice and
react before they can have their service in a working manner.
Our solution for this is our auto-dns setup. We [maintain a
list][auto-dns] of hosts that are providing a service. We [monitor them
closely][mini-nag]. Whenever a server goes away or comes back we
automatically rebuild the zone that contains the element.
This setup also lets us reboot servers cleanly — since one of the
things we monitor is "is there a shutdown running", we can, simply by
issuing a `shutdown -r 30 kernel-update`, de-rotate the machine in
question from DNS. Once the host is back it'll automatically get
re-added to the round-robin zone entry.
The auto-dns system produces two kinds of output:
* In *service*-mode it generates a file with just the address records
for a specific service. This snippet is then included in its zone
using a standard bind `$INCLUDE` directive. Services that work like
this include [`bugs`][bugs] and `static` ([service definition for
static][static]).
* In *zone*-mode, auto-dns produces zonefiles. For each service it
produces a set of zonefiles, one for each out of a set of different
geographic regions. These individual zonefiles are then transferred
using `rsync` to our [GEO-IP enabled][geoip] nameservers. This
enables us to give users a list of `security` mirrors closer to them
and thus hopefully faster for them.
# Tying it all together
Figure 1: The Debian DNS Rube Goldberg Machine.
Once all the individual pieces of source information have been
collected, the `dns-update` and `write_zonefile` scripts from our
[dns-helpers] repository take over the job of building complete
zonefiles and a bind configuration snippet. Bind then loads the zones
and notifies its secondaries.
For geozones, the zonefiles are already produced by auto-dns'
`build-zones` and those are pulled from the geo nameservers via rsync
over ssh, after an ssh trigger.
# and also DNSSEC
All of our zones are signed using DNSSEC. We have a script in
[dns-helpers] that produces, for all zones, a set of rolling signing
keys. For the normal zones, bind 9.9 takes care of signing them
in-process before serving the zones to its secondaries. For our
geo-zones we sign them in the classical `dnssec-signzone` way before
shipping them.
The secure delegation status (DS set in parent matches DNSKEY in child)
is monitored by a set of nagios tests, from both [dsa-nagios] and
[dns-helpers]. Of these, `manage-dnssec-keys` has a dual job: not
only will it warn us if an expiring key is still in the DSset, it can
also prevent it from getting expired by issuing timly updates of the
keys metadata.
# Relevant Git repositories
* [domains]
* [auto-dns]
* [mini-nag]
* [dns-helpers]
* [puppet]
[^ldap]: `ldapsearch -h db.debian.org -x -ZZ -b dc=debian,dc=org -LLL 'host=master'`
[^ldap2]: `ldapsearch -h db.debian.org -x -ZZ -b dc=debian,dc=org -LLL 'dnsZoneEntry=*' dnsZoneEntry`
[DNS]: https://en.wikipedia.org/wiki/Domain_Name_System
[db]: https://db.debian.org/
[rfc4255]: https://tools.ietf.org/html/rfc4255
[rfc6698]: https://tools.ietf.org/html/rfc6698
[www]: https://www.debian.org/
[security]: http://security.debian.org/
[bugs]: http://bugs.debian.org/
[geoip]: https://en.wikipedia.org/wiki/Geolocation_software
[puppet]: http://anonscm.debian.org/gitweb/?p=mirror/dsa-puppet.git;a=tree
[domains]: http://anonscm.debian.org/gitweb/?p=mirror/domains.git;a=tree
[auto-dns]: http://anonscm.debian.org/gitweb/?p=mirror/dsa-auto-dns.git;a=tree
[mini-nag]: http://anonscm.debian.org/gitweb/?p=mirror/dsa-mini-nag.git;a=tree
[dns-helpers]: http://anonscm.debian.org/gitweb/?p=mirror/dns-helpers.git;a=tree
[static]: http://anonscm.debian.org/gitweb/?p=mirror/dsa-auto-dns.git;a=blob;f=services/static.debian.org.service;hb=HEAD
-- Peter Palfrader