Understanding Clustering And How To Configure It

Technitium DNS Server v14 release comes with the much awaited Clustering support. This feature is aimed to allow managing two or more Technitium DNS Server instances much easier by allowing the entire cluster to be managed by logging into the DNS admin web panel of any one of the instances in the cluster. The DNS server instances (or nodes), that are part of the cluster, still work independently like they used to without clustering. The Clustering feature only helps sync configuration across all nodes.

Clustering Design

The Clustering design follows the same concept as that of the zones where you have one single primary zone on a DNS server and have one or more secondary zones on the other DNS servers that you have. The Clustering feature similarly has one primary node and can have one or more secondary nodes. This means that the config changes can be made only when the primary node is online and all the secondary nodes sync config from it.

It also has a notification mechanism similar to that of the zones, to notify all secondary DNS servers whenever there is an update available to be synced. The secondary nodes also has Config Refresh Interval which is used to periodically check for updates but the notification mechanism helps to sync the config quickly. There is also an option available to promote a secondary node to become a primary node in case when the primary node is offline and unrecoverable.

The DHCP Server that comes built-in is still the same as the previous release and does not yet support clustering for its scopes. However, this is planned for a later major release which is also planned to support DHCPv6 and a many other additional DHCP features.

The Clustering feature when enabled will add a node selector drop down control at the top right side of some views in the GUI. This drop down node selector control is available in sections that require switching between nodes. The Dashboard section has "Cluster" as an entry in the drop down selector to allow viewing aggregate stats for the entire cluster. The Settings section too has the "Cluster" entry in the drop down selector but it is used to list options that are common the all nodes in the cluster. For node specific options in the Settings, you need to select the specific node to manage them.

The zones are still independently managed by each DNS server instance. The Clustering feature uses Catalog zones feature to help manage zones in the cluster. To do this, the cluster creates a special cluster catalog zone which can be used to add member zones that must be synced across all the nodes in the cluster. Any DNSSEC signed primary zone added to this special cluster catalog zone gets its DNSSEC private keys synced with all nodes in the cluster. The DNSSEC tag in the GUI will now be shown in blue color when the primary/secondary zone has copy of private keys with it. All existing zones including Catalog zones will keep working the same way and are not affected by clustering. The zones on each server can be managed by selecting the required node from in the node selector drop down.

The Cache and the Log sections too work independently for each DNS server instance and you need to use the node selector drop down to select the node to manage. The options related to them in the Settings section are node specific and will show up when a specific node is selected from the node selector drop down.

The Allowed, Blocked, and Apps sections are covered completely by the clustering feature and they do not have any node selector drop down option. Any changes you make in these section are synced across all nodes automatically. This includes syncing DNS apps and any config changes made for them.

The DNS Client tool on the admin panel also has a node selector drop down near the buttons. This allows you to select the node to query to from via same admin panel you are logged in to. The "This Server" queries now depend on which node is selected in the drop down selector.

The Administration section now has a new "Cluster" tab to manage Clustering. The Users, Groups, and Permissions tabs in this section are synced across all nodes in the Cluster and they do not have any node selector drop down option. The Sessions tab has a node selector drop down since each node manages the login sessions separately. However, the API tokens created on the primary node in the cluster are synced across all nodes and are listed in the Sessions tab for all of them. The Cluster tab too has a node selector drop down to allow viewing node specific cluster management options.

Clustering is a proprietary feature and works only with Technitium DNS Server instances. However, you can continue to use Catalog zones with any other DNS server software that supports the standard for managing zones.

How To Initialize A Cluster

To create a new cluster, you need to first select the DNS server instance that must act as the primary node for the cluster. Login to the selected DNS server instance with an user who is member of Administrators group and navigate to Administration > Cluster section. Click on the Initialize drop down button and click on the New Cluster drop down menu item to open the Initialize New Cluster dialog as shown below.

Initializing A New Cluster

To initialize a new cluster, you need to provide the parameters explained below:

• Cluster Domain: It is the fully qualified domain name to be used for the cluster itself. This domain name is used to create a new cluster primary zone, if a primary zone with that name does not exist. The cluster will manage A, AAAA, and TLSA records for each cluster node in this primary zone.
  
  The cluster will also create a new special cluster catalog zone which uses "cluster-catalog" as the subdomain name of the Cluster Domain. This special cluster catalog zone is managed by the cluster internally. When you add a member zone in this cluster catalog zone, its NS and SOA records are automatically managed by the cluster so that you do not need to maintain NS and SOA records for each zone independently.
  
  The Cluster Domain Name can really be any domain name and can even be a private domain name that resolves only on your private network. If you have a local DNS server setup that is used as a recursive resolver, you can just use a private domain name and it will work with any issues. However, if you are running the cluster to host authoritative zones then using a public domain name will help make the setup easier due to the feature of automatic management of the NS and SOA records for member zones.
• Primary Node IP Addresses: The static IP addresses that can be used by other nodes to reach this DNS server instance. You can either enter the IP addresses manually or use the Quick Add drop down below to add the IP addresses detected by the DNS server. You can add both IPv4 and IPv6 addresses, and the cluster will create the required A/AAAA records for it in the cluster primary zone for this DNS server instance. Its important to note that these IP addresses must be either static or care must be taken to ensure that they do not change later to avoid breaking the cluster unexpectedly.

When the Cluster is initialized, the DNS Server Domain Name will be updated such that the current hostname is a subdomain name of the Cluster Domain Name. For example, if the current DNS Server Domain Name is "ns1.mydomain.tld" or just "ns1" then the new domain name will be "ns1.mycluster.tld" where "mycluster.tld" is the Cluster Domain Name. The DNS Server Domain Name can be updated later too by changing it from the Settings > General section and it will be automatically updated in the cluster primary zone.

Initializing the cluster also automatically enables HTTPS for the admin web service with a self-signed TLS certificate. Its recommended to manually configure the HTTPS with a valid TLS certificate before initializing the cluster so as to avoid using self-signed certificate but you can update to use a valid certificate later too by changing it from Settings > Web Service section.

A valid TLS certificate is useful when a secondary node attempts to join the cluster since the primary server can be authenticated using it. The cluster creates TLSA DANE-EE record for each node so later the certificate is validated only using DANE-EE and the domain name in the certificate no longer matters. If your cluster primary zone is a public resolvable one and has DS record set at the parent then DANE-EE will be used for authentication even for the initial joining process.

It is important to select the correct Cluster Domain since it cannot be changed later without having to delete the cluster and starting over again. However, you can change the individual node's DNS Server Domain Name from Settings but it must always be a subdomain name of the Cluster Domain.

How To Join A Cluster

Once you have the cluster initialized, you can configure other DNS server instances to join this cluster as a Secondary node to become a part of it. To join the cluster, login to the DNS server instance that must be configure as a secondary node. Navigate to the Administration > Cluster section on the admin panel and click on the Initialize button. Click on the Join Cluster drop down menu in there to open the Join Cluster dialog as shown below.

Joining A Cluster

To join an existing cluster, you need to provide the parameters explained below:

• Secondary Node IP Addresses: It is the static IP addresses of the secondary node that can be reached by all other nodes in the cluster. You can either enter the IP addresses manually or use the Quick Add drop down below to add the IP addresses detected by the DNS server. You can add both IPv4 and IPv6 addresses, and the cluster will create the required A/AAAA records for it in the cluster Primary zone for this DNS server node. Its important to note that these IP addresses must be either static or care must be taken to ensure that they do not change later to avoid breaking the cluster unexpectedly.
• Primary Node URL: Its the URL of the primary node's web service that is listed in the Administration > Cluster section on the primary node.
• Primary Node IP Address: You can specify an IP address of the primary node that must be used instead of resolving it automatically form the Primary Node URL. This option is useful when your Cluster Domain Name is private and not resolvable otherwise.
• Certificate Validation: This option allows you to specify if the primary node must be authenticated using PKI and DANE, or to ignore the TLS certificate validation errors while performing the joining process. Once the secondary node joins the cluster, it will always use DANE-EE to authenticate the other nodes in the cluster. If the cluster primary zone is publicly resolvable and has a DS record at the parent zone then the joining process will use DANE-EE unless the Ignore Certificate Validation Errors option is selected.
• Primary Node Username: The username of a user on the primary node who is a member of Administrators group.
• Primary Node Password: The password of the user mentioned above. If the user has Two-factor Authentication (2FA) enabled, an additional OTP field will be available on clicking the Join button.

When the joining process starts, it will first authenticate the user and start the initial config data sync process after adding itself as a node in the Cluster. This initial data sync process may take a while depending on the size of the data being synced. If there are apps installed, those are copied too increasing the overall size of the data transfer.

If the joining DNS server does not have HTTPS enabled, it will get enabled automatically with a self-signed certificate similar to the cluster initialization process. It is recommended to configure HTTPS manually with a valid certificate if available. The domain name in the certificate does not really matter since all the nodes in the Cluster will use DANE-EE to authenticate each other.

Its important to note that when a DNS server joins a cluster, it will sync settings from the primary node in the cluster and thus overwrite existing config files on the server. The existing zones in the cluster are not affected and will continue to work independently.

Cluster Options

The cluster options (shown below) can be updated by selecting the primary node in the node selector drop down in the Administration > Cluster section on the admin panel. These options are synced across the cluster automatically and sets the various timers in each node.

Cluster Options

All nodes check status of all the other nodes in the cluster at regular intervals to refresh their state. This "heartbeat" status refresh time is configurable from the Heartbeat Refresh Interval option. The Heartbeat Retry Interval is used to retry when the refresh process fails.

Nodes also attempt to refresh config from the primary node periodically even though there is a notification mechanism available to notify nodes when updates are available on the primary node. The Config Refresh Interval is used to set the timer to check for config updates periodically. The Config Retry Interval is used when to retry if the config refresh process fails.

Cluster Zone

The cluster zone is used by all the nodes to co-ordinate with each other. This zone holds IP addresses for all nodes and also has TLSA record for them to allow DANE-EE authentication. The cluster secondary zone on each secondary node in the cluster is updated via zone transfer mechanism whenever there is any change in the cluster primary zone. The zone transfer uses TSIG key for security which was created when the Cluster was initialized. Its important that the cluster secondary zone always sync and is up to date. If the cluster secondary zone fails to sync, it may have outdated A/TLSA records which can cause it to fail to communicate with the other nodes. If you see error messages in the DNS logs related to TLS certificate validation just after a node joins the cluster then you must check the cluster secondary zone and ensure that it is synced.

Cluster Administration

The Administration > Cluster section provides various options to manage the cluster. You need to select the required node using the node selector drop down to see the options available on that specific node. When the primary node is selected, you will have the Options button to configure cluster options and a Delete Cluster button to break the cluster.

Cluster Primary Node View

When in the secondary node view, the cluster options are available but are read-only. There is a Resync button to force a complete resync of cluster config to ensure that the secondary node has all the latest updates. This manual resync option is useful if the secondary node fails to update due to any operational issues. The Leave Cluster button allows removing the secondary node from the cluster.

Cluster Secondary Node View

The node table lists all cluster nodes and has a context menu to allow using additional options for each node. The self node's context menu has Edit Node option which allows you to update the node IP addresses if they need to be changed. On the primary node view, there is an context menu option to remove the secondary node from the cluster. On the secondary node view, there is Edit Node option for editing the primary node's IP addresses and the URL just in case if they were updated and the secondary node could not sync them automatically due to network issues.

In the secondary node view, the self node's context menu has a Promote To Primary option. This option when used will kick out the current primary node from the cluster and convert this secondary node into a primary node. This option is useful if you wish to decommission the existing primary node or if the primary node has failed and is unrecoverable for any reason. All secondary nodes in the cluster have a copy of DNSSEC private keys for all signed zones that are member of the cluster catalog zone and thus using this private key, the signed secondary zones are automatically converted to primary zones on the promoted node.

Conclusion

The Clustering feature which was one of the top requested features by many people is now available to be deployed with the latest release. It makes it easier to manage two or more DNS server instances by keeping all of the settings and options in sync across all nodes. Using the Promote To Primary option, you can promote any secondary node to become a primary node in case of failures and bring back the cluster to the operational state within few minutes. This feature is useful for all kinds of deployments be it a homelab or production DNS service in a large network.

There are a few more options being planned to improve the Clustering support. If you have any suggestions or feedback, do let me know in the comments section below or send an email to support@technitium.com.