Contact Us

Is your app or device ready for deployment?
It can be.

Uncover performance, compliance, and operational issues prior to deployment.


Watch Overview



network emulation, Test you apps under all network conditions


Network Emulation


Test your app under all network conditions—3G/4G, satellite, WAN, cloud—in your lab.

network protocol testing,


Protocol Testing


Are the network protocols implemented properly and securely, conforming to the specs, in your app or device?




Test your app or device under all network conditions – from the routine to the extreme. Automate a wide variety of network conditions from 3G/4G to Cloud to WAN to Satellite to Internet. Uncover performance and operational issues prior to deployment.

Learn More



Finding and fixing software defects constitutes the largest expense for the software industry. Learn why line speed testing isn’t enough and the importance of functional testing!

Learn More



Trial deployments are expensive and yield limited data. Learn how to properly characterize your application or device performance prior to deployment.

Learn More

View More Papers


"Apple uses Maxwell to emulate all varieties of customer networks and access points — from limited network bandwidth to extreme latency to packet drops.

By testing our products with Maxwell prior to deployment, we ensure that customers have a solid experience when they get new Apple software or hardware in their hands."

— Dmitry Halavin, Apple Inc.

DNSSEC - Domain Name System Security - is not trivial.  In this article we talk about our experiences setting up DNSSEC using Bind 9.x on a Linux system with SELinux and with the whole thing in a chroot jail.

Over the last few days I have been activating DNSSEC for some of the domain names used by InterWorking Labs.

We run a stealth master server that feeds a cloud of secondary servers.

Our systems use Fedora 19 Linux with BIND 9.9 running in a chroot jail and constrained by SELinux.

We use static zone files – our update tool is Emacs. We do not do dynamic updates to our zones. We let BIND do the actual signing of the zones via the inline-signing yes; directive.

In addition we have set up our servers to follow the guidelines of the “Secure BIND Template” from TEAM CYMRU ( We run our servers with internal and external “views” so that we do will do full recursive resolution of any name for our own (internal) clients but for (external) other clients we only answer queries for names for which we are authoritative, i.e. our own domain names.

For DNSSEC we follow the advice given by Michael W. Lucas in his book DNSSEC Mastery ( However, useful as that book is, it does not deal with SELinux and only lightly mentions chroot jails. And it is silent about applying DNSSEC to “views”.

This note is a collection of things we tripped over while getting things running. This note does not claim to be complete or even well organized. However, we do hope that it may save others some time, effort, and frustration. We hope that this note will help promote the broad deployment of DNSSEC.

Let's begin with SELinux issues:

As I mentioned, we use static zone files. For non-DNSSEC configurations there really is no need to give BIND write access to the master zone filess; read access is sufficient.

But that changes under DNSSEC. We let BIND do the heavy lifting by specifying the “inline-signing yes;” directive in our named.conf file. (All of this is described in Lucas' DNSSEC Mastery.)

When BIND does the signing it creates several new files in the same directory as the static master zone file. These files have suffixes such as .jnl, .jbk, .signed, and .signed.jnl. (Note, not all of these files appear immediately upon signing of the zone; some appear later.)

So you need to tell SELinux that it is OK for BIND to write these files.

As root, issue the following command to tell SELinux to permanently allow access:

setsebool -P named_write_master_zones

The Selinux tags on the KSK and ZSK files should be unconfined_u:object_r:named_zone_t:s0.

Remember, SELinux acts in addition to any or the normal Linux permission flags. In other words even after your chown, chgrp, and chmod commands you have to open the SELinux door else BIND will find itself unable to write these files.

File Ownership and Permission Bits

I found that it is best if the files and directories that I touch are owned by user “root” and are in group “named”.

I generally try to set the permissions flags so that only root and named have access.

Make sure that BIND (user named) has write permission to the directory where the master zone files are stored. This triggered my own personal security alarms – what if BIND were to destroy my zone masters? The answer is “keep a backup”, advice that I will no doubt fail to adequately heed.

Where To Put Keys

I created a directory /var/named/chroot/var/named/keys where I put all of the DNSSEC keys. In that directory I have a subdirectory for each of my signed zones. In that subdirectory I put both the .key and .private files for my KSK and ZSK for that zone. I make double sure to remove the public flags from the file mode for the .private files.

Chroot Jail

The chroot jail itself did not cause any difficulties. However, one has to remember that everything lives down below /var/named/chroot.

Dual Horizon Views

If one follows the “Secure BIND Template” from TEAM CYMRU ( one will end up using the same master zone files for both the “internal-in” and “external-in” views.

This works until DNSSEC is activated.

When you turn on DNSSEC, BIND will create shadow versions of the master zone file for each view. These shadow files have the same name as the master zone file but with suffixes such as .jnl, .signed, .jbk, and .signed.jnl.

The trouble arises because each view uses the same master file. For each view, BIND tries to create shadow files with the same names. This confuses BIND, particularly with regard to zone serial number increments that are generated as part of BIND's inline signing process.

The trick is to create two master zone files, one for each view.

So where previously I may have had a master zone file called db.example I would now have an internal-in master named db.example_i and an external-in master named db.example_e.

If each of those files contained the zone data there would be maintenance trouble if the contents of those two files were to get out of sync – which is almost a certainty.

As they say in Computer Science 101 - “there is no problem in computer science that can not be solved by adding a level of indirection.”

Following that advice, what I did was to rename the original master zone file to something like example.rev. (The “rev” suffix is arbitrary, I just inherited that habit and never changed to something with meaning.) Then I created the two new master files, one for the internal view and one for the external view, each with its own name, and each merely incorporating by reference (that's our level of indirection) the original zone file.

Thus the db.example_i and db.example_e files become simple one line files with this single line of content:

$INCLUDE masters/example.rev

Don't forget to update the file directives in the zone sections in named.conf so that they point to the new file names as appropriate for each view.

Now when BIND does its inline signing it will create a set of shadow files for each of db.example_i and db_example_e without causing a collision.

Rndc Issues

Rndc did not always fully update BIND with changes that I had made. I found that it was often necessary to restart BIND entirely:

systemctl restart named-chroot

When using Rndc with a dial-view setup you need a few more parameters.

Thus to reload the zone example.tld you would need to send two Rndc commands:

rndc reload example.tld IN external-in
rndc reload example.tld IN internal-in

Starting With a Clean Slate

Several times I made a real mess of things. So I found it useful to remove all of those shadow files mentioned in the section “Dual Horizon Views”, above before using systemctl to restart named-chroot.

Remember, under the chroot jail you want to use the service named-chroot rather than the vanilla named.

Update Your Zone Serial Number

It shouldn't be necessary to mention – except that I forgot to do it several times – remember to update your zone serial numbers. I found it useful to update the serial number ever time I made any DNSSEC related change to a zone.

Generating KSK, ZSK, and DS

I keep a master set of all of my KSK, ZSK, and DS information in a named/key-archive directory. I keep all of my current and past key files here. BIND does not know about this directory and the access permissions should be set to block probing eyes.

I wrote a simple shell script to generate new KSK, ZSK, and DS files. This script takes a the list of names of the zones to be signed and produces a set of directory hierarchies, one for each zone, containing KSK, ZSK, and DS materials. If there are older materials those are retained. If I get confused about which ones are the most recent creations there's our old friend ls -lt.

There is a note at the end of the script that is fairly useful:

# Copy the resulting keys - both .key and .private -in the ksk and zsk directories to
# ../keys.
# Change to root.named
# Mode on .key files should be 644
# Mode on .private files should be 640
# Selinux tag on these should be unconfined_u:object_r:named_zone_t:s0

You will need to edit the script to insert your own list of zones rather than example.tld.

Here's the script:




function run_cmd() {
 echo Running: $*

function mk_dnssec_keys() {
 [ ! -d $1 ] && mkdir $1
 [ ! -d $1/ksk ] && mkdir $1/ksk
 [ ! -d $1/zsk ] && mkdir $1/zskM$include
 kf=`dnssec-keygen -q -f KSK -a ${KSK_ALG} -b ${KSK_BITS} -r /dev/urandom -K $1/ksk $1`
 zf=`dnssec-keygen -q -a ${ZSK_ALG} -b ${ZSK_BITS} -r /dev/urandom -K $1/zsk $1`
 dnssec-dsfromkey $1/ksk/${kf} > $1/${kf}.ds

for z in \
 example.tld \
 ; do
  mk_dnssec_keys ${z}

# Copy the resulting keys - both .key and .private -in the ksk and zsk directories to
# ../keys.
# Change to root.named
# Mode on .key files should be 644
# Mode on .private files should be 640
# Selinux tag on these should be unconfined_u:object_r:named_zone_t:s0


None of this is hard, rather there are just a lot of details. After every step you should take a look at your system log file – it will generally give you useful feedback.

Be patient. You may be generating DNS notification and your secondary servers will be picking up your new zone files. This may happen quickly or it may take some time depending on how responsive your secondaries are.

And once you start giving DS or DNSKEY data to your domain name registrar you will should expect at least five minutes for those to propagate to root servers, maybe longer. And if you are going through resolvers, those may have caches that could take a while to clear out – possibly an hour or two.

You may discover that your domain name registrar does not accept DNSSEC information through its web pages. In that case you may want to inquire whether they do this via special request or whether they simply do not support DNSSEC at all. If the latter your only alternative is to shift the domain name registration to another registrar. All ICANN accredited registrars are obligated under the 2013 agreement to support DNSSEC, but not all are yet in compliance.


Over 6000 network professionals trust IWL