NSD and OpenDNSSEC under FreeBSD 10 [Part 5: SafeNet HSM]

This is the fifth part in the series of articles explaining how to run NSD and OpenDNSSEC under FreeBSD 10.

This time we’re going to integrate proper hardware HSM support in our setup — a pair of SafeNet Network HSMs (aka Luna SA).

Here is how our updated installation diagram looks like:

2016051701

Before we jump into technical details there are a couple of assumptions:

— I assume that HSMs are already configured and partitioned. HSM installation is outside of scope of this guide since it’s a lengthy and pretty time consuming process which has nothing to do with OpenDNSSEC. It also involves a big chunk of work to be done on the access federation field (different teams accessing different partitions with different PEDs or passwords). SafeNet HSM’s documentation is quite solid though, so make sure this part is completed. In our setup, both HSMs run the latest software 6.2.0-15 and there is one partition created on both units called TEST. TEST partition is activated and we’re going to create High Availability group, add both HSMs to the HA group and allow NS-SIGN to access it;

— As you might have noticed, I decided to leave ZSKs to be handled by SoftHSM. One of the things that you’ll have to keep an eye on with network HSMs is the HDD space. The way it works with SafeNet is that you have an appliance with some fixed amount of disk space (let’s say 2MB). Then you create partitions and allocate space out of total amount for each partition (by default it’s equal distribution). So let’s assume we created five partitions 417274 bytes each. Normally, storing a pair of public/private key consumes very little, but with OpenDNSSEC we’re talking about a number of domains each storing a pair of public/private keys for both KSK and ZSK. It’s very important to understand how far you can go, so you’re not surprised after several years when you discover that you run out of space.

Let’s do some basic math: one domain, with both ZSK (1024) and KSK (2048) stored on HSM, will consume 2768 bytes, so with 417274 bytes partition you should be able to handle ~150 domains. However, during ZSK or KSK rollover, another pair will be temporarily created, and although ZSK/KSK rollover shouldn’t happen at the same time and OpenDNSSEC will purge expired keys after the rollover is completed, you’ll have to consider extra 2768 bytes per domain (for a period of time defined in <Purge> stanza in kasp.xml), which leaves you 75 domains. As you can see this isn’t much. That’s why I decided to keep SoftHSM for ZSKs to save some HSM space (which is not cheap to say the least!).

One of the disadvantages of keeping both storage engines is that you’ll have one more dependency to worry about should you consider to upgrade (for example to SoftHSM2), hence the choice is yours. Another option would be to store private keys in HSM and leave public keys aside (<SkipPublicKey/> option under conf.xml), but I’ve read that it’s very much dependent on the HSM provider and could lead to unexpected results. And one more option would be to use <ShareKeys/> under kasp.xml — that way you can share the same key for multiple domains.

So, let’s access NS-SIGN, adjust the firewall policy, and install prerequisites first. In terms of ports, NS-SIGN will communicate with HSM-1 and HSM-2 using three ports: 1792/tcp, 1867/tcp and 22/tcp. Opening SSH is not strictly needed (it’s mainly for initial configuration) and will be removed once the integration is completed. 1792/tcp and 1867/tcp are mandatory though: 1792/tcp is used for Network Trust Link Service (NTLS), and 1867/tcp is used for a Network Trust Link (NTL) between a client and a partition. Since we’re using pf here is how it looks like:

luna_hsm_services = "{ ssh, 1792, 1867 }"   # SafeNet Luna HSM ports
...
table  const { 10.9.128.42, 10.9.128.43 }
...
# Outgoing traffic to SafeNet Luna HSMs
pass out quick on xl0 inet proto tcp from xl0 to  port $luna_hsm_services flags S/SA modulate state

Where 10.9.128.42 and 10.9.128.43 are IPs of our HSMs.

SafeNet HSM client software does support FreeBSD, but it’s compiled for FreeBSD 9 with libstdc++.so dependency, while I’m under FreeBSD 10. You will still be able to install and use it though.

Let’s install gcc first, to get libstdc++.so dependency resolved:

% cd /usr/ports/lang/gcc && make install clean

Download SafeNet Luna HSM client package from the SafeNet website, unpack and copy the content of freebsd\9\64\ directory to NS-SIGN. We’ll need to install the following five packages:

cmuFreebsd-6.2.0.tar.gz
htl_clientFreebsd-6.2.0.tar.gz
libcryptokiFreebsd-6.2.0.tar.gz
lunacmFreebsd-6.2.0.tar.gz
vtlFreebsd-6.2.0.tar.gz

Out of those five, libcryptoki is the only really needed by OpenDNSSEC. This is a PKCS #11 library through which OpenDNSSEC will communicate with the HSMs. Remaining four are SafeNet utilities and tools.

CMU is the SafeNet Certificate Management Utility. Could be useful if you want to deal with objects placed in the HSM (like to delete or to list an attribute).

HTL client (optional) is a quite important piece. HTL or Network Trust Link is used when you want to add additional protection layer on top of NTLS for virtual servers, to ensure that the HSM connects only to a trusted client instance that is in possession of a one-time-token generated by the HSM. Basically this is a service that constantly checks whether the link between NS-SIGN and HSM-1/2 is up and if it’s discovered to be down for more than 300 seconds NTLS is dropped and no communication to HSMs is possible anymore until you re-generate and re-import new tokens. In practical terms that means that you can reboot NS-SIGN no problem, but if someone clones it and then powers it on, they’ll most likely hit 300 seconds delay and HTL has to be re-established manually.

LunaCM is the SafeNet client-side administrative command interface for SafeNet HSMs. We’ll need it for creating HA groups.

VTL — the SafeNet Virtual Token Library is the legacy utility.

During the installation, you’ll get a time-out and packagesite mismatch error that could be ignored:

% pkg install libcryptokiFreebsd-6.2.0.txz

Updating FreeBSD repository catalogue...
Repository FreeBSD has a wrong packagesite, need to re-create database
pkg: http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest/meta.txz: Operation timed out
repository FreeBSD has no meta file, using default settings
pkg: http://pkg.FreeBSD.org/FreeBSD:10:amd64/latest/packagesite.txz: Operation timed out
Unable to update repository FreeBSD
All repositories are up-to-date.
Repository FreeBSD has a wrong packagesite, need to re-create database
pkg: Repository FreeBSD cannot be opened. 'pkg update' required
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
        libcryptokiFreebsd: 6.2.0

Proceed with this action? [y/N]: y
[1/1] Installing libcryptokiFreebsd-6.2.0...
x libcryptoki/
x libcryptoki/pkg-deinstall
x libcryptoki/pkg-install
x libcryptoki/libCryptoki2_64.so
x libcryptoki/Chrystoki.conf
x libcryptoki/Makefile
x libcryptoki/pkg-descr
Checking for /etc/Chrystoki.conf.bsdsave
Copying default Chrystoki.conf file to /etc/
Added ReceiveTimeout entry.
Added SSLConfigFile entry.
Added ClientPrivKeyFile entry.
Added ClientCertFile entry.
Added ServerCAFile entry.
Added NetClient entry.
Added TCPKeepAlive entry.

% pkg install cmuFreebsd-6.0.0.txz

% pkg install htl_clientFreebsd-6.0.0.txz

% pkg install lunacmFreebsd-6.0.0.txz

% pkg install vtlFreebsd-6.0.0.txz

Upon completion, you should have /usr/safenet/lunaclient directory created with bin/, cert/, htl/ and lib/ directories underneath. You’ll also have /etc/rc.d/htl_service and /etc/Chrystoki.conf files dropped.

Before we start with the configuration make sure that NS-SIGN, HSM-1 and HSM-2 have DNS entries registered and they can resolve each other using FQDNs. We’ll be dealing with certificates hence DNS names are crucial. In my example, they all have the following internal DNS names created: ns-sign.local.net, hsm-1.local.net and hsm-2.local.net.

Let’s start with the configuration. First, we need to copy server.pem certificate to NS-SIGN from both HSMs. So, from NS-SIGN:

% cd /usr/safenet/lunaclient

% scp admin@hsm-1.local.net:server.pem cert/server
admin@hsm-1.local.net password:

server.pem          100% 1172     1.1KB/s   00:00

% ./bin/vtl addServer -n hsm-1.local.net -c cert/server/server.pem -htl

New server hsm-1.local.net successfully added to server list.

% scp admin@hsm-2.local.net:server.pem cert/server
admin@hsm-2.local.net password:

server.pem          100% 1172     1.1KB/s   00:00

% ./bin/vtl addServer -n hsm-2.local.net -c cert/server/server.pem -htl

New server hsm-1.local.net successfully added to server list.

% ./bin/vtl listServers
Server: hsm-1.local.net HTL required: yes
Server: hsm-2.local.net HTL required: yes

Now we need to generate NS-SIGN certificate and send it to both HSMs. From NS-SIGN:

% cd /usr/safenet/lunaclient

% ./bin/vtl createCert -n ns-sign.local.net -c US -s NY -l NY -o ORG -u DEV
Private Key created and written to: /usr/safenet/lunaclient/cert/client/ns-sign.local.netKey.pem
Certificate created and written to: /usr/safenet/lunaclient/cert/client/ns-sign.local.net.pem

% scp cert/client/ns-sign.local.net.pem admin@hsm-1.local.net:
admin@hsm-1.local.net password:

ns-sign.local.net.pem   100%  867     0.9KB/s   00:00

% scp cert/client/ns-sign.local.net.pem admin@hsm-2.local.net:
admin@hsm-2.local.net password:

ns-sign.local.net.pem   100%  867     0.9KB/s   00:00

Enable HTL service on NS-SIGN by editing /etc/rc.conf:

# enable htl
htl_client_enable="YES"

The default /etc/rc.d/htl_service script didn’t work for me out of the box, therefore I had to modify it (# KEYWORD: shutdown):

#!/bin/sh -
#
# Copyright (c) 2004  The FreeBSD Project
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
#
# $FreeBSD: release/9.2.0/etc/rc.d/mixer 242153 2012-10-26 18:06:49Z obrien $
#

# PROVIDE: htl_service
# REQUIRE: DAEMON
# BEFORE: LOGIN
# KEYWORD: shutdown

. /etc/rc.subr

name="htl_client"
rcvar="htl_client_enable"
stop_cmd="htl_client_stop"
start_cmd="htl_client_start"

load_rc_config $name
: ${htl_client_enable:=no}

htl_client_start()
{
   /usr/safenet/lunaclient/htl/htl_client
   echo "Started Htl Service"
}

htl_client_stop()
{
   HTLC_P=`pgrep -f htl_client`
   if [ "$HTLC_P" != "" ] ; then
        # First send a kill signal and wait a bit
        kill $HTLC_P
        sleep 10

        # If it is still running we now force kill it
        HTLC_P=`pgrep -f htl_client`
        if [ "$HTLC_P" != "" ] ; then
            # try three times just in case
            kill -9 $HTLC_P
            kill -9 $HTLC_P
            kill -9 $HTLC_P
        else
                echo "OK"
        fi
   fi
}

load_rc_config $name
run_rc_command "$1"

I also modified the startup script of OpenDNSSEC to make sure that HTL service is started prior to OpenDNSSEC:

% cat /usr/local/etc/rc.d/opendnssec | grep -i htl
# REQUIRE: LOGIN DAEMON htl_service

Here is how you can check the startup order of services under FreeBSD by the way:

% rcorder /etc/rc.d/* /usr/local/etc/rc.d/*
...
/etc/rc.d/htl_service
...
/usr/local/etc/rc.d/opendnssec

Start the HTL service:

% /etc/rc.d/htl_service start

SSH to HSM-1 and register the client. From HSM-1:

[hsm-1] lunash:> client register -client ns-sign -hostname ns-sign.local.net -requirehtl -generateott

'client register' successful.

Generating one-time token...
One-time token for client ns-sign is ready to use.
Filename is ns-sign.ott

Command Result : 0 (Success)

[hsm-1] lunash:>client hostip map -client ns-sign -ip 10.9.128.2

Command Result : 0 (Success)

‘client hostip map’ is not really needed. This is to ensure that even if DNS is down and HSMs cannot resolve ns-sign.local.net the link is still up by using the IP address.

SSH to HSM-2 and register the client. From HSM-2:

[hsm-2] lunash:> client register -client ns-sign -hostname ns-sign.local.net -requirehtl -generateott

'client register' successful.

Generating one-time token...
One-time token for client ns-sign is ready to use.
Filename is ns-sign.ott

Command Result : 0 (Success)

[hsm-2] lunash:>client hostip map -client ns-sign -ip 10.9.128.2

Command Result : 0 (Success)

ott files used to establish HTL have been generated and we need to transfer them to NS-SIGN. From NS-SIGN:

% cd /usr/safenet/lunaclient

% scp admin@hsm-1.local.net:ns-sign.ott htl/hsm-1.local.net.ott
admin@hsm-1.local.net password:

ns-sign.ott              100%   32     0.0KB/s   00:00

% scp admin@hsm-2.local.net:ns-sign.ott htl/hsm-2.local.net.ott
admin@hsm-2.local.net password:

ns-sign.ott              100%   32     0.0KB/s   00:00

If everything is fine, you should see .key and .pem files created under htl/ directory for each HSM. From NS-SIGN:

% cd /usr/safenet/lunaclient

% ls -al htl/hsm*

-rw-r--r--  1 root  wheel  1743 May 16 16:03 htl/hsm-1.local.net.key
-rw-r--r--  1 root  wheel    32 May 18 16:54 htl/hsm-1.local.net.ott
-rw-r--r--  1 root  wheel  1306 May 16 16:03 htl/hsm-1.local.net.pem
-rw-r--r--  1 root  wheel  1751 May 16 16:03 htl/hsm-2.local.net.key
-rw-r--r--  1 root  wheel    32 May 18 16:54 htl/hsm-2.local.net.ott
-rw-r--r--  1 root  wheel  1306 May 16 16:03 htl/hsm-2.local.net.pem

Another way is to check it from one of the HSMs. From HSM-1:

[hsm-1] lunash:>htl show

HTL Grace period   :  60 seconds
Default OTT expiry :  300 seconds

 Client Name         HTL Status     OTT Status     OTT Expiry Time
 -----------------------------------------------------------------
 ns-sign             Up             In use         300 (default)

Command Result : 0 (Success)

Now we need to assign TEST partition to NS-SIGN. SSH to HSM-1 and repeat the same from HSM-2:

[hsm-1] lunash:>client assignPartition -client ns-sign -partition TEST

'client assignPartition' successful.

Command Result : 0 (Success)

[hsm-2] lunash:>client assignPartition -client ns-sign -partition TEST

'client assignPartition' successful.

Command Result : 0 (Success)

Finally we need to create HA group and add both HSMs into it. In order to create HA, we need to know the serials of the TEST partition. From NS-SIGN:

% cd /usr/safenet/lunaclient

% ./bin/lunacm
LunaCM v6.2.0-15. Copyright (c) 2006-2015 SafeNet, Inc.

        Available HSMs:

        Slot Id ->              0
        HSM Label ->            TEST
        HSM Serial Number ->    123456789
        HSM Model ->            LunaSA 6.2.0
        HSM Firmware Version -> 6.10.9
        HSM Configuration ->    Luna SA Slot (PED) Signing With Cloning Mode
        HSM Status ->           OK


        Slot Id ->              1
        HSM Label ->            TEST
        HSM Serial Number ->    987654321
        HSM Model ->            LunaSA 6.2.0
        HSM Firmware Version -> 6.10.9
        HSM Configuration ->    Luna SA Slot (PED) Signing With Cloning Mode
        HSM Status ->           OK

        Current Slot Id: 0

lunacm:>hagroup createGroup -label TESTHA -serialNumber 123456789 -password XXXX-XXXX-XXXX-XXXX

Warning:  There are objects currently on the new member.
          Do you wish to propagate these objects within the HA
          group, or remove them?

          Type 'copy' to keep and propagate the existing
          objects, 'remove' to remove them before continuing,
          or 'quit' to stop adding this new group member.
          > copy

        New group with label "TESTHA" created with group number 123456789.
        Group configuration is:

         HA Group Label:  TESTHA
        HA Group Number:  1234567890
       HA Group Slot ID:  Not Available
       Synchronization: enabled
          Group Members:  123456789
             Needs sync:  no
        Standby Members:  

Slot #    Member S/N                      Member Label    Status
======    ==========                      ============    ======
     0     123456789                              TEST     alive

Command Result : No Error

lunacm:>hagroup addMember -group TESTHA -serialNumber 987654321 -password XXXX-XXXX-XXXX-XXXX

Warning:  There are objects currently on the new member.
          Do you wish to propagate these objects within the HA
          group, or remove them?

          Type 'copy' to keep and propagate the existing
          objects, 'remove' to remove them before continuing,
          or 'quit' to stop adding this new group member.
          > copy

        Member 123456789 successfully added to group TESTHA. New group
        configuration is:

         HA Group Label:  TESTHA
        HA Group Number:  1234567890
       HA Group Slot ID:  5
       Synchronization: enabled
          Group Members:  123456789, 987654321
             Needs sync:  no
        Standby Members:  

Slot #    Member S/N                      Member Label    Status
======    ==========                      ============    ======
     0     123456789                              TEST     alive
     1     987654321                              TEST     alive

        Please use the command "ha synchronize" when you are ready
        to replicate data between all members of the HA group.
        (If you have additional members to add, you may wish to wait
        until you have added them before synchronizing to save time by
        avoiding multiple synchronizations.)

Command Result : No Error

lunacm:>exit

Note that XXXX-XXXX is the partition secret. It’s generated during HSM configuration phase (actually, when you create partitions).

The way HA organized with SafeNet might look a bit confusing. There is no clustering per se — both HSMs are actually not aware about each other. So if you place a file on the slot 0 of HSM-1 it will never be synced to HSM-2. HA is configured on the client side instead, so NS-SIGN, once it has something to commit, will sequentially write to a “virtual” slot 5 (labeled TESTHA) which will be propagated to both HSMs. If one of the HSMs is not available at the moment of commit, the client will keep retrying, and if fails, nothing will be committed.

Pay attention on how you label the HA slot — we’ll need it later on for OpenDNSSEC.

You might have also noticed a warning message about objects already present on the partition, so you have a choice to either ‘copy’ or ‘remove’ the content. This is because I have had some data already stored on the TEST partition. Be careful — if you type ‘remove’ all data will be destroyed, so if you share TEST partition with other services/servers like I do always use ‘copy’.

You can have a look at /etc/Chrystoki.conf file, which should be updated with your HSMs and HA slot:

Chrystoki2 = {
   LibUNIX = /usr/lib/libCryptoki2.so;
   LibUNIX64 = /usr/lib/libCryptoki2_64.so;
}

Luna = {
  DefaultTimeOut = 500000;
  PEDTimeout1 = 100000;
  PEDTimeout2 = 200000;
  PEDTimeout3 = 10000;
  KeypairGenTimeOut = 2700000;
  CloningCommandTimeOut = 300000;
  CommandTimeOutPedSet = 720000;
}

CardReader = {
  RemoteCommand = 1;
}

Misc = {
  PE1746Enabled = 0;
   ToolsDir = /usr/safenet/lunaclient/bin;
}
LunaSA Client = {
   ReceiveTimeout = 20000;
   SSLConfigFile = /usr/safenet/lunaclient/bin/openssl.cnf;
   ClientPrivKeyFile = /usr/safenet/lunaclient/cert/client/ns-sign.local.netKey.pem;
   ClientCertFile = /usr/safenet/lunaclient/cert/client/ns-sign.local.net.pem;
   ServerCAFile = /usr/safenet/lunaclient/cert/server/CAFile.pem;
   NetClient = 1;
   TCPKeepAlive = 1;
   ServerName00 = hsm-1.local.net;
   ServerPort00 = 1792;
   ServerHtl00 = 1;
   ServerName01 = hsm-2.local.net;
   ServerPort01 = 1792;
   ServerHtl01 = 1;
}
VirtualToken = {
   VirtualToken00Label = TESTHA;
   VirtualToken00SN = 1234567890;
   VirtualToken00Members = 123456789,987654321;
}
HASynchronize = {
   TESTHA = 1;
}

Another interesting test would be to reboot NS-SIGN and monitor htl status on one of the HSMs. You will notice that the status would change to ‘Grace period’ and back to ‘Up’ once NS-SIGN is up:

[hsm-1] lunash:>htl show

HTL Grace period   :  60 seconds
Default OTT expiry :  300 seconds

 Client Name         HTL Status     OTT Status     OTT Expiry Time
 -----------------------------------------------------------------
 ns-sign             Grace period   In use         300 (default)

Command Result : 0 (Success)

[hsm-1] lunash:>htl show

HTL Grace period   :  60 seconds
Default OTT expiry :  300 seconds

 Client Name         HTL Status     OTT Status     OTT Expiry Time
 -----------------------------------------------------------------
 ns-sign             Up             In use         300 (default)

Command Result : 0 (Success)

By the way, if for some reason you lose HTL link, you can always regenerate ott files on HSMs. You will have to transfer new ott files to NS-SIGN using the same procedure as mentioned above:

[hsm-1] lunash:>htl generateOtt -client ns-sign

To summarize: NS-SIGN is fully registered and ready to use SafeNet Network HSMs. OpenDNSSEC hasn’t been configured yet and is not aware about HSMs — that’s what we’re going to do tomorrow.

Tags: , , , , , , , , , ,

Leave a Reply