Managing multiple FreeBSD servers

If you run multiple installations of FreeBSD sooner or later you will face with the issue of how to update them all in the most efficient and centralized way. Building kernel/world for a FreeBSD server with one CPU and couple of GB of RAM will take hours to complete. Fortunately, there is a way to optimize it.

In short: you build kernel/world on a powerful server and then expose /usr/src and /usr/obj via NFS for others to mount and install. In such case the time spent on upgrade is greatly reduced.

For example, in my case all FreeBSD servers are virtual. They use the same architecture (amd64), run the same version of FreeBSD (10+) and the same customized kernel. There is a separate, physical server (let’s call it the Master) with plenty of CPU and fast hard drives, where I build kernel and world for virtual FreeBSD servers (let’s call them Slaves). The master acts as a NFS server with /usr/src and /usr/obj exposed to the slaves. When I update the slave, I boot it to the single-user mode, mount /usr/src and /usr/obj from the master, and simply installkernel/installworld.

Note: it’s quite important to note that /etc/src.conf and /etc/make.conf files on the master must be identical to the one’s on the slaves.

Things to be done on the master:

Configure the master to act as a NFS server. Here is a relevant part from /etc/rc.conf file:

  1. # nfs stuff
  2. rpcbind_enable="YES"
  3. nfs_server_enable="YES"
  4. nfs_server_flags="-t"
  5. mountd_enable="YES"
  6. mountd_flags="-r -p 957"

Note: my NFS server is firewalled so I fix the mountd port to 957/tcp, otherwise it uses a random port after each restart.

Here is how /etc/exports file looks like:

  1. /usr/src /usr/ports /usr/obj    -maproot=root           192.168.x.x 192.168.y.y

Note: 192.168.x.x/192.168.y.y are IP addresses of the slaves. Execute ‘service mountd onereload’ every time you modify /etc/exports file.

Update source files, build and install kernel/world on the master using the standard procedure. Once the master is up to date, get a copy of the kernel installed on the slave, copy it to the /usr/src/sys/arch/conf directory on the master, build it but do not install:

  1. % cd /usr/src
  2. % make -j4 buildkernel KERNCONF=SLAVE01

Where SLAVE01 is the kernel name from the slave.

Things to be done on the slave:

Drop the slave into single-user mode:

  1. % mount -u /
  2. % mount -a -t ufs
  3. % swapon -a
  4. % adjkerntz -i

Start the network:

  1. % /etc/rc.d/netif start
  2. % /etc/rc.d/routing start

Mount /usr/src and /usr/obj directories:

  1. % mount ip.of.the.master:/usr/src /usr/src
  2. % mount ip.of.the.master:/usr/obj /usr/obj

Install the kernel, run mergemaster and installworld:

  1. % cd /usr/src
  3. % make installkernel KERNCONF=SLAVE01
  4. % mergemaster -p
  5. % make installworld
  6. % mergemaster -iF
  7. % make delete-old

Reboot the slave.

If you update between major versions (like 9 to 10, or 10 to 11) you have to rebuild all ports prior to deleting old libraries. Here is how you do it with portupgrade:

  1. % mount ip.of.the.master:/usr/ports /usr/ports
  2. % portupgrade -f pkg
  3. % portupgrade -af

Finally, clean up old libraries:

  1. % mount ip.of.the.master:/usr/src /usr/src
  2. % cd /usr/src
  3. % make delete-old-libs

Tags: , , ,

One Response to “Managing multiple FreeBSD servers”

  1. […] This is a follow up post on how to manage ports for multiple FreeBSD servers. If you’re looking for how to update the operating system itself, have a look at my almost three years old post: Managing multiple FreeBSD servers. […]

Leave a Reply