Managing ports for multiple FreeBSD servers

This is a follow up post on how to manage ports for multiple FreeBSD servers. If you’re looking for how to update the operating system itself, have a look at my almost three years old post: Managing multiple FreeBSD servers.

Alright, so what we’re trying to solve is this: multiple VMs running the same (or different) release of FreeBSD, and you’re looking for a way to centralize delivery of packages to your FreeBSD VMs.

There are several solutions you can explore:

– You can reuse the same server you use to build world, and expose /usr/ports via NFS. Your VMs will mount /usr/ports, and build the software themselves. Obviously, apart from maintaining one common ports repository you don’t gain much. Moreover, the fundamental issues with this approach are a) you’re going to build software on VMs that might/might not be clean when it comes to dependencies, and you’ll still need to take care of that by using portupgrade for example (which automatically means ruby and friends), and b) you’re going to spend VM resources to build the software which might be an overkill (for example to build lang/gcc5 on a physical server with 8GB of RAM and plenty of cores takes ~1 hour).

– You can abandon the idea of building software from ports completely, and switch to packages (prebuilt binaries) installed from one of the public FreeBSD repositories. This is a fair approach that might suit you. The issue with this approach is that you might want certain ports to be compiled with certain options.

– Or, you can setup and use your own packages repository by using an official FreeBSD tool poudriere, or, tadaa!, Synth developed by John R. Marino. So you’re still going to compile software with whatever options you need, but it will be done in a clean environment, by a standalone (and hopefully powerful enough) server, and what you’re going to get at the end is a bunch of txz files that you can deliver to your VMs however you like (via file/scp/http/…).

Initially, I looked into poudriere, but quickly realized that it’s probably way too much for my environment. Synth, from the other hand, was extremely easy to setup and learn. There were a couple of confusion moments from my side, so in case you’re in the same boat I’ll try to summarize them straight away:

CONFUSION 1: To update my hosts I must install Synth on each and every server, and considering that to run Synth you’ll need lang/gcc6-aux installed it’s not going to be fun. That’s false: you CAN of course install Synth on all VMs, and it will take care of dependencies and updating of packages just fine, but you DON’T HAVE TO since it defeats the purpose. Synth is a tool to build ports (concurrently, and in a clean environment) that will result in a package repository. All you have to do is to expose this repository to your hosts, configure them to use it, and then it’s just a matter of ‘pkg upgrade’. Moreover, the repository doesn’t have to reside on the server where Synth is installed. Once the rebuild of repository is completed, you can copy/sync txz files anywhere you want.

CONFUSION 2: Okay, in this case I must install all packages that I’d like to be maintained on the Synth server otherwise how can I pass specific config options? That’s false too: again, you can do this, but it’s not needed. What you’ll have to do is to run ‘config make’ for a particular port on the Synth server, so the config options are saved to a file which Synth will use (and re-use) to (re-)build the package. The resulting compiled binary doesn’t have to be installed on the Synth server at all.

So, let’s install and configure Synth on a server, and test it with local packages first (I mean packages that are installed on the Synth server itself). Then we will extend it with all other packages installed on VMs.

To install Synth (ccache is a compiler cache — not required but recommended):

  1. % svnlite update /usr/ports
  2. % cd /usr/ports/devel/ccache && make install clean
  3. % cd /usr/ports/ports-mgmt/synth && make install clean

Adjust /root/.ccache/ccache.conf file to fit your environment (I allocated a separate disk for ccache):

  1. max_size = 18.0G
  2. cache_dir = /ccache

Before starting using Synth we need to configure it:

  1. % synth configure
  1. Synth configuration profile: LiveSystem
  2. ===============================================================================
  3.    [A] Ports directory            /usr/ports
  4.    [B] Packages directory         /synth/live_packages
  5.    [C] Distfiles directory        /usr/ports/distfiles
  6.    [D] Port options directory     /var/db/ports
  7.    [E] Build logs directory       /synth/log
  8.    [F] Build base directory       /usr/obj/synth-live
  9.    [G] System root directory      /
  10.    [H] Compiler cache directory   /ccache
  11.    [I] Num. concurrent builders   12
  12.    [J] Max. jobs per builder      5
  13.    [K] Use tmpfs for work area    true
  14.    [L] Use tmpfs for localbase    true
  15.    [M] Display using ncurses      true
  16.    [N] Fetch prebuilt packages    false
  17.  
  18.    [>]   Switch/create profiles
  19.    [RET] Exit

Synth uses profiles, which means you can setup different profiles for different scenarios (like LiveSystem is for the server where Synth is installed, and another profile for building packages for your VMs which might use a different ports tree and etc). In my case, LiveSystem was sufficient. The rest is very much self-explanatory: I used a separate drive for Synth (/synth), and set “Fetch prebuilt packages” to false, meaning that every package will be rebuilt regardless (even if there is a public prebuilt version already).

Let’s execute our first dry-run, which won’t build anything, but only compare installed packages against the ports tree:

  1. % synth status
  1. Querying system about current package installations.
  2. Stand by, comparing installed packages against the ports tree.
  3. These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  4. Total packages that would be built: 0
  5. The complete build list can also be found at:
  6. /tmp/synth_status_results.txt

Since we’ve just executed ‘svnlite update /usr/ports’ prior to building Synth it’s obvious that there is no discrepancy, hence there is nothing to build (yet). Otherwise, if there is something new, you will be presented with:

  1. % svnlite update /usr/ports
  2. % synth status
  1. Scanning existing packages.
  2. These are the ports that would be built ([N]ew, [R]ebuild, [U]pgrade):
  3.   U => editors/nano (2.8.5 => 2.8.6)
  4. Total packages that would be built: 1
  5. The complete build list can also be found at:
  6. /tmp/synth_status_results.txt

To build and automatically install the new package on the Synth server, execute:

  1. % synth upgrade-system

Note that the procedure for building packages for our VMs will be different — we won’t install anything, just build. For the time being we’re just dealing with packages installed on the Synth server (read local).

Let’s collect all packages from all our virtual FreeBSDs in a file, and feed it to Synth. Access your VMs and generate the list of installed packages:

  1. % pkg info -ao | awk '{print $2}' | sort > ports.2.build
  1. % cat ports.2.build
  1. databases/db5
  2. databases/sqlite3
  3. devel/gettext
  4. devel/gmake
  5. devel/icu
  6. devel/libedit
  7. devel/libevent2
  8. devel/libffi
  9. devel/libtool
  10. devel/m4
  11. devel/p5-Locale-gettext
  12. devel/pkgconf
  13. devel/readline
  14. devel/scons
  15. devel/subversion
  16. dns/bind-tools
  17. dns/idnkit
  18. dns/ldns
  19. dns/opendnssec
  20. editors/nano

Copy ports.2.build file to the Synth server (for example into /usr/local/etc/synth/ directory).

Now, before you feed it to Synth, you need to go through each and every port listed in this file, and execute ‘make config’ on the Synth server. If there are no config options for a particular port then there is nothing to do, otherwise either accept default settings, or modify as per your needs. If you can’t remember the options that were used to build a particular port, you can always look at /var/db/ports/ directory — this is the place where configs are saved.

Once you’ve done with ‘make config’s, change your current directory to be outside of /usr/ports, and feed the file to Synth:

  1. % cd /
  2. % synth status /usr/local/etc/synth/ports.2.build

You should see that most of the ports will be reported as [N]ew. Let’s build them:

  1. % synth just-build /usr/local/etc/synth/ports.2.build

Multiple instances of builders will be launched with periodical status updates. If you want to interrupt the build process don’t use Ctrl-C, but rather Ctrl-Q.

If you encounter that some ports failed to build, check whether you did ‘make config’ for that port, otherwise, look at the Synth logs (which are located under /synth/log in our example).

Once the build is completed, we need to regenerate the local repository:

  1. % synth rebuild-repository

At this stage, we’re ready to expose the content of /synth/live_packages/ to our VMs. How you expose it is really up to you. I simply rsync /synth/live_packages/ to another web server once ‘synth just-build/rebuild-repository’ are finished.

To add your custom package repository configure the following on all your clients:

  1. % mkdir -p /usr/local/etc/pkg/repos
  2. % cat /usr/local/etc/pkg/repos/FreeBSD.conf
  1. FreeBSD: {
  2.   enabled: no
  3. }
  4.  
  5. Synth: {
  6.   url: "pkg+http://your.local.web.server/",
  7.   mirror_type: "srv",
  8.   signature_type: "none",
  9.   fingerprints: "/usr/share/keys/pkg",
  10.   enabled: yes
  11. }

Next time you run ‘pkg upgrade’ you will be served by your shiny repository!

Tags: , , ,

One Response to “Managing ports for multiple FreeBSD servers”

  1. Thanks for your great introduction and receipe. I’ve tried poudriere but it’s just too much of a hassle compared to synth. At least for my purposes. You’ve saved me a lot of time.

Leave a Reply