Host Your Own Services With FreeBSD: Poudriere

FreeBSD offers two basic technologies for installing third-party applications. The easiest way is installing precompiled packages from official FreeBSD's package repositories. Somewhat more complicated, but more flexible way is compiling and installing applications using ports. Main shortcoming of official precompiled packages is the fact that they are being built with default set of options, whereas advanced setups often need to enable non-default option (e.g. LDAP authentication), change target database support (e.g. PostgreSQL instead of MySQL), or to disable undesirable default option (e.g. X11 support on headless servers). Luckily, there's additional way which combines flexibility of ports with simplicity of packages provided by Poudriere. The following article will help you set your own FreeBSD package repository where you will be able to compile your own packages from ports, and serve them to your other FreeBSD boxes. In a jail, of course!

jail.conf Tweaking

Previous article in this series, Initial Jail Setup, gives instruction about setting up a jail. Now it's time to put it to good use - compiling packages for many more jails that are about to be set up in the future, using Poudriere. Poudriere uses jails to build packages, which means it will spawn additional hierarchical jails. Our current jail.conf on jailhost does not permit this, which is why we need to edit it as follows:

# jailhost.example.org:/etc/jail.conf

path          = "/usr/jail/${host.hostname}";
exec.start    = "/bin/sh /etc/rc";
exec.stop     = "/bin/sh /etc/rc.shutdown";
exec.clean;
mount.devfs;

pkg_example_org {
  host.hostname    = pkg.example.org;
  host.domainname  = example.org;
  ip4.addr         = 'lo1|127.0.1.11/32';
  ip4.addr        += 'em0|192.0.2.11/32';

  ip4.addr        += 'lo0|127.0.0.1/32';
  children.max     = 20;
  enforce_statfs   = 1; 
  sysvshm          = new;
  sysvsem          = new;
  persist;
  allow.chflags;
  allow.mount.devfs;
  allow.mount.procfs;
  allow.mount;
  allow.mount.devfs;
  allow.mount.procfs;
  allow.mount.nullfs;
  allow.mount.tmpfs;
  allow.socket_af;
}

In first article of this series, Initial Jail Setup, I wrote that our jails will have their own loopback interfaces, in order to prevent them from communicating directly over jail host's loopback. This jail, pkg.example.org will be exception to the rule, because poudriere currently requires access to lo0 interface on 127.0.0.1.

Install Required Packages

After jail restart, we can ssh into it and install some packages from default official FreeBSD quarterly package repository.

Bootstrap pkg first:

pkg

Then install required packages:

pkg install \
  databases/memcached \
  devel/ccache-memcached-static \
  ports-mgmt/poudriere \
  sysutils/screen \
  www/apache24

Poudriere Configuration

We need to edit poudriere.conf as follows:

# pkg.example.org:/usr/local/etc/poudriere/poudriere.conf

NO_ZFS=yes
FREEBSD_HOST=https://download.FreeBSD.org
RESOLV_CONF=/etc/resolv.conf
BASEFS=/usr/local/poudriere
USE_PORTLINT=no
USE_TMPFS=yes
DISTFILES_CACHE=$BASEFS/distfiles
PKG_REPO_SIGNING_KEY=/etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem
CCACHE_DIR=$BASEFS/ccache
CCACHE_STATIC_PREFIX=/usr/local
CCACHE_DIR_NON_ROOT_SAFE=yes
RESTRICT_NETWORKING=no
PARALLEL_JOBS=2
PREPARE_PARALLEL_JOBS=6
NOLINUX=yes
ALLOW_MAKE_JOBS=yes
URL_BASE=http://pkg.example.org
MAX_EXECUTION_TIME=172800
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=5
BUILDER_HOSTNAME=pkg.example.org
BUILD_AS_NON_ROOT=yes
PORTBUILD_USER=nobody
HTML_TRACK_REMAINING=yes

Poudriere Jail Management

We are now ready to create our poudriere jail:

poudriere jail -j 12_2:x86:64 -c -v 12.2-RELEASE

If something went bad, we can start from scratch after deleting the jail:

poudriere jail -j 12_2:x86:64 -d

Poudriere Ports Tree Management

Next thing to do would be to fetch current snapshot of ports tree:

poudriere ports -c

If something went bad, we can start from scratch after deleting it:

poudriere ports -d

After initial ports tree creation, consequent updates are done with:

poudriere ports -u

Tweaking Port Options

Poudriere has the ability to build multiple sets of ports. I find it useful for having separate sets for headless servers, whose ports don't need X11 port options, and workstations who do. Options are being configured by specifically named config files which are actually make.conf(5), where parts of their filenames have significance for poudriere.

In order for options to be applied to ports built in poudriere jail named 12_2:x86:64, from default ports tree, for server set, a file named 12_2:x86:64-default-server-make.conf should be placed under /usr/local/etc/poudriere.d/. Here is how ours look like:

# pkg.example.org:/usr/local/etc/poudriere.d/12_2:x86:64-default-server-make.conf

MAKE_JOBS_NUMBER= 6
OPTIONS_UNSET+= CUPS DOCS X11
devel_apr1_SET+= LDAP
www_apache24_SET+= AUTHNZ_LDAP LDAP

MAKE_JOBS_NUMBER instructs poudriere to use up to 6 CPU cores simultaneously to build single port. Product of MAKE_JOBS_NUMBER and PARALLEL_JOBS shouldn't exceed nuber of CPU cores on jail host. Building ports is RAM and CPU intensive process, so we need to take care not to exhaust all the resources needed for other jails that will reside on same jail host.. Any options we want to enable or disable globally for all the ports should be declared in OPTIONS_SET and OPTIONS_UNSET lines, respectively. Any options we want to enable or disable for specific port should be declared in category_port_SET and category_port_UNSET lines, respectively. Specific port options declared after global options will override global options.

Creating Required Folders and Files

We need to create folder where distfiles (source tarballs) are going to be stored, as declared earlier in poudriere.conf, as well as folder to store our lists of ports to be built:

mkdir /usr/local/poudriere/distfiles
mkdir /usr/local/etc/poudriere.d/pkglists/

Inside this folder we will create file named server containing port names in category/port origin format:

# pkg.example.org:/usr/local/etc/poudriere.d/pkglists/server

ports-mgmt/pkg
www/apache24

We need to create directory for storing private key to sign our packages, as well as public certificate which will need to be imported on all the hosts which will use this repository. We will also set appropriate permissions - private key should be readable only to root, while public certificate should be world-readable.

mkdir -p /etc/ssl/selfsigned/example.org/poudriere/
openssl genrsa -out /etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem 4096
openssl rsa -in /etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem \
  -pubout -out /etc/ssl/selfsigned/example.org/poudriere/poudriere-cert.pem
chmod 400 /etc/ssl/selfsigned/example.org/poudriere/poudriere-privkey.pem
chmod 444 /etc/ssl/selfsigned/example.org/poudriere/poudriere-cert.pem

Finally, we need to create ccache directory in order to cache compilation results and hopefully speed up consequent builds:

mkdir /usr/local/poudriere/ccache
chown nobody:nobody /usr/local/poudriere/ccache

A Few Words About screen

I prefer to build packages from screen session which gives me the ability to detach from console and leave build job running, as opposed to standard ssh session where closing session kills the job running inside of it. I always prefer to start screen as unprivileged user and get root with su once inside, not the other way around - get root with su and start screen. Actually I prefer to use sudo but we haven't installed and configured it so far - we will do it later in the series.

A short digression for crash course in screen:

• screen session is started with screen command without any additional arguments.
• Detaching from screen session is done by pressing first ctrl+a followed by d.
• Listing screen sessions belonging to currently logged user is done by screen -l. Sessions will be listed as attached or detached.
• If there's only one deteched screen we can attach to it with screen -r.
• If there are multiple detached screen sessions we can attach to each one with screen -r <id>.
• If we need to attach to already attached screen session (e.g. when we remained attached on another computer), we can do it with screen -d -r <id>. This will detach session that was previously attached.

Bulk Building

I like to symlink my pkglists to root's home dir so I don't need to type their full path.

ln -s /usr/local/etc/poudriere.d/pkglists/server ~/server

We can now start our first bulk build:

poudriere bulk -j 12_2:x86:64 -z server -f ~/server

Setting Up Apache Web Server

While packages are getting built, we need to set up Apache web server so we can monitor building process and check build logs in nice web inteface. Also, other FreeBSD hosts will acces package repository over http.

We need a few config files for apache:

# pkg.example.org:/usr/local/etc/apache24/httpd.conf

ServerRoot "/usr/local"
Listen pkg.example.org:80

LoadModule authn_file_module libexec/apache24/mod_authn_file.so
LoadModule authn_core_module libexec/apache24/mod_authn_core.so
LoadModule authz_host_module libexec/apache24/mod_authz_host.so
LoadModule authz_groupfile_module libexec/apache24/mod_authz_groupfile.so
LoadModule authz_user_module libexec/apache24/mod_authz_user.so
LoadModule authz_core_module libexec/apache24/mod_authz_core.so
LoadModule access_compat_module libexec/apache24/mod_access_compat.so
LoadModule auth_basic_module libexec/apache24/mod_auth_basic.so
LoadModule reqtimeout_module libexec/apache24/mod_reqtimeout.so
LoadModule filter_module libexec/apache24/mod_filter.so
LoadModule mime_module libexec/apache24/mod_mime.so
LoadModule log_config_module libexec/apache24/mod_log_config.so
LoadModule env_module libexec/apache24/mod_env.so
LoadModule headers_module libexec/apache24/mod_headers.so
LoadModule setenvif_module libexec/apache24/mod_setenvif.so
LoadModule version_module libexec/apache24/mod_version.so
LoadModule unixd_module libexec/apache24/mod_unixd.so
LoadModule status_module libexec/apache24/mod_status.so
LoadModule autoindex_module libexec/apache24/mod_autoindex.so
LoadModule dir_module libexec/apache24/mod_dir.so
LoadModule alias_module libexec/apache24/mod_alias.so
LoadModule mpm_event_module libexec/apache24/mod_mpm_event.so

IncludeOptional etc/apache24/modules.d/[0-9][0-9][0-9]_*.conf

<IfModule unixd_module>
  User www
  Group www
</IfModule>

ServerAdmin webmaster@example.org
ServerName pkg.example.org

<Directory />
  AllowOverride none
  Require all denied
</Directory>

<Files ".ht*">
  Require all denied
</Files>

ErrorLog "/var/log/httpd-error.log"
LogLevel warn

<IfModule log_config_module>
  LogFormat "%v %h %t \"%r\" %>s %b" vhost_common
  CustomLog "/var/log/httpd-access.log" vhost_common
</IfModule>

<IfModule headers_module>
  RequestHeader unset Proxy early
</IfModule>

<IfModule mime_module>
  TypesConfig etc/apache24/mime.types
  AddType application/x-compress .Z
  AddType application/x-gzip .gz .tgz
</IfModule>

Include etc/apache24/extra/httpd-mpm.conf
Include etc/apache24/extra/httpd-vhosts.conf
Include etc/apache24/extra/httpd-default.conf
Include etc/apache24/Includes/*.conf

# pkg.example.org:/usr/local/etc/apache24/extra/httpd-mpm.conf

<IfModule !mpm_netware_module>
  PidFile "/var/run/httpd.pid"
</IfModule>
<IfModule mpm_event_module>
  StartServers             3
  MinSpareThreads         75
  MaxSpareThreads        250
  ThreadsPerChild         25
  MaxRequestWorkers      400
  MaxConnectionsPerChild   0
</IfModule>
<IfModule !mpm_netware_module>
  MaxMemFree            2048
</IfModule>

# pkg.example.org:/usr/local/etc/apache24/extra/httpd-defaut.conf

Timeout 60
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 5
UseCanonicalName Off
AccessFileName .htaccess
ServerTokens Prod
ServerSignature Off
HostnameLookups Off
<IfModule reqtimeout_module>
  RequestReadTimeout header=20-40,MinRate=500 body=20,MinRate=500
</IfModule>

# pkg.example.org:/usr/local/etc/apache24/extra/httpd-vhosts.conf

<VirtualHost _default_:80>
  ServerName pkg.example.org
  ServerAdmin webmaster@example.org
  <IfModule dir_module>
    DirectoryIndex index.html
  </IfModule>
  DocumentRoot "/usr/local/share/poudriere/html"
  <Directory "/usr/local/share/poudriere/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
  </Directory>
  Alias /data "/usr/local/poudriere/data/logs/bulk"
  <Directory "/usr/local/poudriere/data/logs/bulk">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
  </Directory>
  Alias /packages "/usr/local/poudriere/data/packages"
  <Directory "/usr/local/poudriere/data/packages">
    Options Indexes FollowSymLinks
    IndexOptions FancyIndexing FoldersFirst SuppressDescription \
                 SuppressColumnSorting NameWidth=50 VersionSort
    AllowOverride None
    Require all granted
  </Directory>
</VirtualHost>

Once all above config files are in place, we can set apache to start at boot time and start it:

sysrc apache24_enable=YES
service apache24 start

Poudriere web interface is now accessible at web URL we configured in Apache.

Changing pkg Repository On Clients

By default, FreeBSD is configured to use official quarterly repo for installing packages by means of /etc/pkg/FreeBSD.conf. In order to switch to our repo, we won't be editing this file directly, but overriding it in another config file, as instructed in comments of /etc/pkg/FreeBSD.conf.

First, we need to disable official quarterly repo:

mkdir -p /usr/local/etc/pkg/repos
echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf

Next, we need to create our config file which points to our repo:

# pkg.kappastar.com:/usr/local/etc/pkg/repos/example.org

example: {
        url: "http://pkg.example.org/packages/12_2:x86:64-default-server/",
        mirror_type: "http",
        signature_type: "pubkey",
        pubkey: "/etc/ssl/selfsigned/example.org/poudriere/poudriere-cert.pem",
        enabled: yes
}

A Few Words About Transport Security

One could reasonably ask how come we serve our repositories over http, an insecure protocol, and not over https which is supposed to be more secure. I see three reasons:

• We have cryptographically signed packages with poudriere-privkey.pem, and configured clients with poudriere-cert.pem. Thanks to this, clients would refuse to install packages if they were altered in transit.
• With above authenticity mechanism in place, it would be redundant, but also quite time-consuming, to create self-signed keys and certificates for apache and to configure clients to trust them.
• Even with globally trusted SSL certificates, until 12.2, FreeBSD did not trust any root CAs upon basic install. In order to access anything over SSL one would need to install third party package - security/ca_root_nss - Root certificate bundle from the Mozilla Project. Requiring package installation for accessing repo which provides that same packages creates chicken and egg problem.

Wait, there's fourth reason. I had the pleasure to see Marc Espie of OpenBSD fame presenting his talk titled Advances in OpenBSD packages: https is a lie in person on EuroBSDcon 2018 in Bucharest, Romania. I admit I didn't understand all the nifty details, but what I did understand is that, talking about package management, SSL alone can't save us from bad guys, that additional protection mechanisms need to be implemented, and that SSL can sometimes bring more harm than good.

Increasing ccache Disk Space

Remember we have configured our poudriere to make use of ccache in order to speed up builds by caching previous compilations and detecting when the same compilation is being done again? The thing is, default size of cache is just 5Gb, which will get filled up quite soon once we start to build a lot of packages, and we will lose any benefits because of overwrites. In order to increase cache size we need to modify ccache.conf:

# pkg.example.org:/usr/local/poudriere/ccache/ccache.conf

max_size = 20.0G

RAM Based ccache Using memchached

Things can get even faster if you have plenty of RAM, as memcached can use RAM to cache compilation instead of disk. We have already istalled it at the beginning of the article, we just need to set it to start at boot time with appropriate flags. After that, we can immediately start it:

sysrc memcached_enable=YES
sysrc memcached_flags="-l 127.0.1.11 -m 20480"
service memcached start

We also need to replace contents of ccache.conf with the following:

# pkg.example.org:/usr/local/poudriere/ccache/ccache.conf

memcached_conf = --SERVER=127.0.1.11:11211
memcached_only = true

Consequent builds should now be even faster.

Conclusion

Above article hopefully taught you all the basics needed for building your own ports from packages and serving them to other FreeBSD systems.

If you have any additional questions, comments or any kind of feedback, drop me a line.