[freebsd banner]

Abstract

This article explains how to install a FreeBSD 11.1 system on a remote server known as a dedibox. It is closely related to my previous HOWTO on 9.2 but with several changes due to FreeBSD 11.1 and the way I want the system to boot (avoiding the pitfalls of broken iDRAC/iLO management systems).

!! Work In Progress !!

Table of content

  1. Abstract
  2. Table of content
  3. Prerequisites
  4. Installed system
  5. Hardware
  6. Notes on ZFS, disks and all that
  7. Constraints
  8. Custom mfsbsd images
  9. Actual FreeBSD installation
  10. Finishing up
  11. Second stage booting
  12. Things to remember
  13. Jails
  14. SoloBSD/HardenedBSD Alternative
  15. Resources
  16. Feedback
  17. History
  18. Footnotes

Prerequisites

Like many dedicated servers in an hosting datacenter, access to a console (such as iLO or iDRAC) is mandatory to be able to manipulate BIOS settings, access to some kind of rescue mode where you can upload an ISO image and boot from it. The example used here will be the Dedibox rescue system (such as described there and used in this howto)

You must have an mfsbsd generated image or the ability to generate one (which means an entire /usr/src tree or the files from a release). See the above mfsbsd URL or below for details.

Installed system

Hardware

The hardware I have now is an ST8 dedicated server (See there for reference), a rather powerful system with the following characteristics:

An earlier version of this document was talking about an HP DL120G7 system and its crappy P410 controller. I’m now happy to report the new machine is back to be a Dell one without that issue. Got two times as much memory & disk, not too bad for a 50€/mo machine. CPU is slightly faster too (index on CPU Benchmarks is 6665 compared to the 3837 of the previous L3426 CPU.)

Notes on ZFS, disks and all that

Please go read this article to find useful information on ZFS, disks and how to use it. It is not specific to FreeBSD ZFS but will apply as well for most things.

Constraints

The fact that we want to use encryption to protect our data is a major constraint on the on-disk architecture and how we lay down/use partitions. The main mfsbsd will not be encrypted but we only load the kernel from it and it is an image, we do not intent to use anything outside of sshd.

Custom mfsbsd images

If you choose to build your own mfsbsd images (to add a missing driver or equivalent, please see this tutorial (which was at the beginning inside this howto but it makes sense to separate the two).

The regular mfsbsd generated from 11.1 should have more things than the one I used before and my patch to add the missing modules has been merged.

Installation of the mfsbsd image

Put the dedibox server in “rescue mode”, FreeBSD is now available which helps. All the setup can be done through this.

There was a time when you needed to have a specially build mfsbsd image because of some missing modules but it is over now, my changes have been incorporated :)

Actual FreeBSD installation

We will more or less follow the instructions in there. Things that we will change are not essential but reflect our special requirements.

NOTE: in recent versions of mfsbsd (the so-called Special Edition named -se), there is a script called zfsinstall that will do most of what is explained there. The reason we are not using it is that it does not support encrypted partitions at all. If you don’t need that, just use the script.

Since 10.1, the bsdinstall script in the distribution is able to create most of what we need including encrypted swap and pool.

Partitioning the drives

The mfsbsd image already contains a GPT partition table so we will copy it with dd(1) over the raw disk then after booting, we will add out swap and zfs partitions.

NOTE: the disks in this new machine are attached to the on-board SATA controller, not on another HBA like the P410 in HP machines or the LSI2009 card in older Dell machines. This means the devices are named adaNN and not daNN.

As we will be using the two disks in mirror mode, we will be replicating the commands we do on ada0 on ada1. That way, if either disk is broken at some point, the system will be able to find all the information it needs to boot.

Before we install our own GPT partition table, we must wipe out the previous partition table installed by the Dedibox installation system.

dd if=/dev/zero of=/dev/ada0 bs=512 count=10
dd if=/dev/zero of=/dev/ada1 bs=512 count=10

In the previous HOWTO, we had a small freebsd-zfs slice for hosting the boot code, then swap and the first, unencrypted pool. Now that we are going to use mfsbsd for booting and running sshd, we need a bigger UFS partition (mfsbsd image mode is UFS below), then the swap as before then a single, encrypted pool. The mfsbsd partition will have only the image and is very easy to regenerate. No need to encrypt that one which means that we can boot (or in case of a power failure) from that one w/o issue.

Now I’m just mirroring the GPT partition table from the mfsbsd image then copying each partition from the image into the disk.

The main security issue is that the kernel mfsbsd will be the one running, not anything from the encrypted pool. Checking the image content after the second stage boot could be interesting to setup.

As we will be also encrypting the swap (it makes no sense encrypting the data and not the swap as well) and also mirror it for safety reasons, swap will be twice the RAM (32 GB in our case).

# mdconfig -f mfsbsd-se-11.0-RELEASE-amd64.img -u 1

/dev/md1 is now the “device” version of the image. It has two partitions in a GPT table and we want to reproduce that:

# gpart show md1
=>    40  423920  md1  GPT  (207M)
      40     472    1  freebsd-boot  (236K)
     512  423448    2  freebsd-ufs  (207M)

NOTE: there is another utility we could use instead of mdconfig(8), called ggatel(8). It uses the geom(4) framework. Not sure this would be very useful there though.

I’ll begin by dd(1) the mfsbsd image onto the disk and fixing a few thing. We also need to “unlock” the first sector of the disk, usually protected. In the previous HOWTO back in the 9.1 days, mfsbsd was probably MBR-based and what follow was not needed.

# sysctl kern.geom.debugflags=16
kern.geom.debugflags: 16

# fetch http://mfsbsd.vx.sk/files/images/11/mfsbsd-se-11.0-RELEASE-amd64.img
# dd if=mfsbsd-se-11.0-RELEASE-amd64.img of=/dev/ada0

# gpart show ada0
=>    40  423920  ada0  GPT  (1.8T) [CORRUPT]
      40     472     1  freebsd-boot  (236K)
     512  423448     2  freebsd-ufs  (207M)

Fixing:

gpart recover ada0
gpart show ada0
=>        40  3906963552  ada0  GPT  (1.8T)
          40         472     1  freebsd-boot  (236K)
         512      423448     2  freebsd-ufs  (207M)
      423960  3906539632       - free -  (1.8T)

Including the “pseudo-MBR” entry:

# fdisk -a -1 ada0
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 3906963631 (1907697 Meg), flag 0
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
Do you want to change the active partition? [n] y
Supply a decimal value for "active partition" [1]
Are you happy with this choice [n] y

We haven't changed the partition table yet.  This is your last chance.
parameters extracted from in-core disklabel are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Information from DOS bootblock is:
1: sysid 238 (0xee),(EFI GPT)
    start 1, size 3906963631 (1907697 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
2: <UNUSED>
3: <UNUSED>
4: <UNUSED>
Should we write new partition table? [n] y

Now we are able to reboot and log into the machine!

294 [2:38] roberto@lonrach> slogin -v root@newmachine
FreeBSD 11.0-RELEASE-p1 (GENERIC) #0 r306420: Thu Sep 29 01:43:23 UTC 2016

Welcome to mfsBSD SE, the memory based FreeBSD distribution.
This is a special version intended for full-ZFS install of FreeBSD

To make a full-ZFS FreeBSD install from this ISO:

1. Mount the CD device
( e.g. mount_cd9660 /dev/cd0 /cdrom )
2. Run "zfsinstall" with path to release directory and your drive
( e.g. zfsinstall -d ada0 -u /cdrom/9.2-RELEASE-amd64 )

Run zfsinstall with the -h flag for help or without flags for options.
I recommend creating a GPT swap partition (e.g. -s 2G for a 2GB swap).

Feel free to email me with any bug reports or feature suggestions.
Martin Matuska <mm@FreeBSD.org>
http://mfsbsd.vx.sk/
root@mfsbsd:~ #

Here is the disk layout after booting that initial mfsbsd boot.

# gpart show ada0
=>        40  3906963552  ada0  GPT  (1.8T)
          40         472     1  freebsd-boot  (236K)
         512      423448     2  freebsd-ufs  [bootme]  (207M)
      423960  3906539632       - free -  (1.8T)

This layout come from the image itself as we have seen above. Everytime you update the mfsbsd image, you will need either to “fix” the partition table with gpart recoveras we did before or update the first two partitions by hand. The latter may seem more complicated but it does not force you to rewrite the partition table everytime.

An issue you want to look for is that now that we have really big hard drives (2 and 3 TB now and soon more), these have been getting 4 KB sectors and run really slowly in 512-bytes sectors so you want your partitions aligned on 4 KB boundaries so, from now on, I will be adding “-a 4k” to the gpart(8) command lines:

To allow for a future larger mfsbsd image to be installed (think FreeBSD 12 and later), we will create the swap and ZFS slices further on the disk. Lets take 1G as the boundary.

We create the disk layout on the rest of ada0:

gpart add -b 1G -s 32G -a 4k -t freebsd-swap -l swap1 ada0
gpart add -a 4k -t freebsd-zfs -l tank0 ada0

We will now create the same environment on the second disk:

dd if=mfsbsd-se-11.0-RELEASE-amd64.img of=/dev/ada1
gpart show ada1
=>    40  423920  ada0  GPT  (1.8T) [CORRUPT]
      40     472     1  freebsd-boot  (236K)
     512  423448     2  freebsd-ufs  (207M)

Fixing:

gpart recover ada1
gpart show ada1
=>        40  3906963552  ada1  GPT  (1.8T)
          40         472     1  freebsd-boot  (236K)
         512      423448     2  freebsd-ufs  (207M)
      423960  3906539632       - free -  (1.8T)

Including the “pseudo-MBR” entry:

# fdisk -a -1 ada1
******* Working on device /dev/ada1 *******
parameters extracted from in-core disklabel are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 3906963631 (1907697 Meg), flag 0
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
Do you want to change the active partition? [n] y
Supply a decimal value for "active partition" [1]
Are you happy with this choice [n] y

We haven't changed the partition table yet.  This is your last chance.
parameters extracted from in-core disklabel are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=243197 heads=255 sectors/track=63 (16065 blks/cyl)

Information from DOS bootblock is:
1: sysid 238 (0xee),(EFI GPT)
    start 1, size 3906963631 (1907697 Meg), flag 80 (active)
	beg: cyl 0/ head 0/ sector 2;
	end: cyl 1023/ head 255/ sector 63
2: <UNUSED>
3: <UNUSED>
4: <UNUSED>
Should we write new partition table? [n] y

gpart add -b 1G -s 32G -a 4k -t freebsd-swap -l swap1 ada1
gpart add -a 4k -t freebsd-zfs -l tank1 ada1

We can just do the alignment part because we are using geli(8) on the disks below ZFS, geli use 4k sectors anyway for ZFS will pick that up at zpool creation time and set the right ashift value (12 in this case). For plain disks zpool, you will have to use the gnop(8) trick to present 4k sectors to ZFS regardless of the actual sector size.

It is probably only needed on the first call as the rest will be aligned due to their sizes. You may also encounter smaller drives that protest when the partitions are not aligned. In these cases (recently seen on a Seagate Barracuda 500 GB), you can specify -b 40 which will give a 4k-aligned partition (128+40 = 168 which is dividable by 4 instead of 34+128 = 162).

We mirror that configuration on ada1:

You can check that the different partitions and labels do now exist in /dev/gpt:

# ls /dev/gpt
boot0	boot1	swap0	swap1	tank0	tank1

As we want to be able to boot from either disk, we mark the 2nd partition as a boot candidate:

gpart set -a bootme -i 2 ada0
gpart set -a bootme -i 2 ada1

You should end up with something like this:

# gpart show
=>        34  3907029101  ada0  GPT  (1.8T)
          34         128     1  freebsd-boot  (236K)
         162     4194304     2  freebsd-ufs  [bootme]  (512M)
     4194466    67108864     3  freebsd-swap  (32G)
    71303330  3835725805     4  freebsd-zfs  (1.8T)

=>        34  3907029101  ada1  GPT  (1.8T)
          34         128     1  freebsd-boot  (236K)
         162     4194304     2  freebsd-ufs  [bootme]  (512M)
     4194466    67108864     3  freebsd-swap  (32G)
    71303330  3835725805     4  freebsd-zfs  (1.8T)

We will load the bootcode in place on both disks, remember that if the first drive fail, you want to be able to boot on the second one. Note again that we are using gptboot not gptzfsboot like in the previous document. Our system there is a UFS/ZFS setup not a full-ZFS anymore. Unless mfsbsd concert itself in generating a ZFS image, this is the way to go.

gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1

Encrypting the disks

Now, you have to choose a passphrase (as usual, not too short, not guessable, remember you need it only at boot-time). You could choose a different passphrase for each disk but I’d not recommend it because that would be giving two cipher-texts for the same clear-text (as the two partitions are going to be mirrored thus contain the exact same data).

geli init -s 4096 -l 256 /dev/gpt/tank0
geli init -s 4096 -l 256 /dev/gpt/tank1

If you recall the previous document, we did specify the -K option with a file name for the key. It is easier to just have a passphrase so no more -K. Another difference is that the key will not be needed at boot time so do not put the -b option.

You will find backups of the metadata in /var/backups, it is probably a good idea not to forget to copy them elsewhere for security just in case.

Attach both drives:

geli attach /dev/gpt/tank0
geli attach /dev/gpt/tank1

NOTE: remember that we did load the aesni kernel module? If it is used, the above commands should display something about “hardware” crypto being used for AES-XTS mode like the following (run dmesg(8)):

cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
GEOM_ELI: Device gpt/tank0.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device gpt/tank1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: hardware

Pool creation

We will also use a different way to create/mount the datasets to make the last part of the install (switching mountpoints) much easier.

Now we create the encrypted and mirrored ZFS partition:

zpool create -o altroot=/mnt -O mountpoint=none tank mirror gpt/tank0.eli gpt/tank1.eli

NOTE: if you used a regular distribution boot disk (like -dvd1) instead of a mfsbsd one, you will find that /boot is read-only meaning that you will not be able to create a proper /boot/zfs/zpool.cache file. In this case, use the -o cachefile=/tmp/zpool.cache at zpool creation. You will move this file in its proper place before reboot.

With recent versions of ZFS, the zpool.cache file is now optional anyway.

The pool should be appearing like this:

NAME    SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
tank   1.78T  1.76G  1.78T     0%  1.00x  ONLINE  -

When we will have created some filesystems on the disks, we will set the bootfs property on the pool. I use a separate root filesystem on my pool, it makes changing the / fs much more simple and allow to have different ones.

Now that the pool has been created, we switch the algorithm used to checksum disk blocks. I used to recommend “fletcher4” which is only slightly slower but better (just like CRC16 vs CRC32) but 11.0 has incorporated new algorithms which are both faster and better crypto-wise: SHA-256 & SHA-512 (the latter being 50% faster on 64-bit arch than the former) and the new edon-R and Skein ones. See the description here.

zfs set checksum=skein tank

I choose Skein over Edon-R because while they were both SHA-3 candidates, the latter was deemed less secure very early in the competition while Skein was a finalist.

Encrypted swap

Swap is slightly different, we will use the “onetime” command to geli(8) through the way we declare swap in /etc/fstab, that way we do not need to enter any passphrase because it is not needed to know it after attaching the partitions (see geli(8) for details).

As I said earlier, we will use encrypted swap and geli has automatic setup for that by adding the .eli suffix in /etc/fstab. So let’s create the gmirror configuration for swap.

gmirror label swap gpt/swap0 gpt/swap1

The /etc/fstab entry will look like the following:

/dev/mirror/swap.eli	none swap sw 0 0

In this schema, we do not encrypt each swap partition but the mirror itself (swap over geli over gmirror) to avoid doing things twice.

Boot Environments (BE)

There is an interesting program called beadm, directly inspired by the Illumos utility of the same name that will be very handy to manage multiple versions of the OS (called boot environments) and upgrades. Now that we are not using the dual-pool setup anymore, BE (Boot Environments) ought to be usable now. We will just have to be extra careful to keep everything in the unencrypted mfsbsd image in sync.

What we will do is create our datasets according to the naming scheme of beadm to ease migration to BE later.

Dan Langille has written some articles about beadm, you may want to have a look at these.

Filesystems

Compression seems to create issues for kernel loading so we will avoid that on tank/root. All other FS will inherit the compression property though.

As for compression scheme, we can select different algorithms for compression. lz4 is the fastest available (it replaces lzjb which is still available) but gzip is better compression-wise. FreeBSD 11.1 has a new version of ZFS, using a feature-based numbering scheme instead of a single version number (see the New features in 10 page).

I have not done any benchmarking yet on this pool-wide compression. It may be too slow to use the default gzip compression (-6), please feel free to experiment there. You may wish to only enable compression on selected filesets. For now, I will use lz4 everywhere and disable it for specific cases like distfiles.

zfs set compression=lz4 tank
zfs create -o compression=off tank/root
zfs create -o mountpoint=/tank/root/usr tank/usr
zfs create -o mountpoint=/tank/root/usr/obj tank/usr/obj
zfs create -o mountpoint=/tank/root/usr/local tank/usr/local

The reason why I create a separate /usr fileset is that I want to have different policies for compression, atime property, snapshots and all that. You can also create another fileset for /usr/local, once again to be able to snapshot separately the base system and the ports you will be using.

To complete what we want under /usr, we will create /usr/src to hold the system’s sources. We will need these to recompile a new trimmed-down kernel.

zfs create -o mountpoint=/tank/root/usr/src tank/usr/src

Now /var and a few useful filesets here with special properties we care about to avoid security issues. Do not set exec=offon /tmp as it would prevent things like installworld to run properly.

zfs create -o mountpoint=/tank/root/var tank/var
zfs create -o exec=off -o setuid=off tank/var/empty
zfs create -o exec=off -o setuid=off tank/var/named
zfs create -o exec=off -o setuid=off tank/var/run

zfs create -o mountpoint=/tank/root/var/tmp tank/var/tmp
zfs set exec=off tank/var/tmp
zfs set setuid=off  tank/var/tmp
chmod 1777 /tank/root/var/tmp

zfs create -o mountpoint=/tank/root/tmp tank/tmp
zfs set setuid=off  tank/tmp
chmod 1777 /tank/root/tmp

I would also recommend to put users’ home in a separate fileset for the same reason, if not a fileset per user if you want to limit users area to specific sizes.

zfs create -o mountpoint=/tank/root/home tank/home

Later, you will want to create tank/usr/ports/{distfiles,packages} w/o compression as well. Properties like snapdir can be changed later on so we are not forced to set them right now. If you are planning to use the new pkg(1) command to deal with binary packages (aka pkgng) then /usr/ports is not needed.

zfs create -o mountpoint=/tank/root/usr/ports -o setuid=off tank/usr/ports
zfs create -o mountpoint=/tank/root/usr/ports/distfiles  -o compression=off -o exec=off -o setuid=off tank/usr/ports/distfiles
zfs create -o mountpoint=/tank/root/usr/ports/packages -o compression=off -o exec=off -o setuid=off tank/usr/ports/packages

If you plan to have jails on this machine, it is a good idea to create a /jails as well:

zfs create -o mountpoint=/tank/root/jails tank/jails

One of the nice things about ZFS is that for many things, you can use zfs create instead of mkdir. It won’t take that much diskspace and will allow you to specify different policies for backups/snapshots/compression for every filesystem.

One thing you want to know about ZFS is that it uses the Copy-On-Write principle and never overwrite data. Anytime you rewrite a block, a fresh one is written elsewhere and pointers updated (very fast summary, see the ZFS docs for more details). The main result is that when you have a completely filled up fileset, you can not remove files to make space as it would require some free space first. A way to mitigate that is ensuring you do not filled up a fileset and you can reserve some space in the “root” fileset.

zfs set reservation=512m tank

Installing the system

There are several ways to extract the various parts of the distribution. You can get the -memstick images, one of the cd9660 images or just download the *.txz files from a FTP site.

Just like in HOWTO on 9.2, we will extract all distributions manually and fetch everything else from the ‘net.

In the example below, I have retrieved one of the -memstick version, mounted it as an md device in /mnt and therefore I can find the *.txz files in /mnt/usr/freebsd-dist directory. Direct download from where you got the kernel.txz works fine as well of course.

-rw-r--r--    1 ftp      ftp          1157 Jul 21 02:15 MANIFEST
-rw-r--r--    1 ftp      ftp      49166232 Jul 21 02:15 base-dbg.txz
-rw-r--r--    1 ftp      ftp     104780108 Jul 21 02:15 base.txz
-rw-r--r--    1 ftp      ftp       1428656 Jul 21 02:15 doc.txz
-rw-r--r--    1 ftp      ftp      66586424 Jul 21 02:15 kernel-dbg.txz
-rw-r--r--    1 ftp      ftp      36773692 Jul 21 02:15 kernel.txz
-rw-r--r--    1 ftp      ftp      12555216 Jul 21 02:15 lib32-dbg.txz
-rw-r--r--    1 ftp      ftp      18499068 Jul 21 02:15 lib32.txz
-rw-r--r--    1 ftp      ftp      36756836 Jul 21 02:15 ports.txz
-rw-r--r--    1 ftp      ftp     147661560 Jul 21 02:15 src.txz
-rw-r--r--    1 ftp      ftp       4312456 Jul 21 02:15 tests.txz

You can also get them from the directory on ftp.freebsd.org with lftp or ncftp. I used ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/11.1-RELEASE.

Extract all distributions

By default, root is using /bin/csh — aka tcsh which used to be my very first shell in 1988 — as login shell, you need to type sh now in order to cut&paste the examples below.

cd /mnt/usr/freebsd-dist
for i in base doc games kernel lib32 src; do \
   xz -d -c $i.txz | tar -C /tank/root/ -xf - ; \
done

Install configuration variables at proper places

We need to add variables to several files needed for the boot phase, you can use echo(1) or even vi(1).

NOTE: these values MUST be put in the loader.conf that you will put in the mfsbsd image, not in the ZFS pool.

In /boot/loader.conf:

# fs modules
zfs_load="YES"
geom_mirror_load="YES"
fdescfs_load="YES"
nullfs_load="YES"

# Crypto stuff
geom_eli_load="YES"
crypto_load="YES"
aesni_load="YES"
cryptodev_load="YES"

# pf stuff
pf_load="YES"
pflog_load="YES"

# tuning
vm.kmem_size="64G"

Using the following should not be necessary anymore as the loader is able to find which dataset will be used automatically:

vfs.root.mountfrom="zfs:tank/root"

Then you can add some tunables for your ZFS installation:

# http://lists.freebsd.org/pipermail/freebsd-stable/2011-February/061388.html
vfs.zfs.txg.timeout="5"

Current recommendations for FS tuning includes setting kmem_size at between 1.5x and 2x the available RAM. Be careful to use the right device name above for the geli_* lines or you will not be able to attach the encrypted partitions. Do not use geli_tank0_ with gpt/tank0 for example, that will NOT work.

Do not forget to add the following (or another value) to /etc/sysctl.conf or you will have issues at boot-time with vnode depletion:

##-- tuning
kern.maxvnodes=260000

Your /etc/rc.conf should have some variables defined to properly boot:

zfs_enable="YES"
sshd_enable="YES"
hostname="hostname.example.com"
ntpd_enable="YES"
ntpd_sync_on_start="YES"
# or ifconfig_em0="DHCP"
ifconfig_em0="inet a.b.c.d netmask 0xffffff00"   # or "DHCP"
geli_swap_flags="-e aes -l 256 -s 4096 -d"

(do not forget things like defaultrouter and all that).

Exit the chroot area if you were in one for easy editing of the previous files.

Finishing up

There are several steps to follow before even rebooting the first time, including ensuring that we can mount and switch / after the 1st stage boot with mfsbsd.

Configuring the encrypted swap in /tank/root/etc/fstab

# cat /tank/root/etc/fstab
/dev/mirror/swap.eli	none swap sw 0 0

Another issue to look for is that by default, you won’t be able to have kernel crash dumps on a gmirror device (see gmirror(8) for the details and solution). We need to use two special scripts used in the boot process to work around that limitation (as we do not want to always use the prefer setting for mirrored swap):

echo 'gmirror configure -b prefer swap'>>/tank/root/etc/rc.early
echo 'gmirror configure -b round-robin swap'>>/tank/root/etc/rc.local

Fixing mount points

cd /
zfs umount -a
zfs set mountpoint=legacy tank
zfs set mountpoint=/jails tank/jails
zfs set mountpoint=/tmp tank/tmp
zfs set mountpoint=/var tank/var
zfs set mountpoint=/var/empty tank/var/empty
zfs set mountpoint=/var/named tank/var/named
zfs set mountpoint=/var/run tank/var/run
zfs set mountpoint=/var/tmp tank/var/tmp
zfs set mountpoint=/usr tank/usr
zfs set mountpoint=/usr/local tank/usr/local
zfs set mountpoint=/usr/obj tank/usr/obj
zfs set mountpoint=/usr/ports tank/usr/ports
zfs set mountpoint=/usr/ports/distfiles tank/usr/ports/distfiles
zfs set mountpoint=/usr/ports/packages tank/usr/ports/packages

…and for all other filesets you added back above without forgetting to set the bootfs property on the right fileset:

zpool set bootfs=tank/root tank

Second stage booting

Now that the system is installed in the encrypted pool, we have to create a script that will be used to mount and attach the encrypted pool after the 1st stage boot and switch the kernel root to use the encrypted pool for all the tree.

It is achieved by using the kenv(8) command to set the root directory and use reboot -r to use the now-decrypted pool as /.

It is easier and less error-prone to write a script for that.

Things to remember

Whe you system is up and working, you will at some point want to update, either to stay close to the branch or because of a security issue or whatever.

Some things to keep in mind:

Jails

For the last few years, I have been using Ezjail for creating and maintaining my jails and using Ansible for accessing them. Iocage is another way to deal with jails. As ezjail has not been really updated (it is still unhappy with jail.conf) for a while, I may switch to iocage at some point.

See also my HOWTO on jails & Ansible

SoloBSD/HardenedBSD Alternative

There is a fork of FreeBSD called HardenedBSD aiming to provide more security out-of-the-box with mechanisms such as W^X, ASLR, StackGuard and more. Guillermo García R. has build the equivalent of mfsbsd for HardenedBSD called SoloBSD and you can find 11.1-based images there.

For remote machines such as the ones I install with this tutorial, it could be an interesting choice with a few constraints (such as using HardenedBSD’s ports tree instead of the FreeBSD one). Food for thoughts in any case.

Resources

NOTE: This is a work-in-progress, please check it regularly for updates.

Feedback

Please send any comment, addition, correction to my FreeBSD mail or my personal mail. Thanks to all who have done it already.

History

1.0 Creation
1.1 Typos, HardenedBSD/SoloBSD addition 1.2 Typos, beadm and more

Footnotes