FreeBSD and UEFI Boot

by Tykling


19. apr 2018 13:30 UTC


FreeBSD got UEFI support not too long ago, including in the installer. This means you can install on servers without BIOS or where boot mode is set to UEFI. Legacy booting is full of surprises and UEFI will hopefully turn out to be a nice replacement. The only problem I've encountered has been recreating the UEFI partition after replacing a disk, which is what this blogpost is about.

The installer takes care of creating the initial partitions, the default layout looks like this:

[tsr@svaneke ~]$ gpart show da2
=>       40  976773088  da2  GPT  (466G)
         40       1600    1  efi  (800K)
       1640       1024    2  freebsd-boot  (512K)
       2664       1432       - free -  (716K)
       4096   20971520    3  freebsd-swap  (10G)
   20975616  955797504    4  freebsd-zfs  (456G)
  976773120          8       - free -  (4.0K)

[tsr@svaneke ~]$ 

As you can see, the installer created a small 1600 blocks / 800 kilobytes partition of type efi before the freebsd-boot partition. It works well until you need to replace a disk for some reason. Usually I would insert the new disk, gpart create/add the partitions, run zpool replace, and run gpart bootcode on the new disk. But this was before efi.

Replace Disk

I physically yank out the old disk and insert the new one. I don't bother to zpool offline it first. It is da1 that has been replaced.

Partitioning

These days we use GPT partitions and the gpart(8) tool to manage them. First I need to create the GPT scheme:

[tsr@svaneke ~]$ sudo gpart create -s GPT da1
Password:
da1 created
[tsr@svaneke ~]$ 

Then I need to add the same partitions the installer created when I installed the server. The complete partition layout is shown above. I add the partitions one by one with the same sizes and offsets the installer used:

[tsr@svaneke ~]$ sudo gpart add -t efi -s 1600 da1
da1p1 added
[tsr@svaneke ~]$ sudo gpart add -t freebsd-boot -s 1024 da1
da1p2 added
[tsr@svaneke ~]$ sudo gpart add -b 4096 -t freebsd-swap -s 10G da1
da1p3 added
[tsr@svaneke ~]$ sudo gpart add -t freebsd-zfs -l disk0 da1
da1p4 added
[tsr@svaneke ~]$ gpart show da1
=>         40  23437770672  da1  GPT  (11T)
           40         1600    1  efi  (800K)
         1640         1024    2  freebsd-boot  (512K)
         2664         1432       - free -  (716K)
         4096     20971520    3  freebsd-swap  (10G)
     20975616  23416795096    4  freebsd-zfs  (11T)

[tsr@svaneke ~]$ 

So, to sum up: First a small efi partition of 800K, then the regular freebsd-boot partition of 512K, then a bit of free space due to 4k alignment, then a 10G freebsd-swap partition, and finally "the rest" for the main freebsd-zfs partition.

Also worth noting is that I always give the ZFS partition a GPT label with -l, and then I use that label instead of the daX device name in the zpool. I GPT label the disk according to the numbering of the physical slot on the server chassis. So the label disk0 means that this disk sits in slot 0 of the chassis.

The result of this is that the disks are named so I can easily replace the correct one at 3am in the datacenter - even if the controller or driver decides to renumber the drives so da0 becomes da17 suddently (yes, that can happen).

Adding to ZFS

This is the easy part. zpool status shows the current status of the pool:

[tsr@svaneke ~]$ zpool status
  pool: zroot
 state: DEGRADED
status: One or more devices has been removed by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: resilvered 448G in 1h51m with 0 errors on Thu Apr 19 10:54:39 2018
config:

        NAME                     STATE     READ WRITE CKSUM
        zroot                    DEGRADED     0     0     0
          mirror-0               DEGRADED     0     0     0
            7452263023654412841  REMOVED      0     0     0  was /dev/gpt/disk0
            gpt/disk1            ONLINE       0     0     0
[tsr@svaneke ~]$ zpool status

All I need to do is to tell ZFS to replace the missing device with my new gpt/disk0 device:

[tsr@svaneke ~]$ sudo zpool replace zroot 7452263023654412841 /dev/gpt/disk0 
Password:
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'zroot', you may need to update
boot code on newly attached disk '/dev/gpt/disk0'.

Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0

[tsr@svaneke ~]$ 

ZFS is now resilvering the mirror-0 vdev. Note the gpart bootcode line which must still be used to fix the freebsd-boot partition, UEFI or not.

When it is done my ZPOOL is healthy again (and quite a bit larger :)). Check the resilver status:

[tsr@svaneke ~]$ zpool status
  pool: zroot
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Thu Apr 19 13:17:49 2018
        3.01G scanned out of 7.20T at 154M/s, 13h34m to go
        164M resilvered, 0.04% done
config:

        NAME                       STATE     READ WRITE CKSUM
        zroot                      DEGRADED     0     0     0
          mirror-0                 DEGRADED     0     0     0
            replacing-0            REMOVED      0     0     0
              7452263023654412841  REMOVED      0     0     0  was /dev/gpt/disk0
              gpt/disk0            ONLINE       0     0     0  (resilvering)
            gpt/disk1              ONLINE       0     0     0
[tsr@svaneke ~]$

Resilver (and scrub as well) always start out slow, but give it 30 minutes or so and the estimated time to go should be more accurate.

Three Ways of Fixing the UEFI Bootcode

The main point of this post was the realization that I have no idea how to fix the efi partition so it boots the system. Looking into it I found out that the builtin gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 command I usually run doesn't deal with the efi partition at all. The efi partition is a regular msdos partition with a couple of files:

[tsr@svaneke ~]$ sudo mount -t msdosfs /dev/da0p1 /mnt/
[tsr@svaneke ~]$ find /mnt/
/mnt/
/mnt/efi
/mnt/efi/boot
/mnt/efi/boot/BOOTX64.EFI
/mnt/efi/boot/STARTUP.NSH
[tsr@svaneke ~]$ cp -prf /mnt/efi/ .
[tsr@svaneke ~]$ ls -l boot/
total 41
-rwxr-xr-x  1 tsr  tsr  131072 Apr 12  2016 BOOTX64.EFI
-rwxr-xr-x  1 tsr  tsr      12 Apr 12  2016 STARTUP.NSH
[tsr@svaneke ~]$ sudo umount /mnt/

The Manual Way

So the manual way would be to format the efi partition on the new disk, and create the same folder structure and copy the files over:

[tsr@svaneke ~]$ sudo newfs_msdos /dev/da1p1
newfs_msdos: trim 25 sectors to adjust to a multiple of 63
/dev/da1p1: 1532 sectors in 1532 FAT12 clusters (512 bytes/cluster)
BytesPerSec=512 SecPerClust=1 ResSectors=1 FATs=2 RootDirEnts=512 Sectors=1575 Media=0xf0 FATsecs=5 SecPerTrack=63 Heads=255 HiddenSecs=0
[tsr@svaneke ~]$ sudo mount -t msdosfs /dev/da1p1 /mnt/
[tsr@svaneke ~]$ sudo mkdir -p /mnt/efi/boot
[tsr@svaneke ~]$ sudo cp -v boot/* /mnt/efi/boot/
boot/BOOTX64.EFI -> /mnt/efi/boot/BOOTX64.EFI
boot/STARTUP.NSH -> /mnt/efi/boot/STARTUP.NSH
[tsr@svaneke ~]$ sudo umount /mnt/

dd From Other Drive

I could also have used dd from the other drive in the mirror to achieve the same result:

[tsr@svaneke ~]$ sudo dd if=/dev/da0p1 of=/dev/da1p1 bs=1M
0+1 records in
0+1 records out
819200 bytes transferred in 0.200710 secs (4081518 bytes/sec)
[tsr@svaneke ~]$ 

dd From Running System

Or I could even dd from the /boot/boot1.efifat device on the running efi booted system. This method can also work to fix an unbootable system, by uefi booting a live installer usbstick to do this from:

[tsr@svaneke ~]$ sudo dd if=/boot/boot1.efifat of=/dev/da0p1
Password:
1600+0 records in
1600+0 records out
819200 bytes transferred in 1.131642 secs (723904 bytes/sec)
[tsr@svaneke ~]$ 

All three ways work and the server booted up nicely the next time I restarted it.

Search this blog

Tags for this blogpost