by Tykling
19. apr 2018 13:30 UTC
FreeBSD got UEFI support not too long ago, including in the installer. This means you can install on servers without BIOS or where boot mode is set to UEFI. Legacy booting is full of surprises and UEFI will hopefully turn out to be a nice replacement. The only problem I've encountered has been recreating the UEFI partition after replacing a disk, which is what this blogpost is about.
The installer takes care of creating the initial partitions, the default layout looks like this:
[tsr@svaneke ~]$ gpart show da2
=> 40 976773088 da2 GPT (466G)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 20971520 3 freebsd-swap (10G)
20975616 955797504 4 freebsd-zfs (456G)
976773120 8 - free - (4.0K)
[tsr@svaneke ~]$
As you can see, the installer created a small 1600 blocks / 800 kilobytes partition of type efi before the freebsd-boot partition. It works well until you need to replace a disk for some reason. Usually I would insert the new disk, gpart create/add the partitions, run zpool replace, and run gpart bootcode on the new disk. But this was before efi.
I physically yank out the old disk and insert the new one. I don't bother to zpool offline it first. It is da1 that has been replaced.
These days we use GPT partitions and the gpart(8) tool to manage them. First I need to create the GPT scheme:
[tsr@svaneke ~]$ sudo gpart create -s GPT da1 Password: da1 created [tsr@svaneke ~]$
Then I need to add the same partitions the installer created when I installed the server. The complete partition layout is shown above. I add the partitions one by one with the same sizes and offsets the installer used:
[tsr@svaneke ~]$ sudo gpart add -t efi -s 1600 da1
da1p1 added
[tsr@svaneke ~]$ sudo gpart add -t freebsd-boot -s 1024 da1
da1p2 added
[tsr@svaneke ~]$ sudo gpart add -b 4096 -t freebsd-swap -s 10G da1
da1p3 added
[tsr@svaneke ~]$ sudo gpart add -t freebsd-zfs -l disk0 da1
da1p4 added
[tsr@svaneke ~]$ gpart show da1
=> 40 23437770672 da1 GPT (11T)
40 1600 1 efi (800K)
1640 1024 2 freebsd-boot (512K)
2664 1432 - free - (716K)
4096 20971520 3 freebsd-swap (10G)
20975616 23416795096 4 freebsd-zfs (11T)
[tsr@svaneke ~]$
So, to sum up: First a small efi partition of 800K, then the regular freebsd-boot partition of 512K, then a bit of free space due to 4k alignment, then a 10G freebsd-swap partition, and finally "the rest" for the main freebsd-zfs partition.
Also worth noting is that I always give the ZFS partition a GPT label with -l, and then I use that label instead of the daX device name in the zpool. I GPT label the disk according to the numbering of the physical slot on the server chassis. So the label disk0 means that this disk sits in slot 0 of the chassis.
The result of this is that the disks are named so I can easily replace the correct one at 3am in the datacenter - even if the controller or driver decides to renumber the drives so da0 becomes da17 suddently (yes, that can happen).
This is the easy part. zpool status shows the current status of the pool:
[tsr@svaneke ~]$ zpool status
pool: zroot
state: DEGRADED
status: One or more devices has been removed by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
scan: resilvered 448G in 1h51m with 0 errors on Thu Apr 19 10:54:39 2018
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
7452263023654412841 REMOVED 0 0 0 was /dev/gpt/disk0
gpt/disk1 ONLINE 0 0 0
[tsr@svaneke ~]$ zpool status
All I need to do is to tell ZFS to replace the missing device with my new gpt/disk0 device:
[tsr@svaneke ~]$ sudo zpool replace zroot 7452263023654412841 /dev/gpt/disk0
Password:
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'zroot', you may need to update
boot code on newly attached disk '/dev/gpt/disk0'.
Assuming you use GPT partitioning and 'da0' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
[tsr@svaneke ~]$
ZFS is now resilvering the mirror-0 vdev. Note the gpart bootcode line which must still be used to fix the freebsd-boot partition, UEFI or not.
When it is done my ZPOOL is healthy again (and quite a bit larger :)). Check the resilver status:
[tsr@svaneke ~]$ zpool status
pool: zroot
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Thu Apr 19 13:17:49 2018
3.01G scanned out of 7.20T at 154M/s, 13h34m to go
164M resilvered, 0.04% done
config:
NAME STATE READ WRITE CKSUM
zroot DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
replacing-0 REMOVED 0 0 0
7452263023654412841 REMOVED 0 0 0 was /dev/gpt/disk0
gpt/disk0 ONLINE 0 0 0 (resilvering)
gpt/disk1 ONLINE 0 0 0
[tsr@svaneke ~]$
Resilver (and scrub as well) always start out slow, but give it 30 minutes or so and the estimated time to go should be more accurate.
The main point of this post was the realization that I have no idea how to fix the efi partition so it boots the system. Looking into it I found out that the builtin gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0 command I usually run doesn't deal with the efi partition at all. The efi partition is a regular msdos partition with a couple of files:
[tsr@svaneke ~]$ sudo mount -t msdosfs /dev/da0p1 /mnt/ [tsr@svaneke ~]$ find /mnt/ /mnt/ /mnt/efi /mnt/efi/boot /mnt/efi/boot/BOOTX64.EFI /mnt/efi/boot/STARTUP.NSH [tsr@svaneke ~]$ cp -prf /mnt/efi/ . [tsr@svaneke ~]$ ls -l boot/ total 41 -rwxr-xr-x 1 tsr tsr 131072 Apr 12 2016 BOOTX64.EFI -rwxr-xr-x 1 tsr tsr 12 Apr 12 2016 STARTUP.NSH [tsr@svaneke ~]$ sudo umount /mnt/
So the manual way would be to format the efi partition on the new disk, and create the same folder structure and copy the files over:
[tsr@svaneke ~]$ sudo newfs_msdos /dev/da1p1 newfs_msdos: trim 25 sectors to adjust to a multiple of 63 /dev/da1p1: 1532 sectors in 1532 FAT12 clusters (512 bytes/cluster) BytesPerSec=512 SecPerClust=1 ResSectors=1 FATs=2 RootDirEnts=512 Sectors=1575 Media=0xf0 FATsecs=5 SecPerTrack=63 Heads=255 HiddenSecs=0 [tsr@svaneke ~]$ sudo mount -t msdosfs /dev/da1p1 /mnt/ [tsr@svaneke ~]$ sudo mkdir -p /mnt/efi/boot [tsr@svaneke ~]$ sudo cp -v boot/* /mnt/efi/boot/ boot/BOOTX64.EFI -> /mnt/efi/boot/BOOTX64.EFI boot/STARTUP.NSH -> /mnt/efi/boot/STARTUP.NSH [tsr@svaneke ~]$ sudo umount /mnt/
I could also have used dd from the other drive in the mirror to achieve the same result:
[tsr@svaneke ~]$ sudo dd if=/dev/da0p1 of=/dev/da1p1 bs=1M 0+1 records in 0+1 records out 819200 bytes transferred in 0.200710 secs (4081518 bytes/sec) [tsr@svaneke ~]$
Or I could even dd from the /boot/boot1.efifat device on the running efi booted system. This method can also work to fix an unbootable system, by uefi booting a live installer usbstick to do this from:
[tsr@svaneke ~]$ sudo dd if=/boot/boot1.efifat of=/dev/da0p1 Password: 1600+0 records in 1600+0 records out 819200 bytes transferred in 1.131642 secs (723904 bytes/sec) [tsr@svaneke ~]$
All three ways work and the server booted up nicely the next time I restarted it.