10. oct 2020 16:22 UTC
I've recently added a wall mounted 30" monitor for Grafana in my home. I can highly recommend doing the same, especially in a world where more work from home is becoming the norm.
Having metrics visible at all times can be incredibly helpful in spotting trends and issues. This is the reason we all have wall mounted Grafanas in our workplaces! Since we are all going to be working from home for the foreseeable future it makes sense to have visible metrics at home as well. Also, much of the stuff I need metrics for is not work related at all - UncensoredDNS for example, the BornHack infrastructure, and so on. I see no reason why projects run outside of $dayjob should receive any less attention, such as visualisation of metrics.
Sometime before summer I physically mounted the screen using this very nice over/under 2 monitor stand, and then started considering which computer to use for it. Ideally something I already had lying around, and of course something with enough power to run some Grafana dashboards without choking.
Not for a moment had I anticipated that actually displaying the graphs in a browser that doesn't crash constantly would be the most time-consuming part of getting this up and running. I mean, I've spent hours playing with various Prometheus exporters for all the different servers I run (and Ansible roles for them). I've also spent hours fiddling with Grafana dashboards until they show the level of information I like. Both of these are complex jobs and and I am absolutely fine with spending time on them. But when I finally had nice working dashboards in my Firefox browser on my laptop I fully expected it to be trivial to put those on a monitor on the wall. It was not. This tale is left here for future metrics aficionados in the hopes that you will spend less time on this dreary task than I did.
I've documented the failed attempts as well as the final well-working solution. Feel free to skip over the first two sections if you are just here to learn what works :)
Originally I wanted to use an ODROID C2 which I had lying around because I used it as a mediacenter with
Kodi provided by
Liberelec previously. This seemed like a good choice, I mean if it can show a 1080p video without choking it should be able to render a few
Grafana dashboards, right?
ODROID-C2 is an
Amlogic S905 SOC with a
Cortex A53 ARMv8 64bit CPU. It has 4 cores and 2GB RAM. It supports
eMMC storage which is a nice and faster alternative to the SD cards used by many of the "Raspberry PI-sized" computers that exists today.
I tried to figure out wether it was possible to run
FreeBSD on it, but the ARM Wiki page seems to say that 64bit ARM support is still being worked on. I thought it was too much to hope for anyway.
Once I got Ubuntu up and running and found a Firefox browser it quickly became apparent that it would not work. The browser kept freezing up, or crashing, or both. I tried using Chromium which annoyingly worked a bit better. It was not stable, but it would run for half a day or maybe a day and then stop working somehow. And this was with just one dashboard, If I opened another tab with another Grafana dashboard it made matters worse.
It wasn't immediately clear what the problem was, in hindsight it was probably RAM, but either way I decided to try something else.
I had an RPI3b lying around and decided to try that instead. I am not a big fan of the RPI platform in general, I've had way too many weird issues over the years, I guess I just prefer a more normal computer. But I needed something to show my graphs, so I launched NOOBS and asked it to install
Rasbbian and I was pretty soon up and running with graphs on my wall again.
The Raspberry PI 3b+ has a full 1GB of RAM, much more than previous PIs, but only half of what the
ODROID-C2 has. It still performed about the same as the
ODROID-C2. It didn't work at all with Firefox, with Chromium it was okay with a single tab or two, but after 12-24 hours it would stop working. Sometimes it would be an "unresponsive tab" error from the browser, and sometimes the whole OS would freeze up completely.
Around this time the annual BornHack was approaching, so my attention was needed elsewhere. I made due with the unstable PI for a couple of months - daily reboot helped a little, but it isn't exactly a nice solution.
So last week I finally got around to taking another whack it. No more beating around the bush, I was going to buy something with enough power, and then some. I started looking around and soon found that a Danish shop with pickup service had an Intel
NUC8I7BEH in stock. While physically larger than the Raspberry PI and ODROID-C2 (it is just under 12x12cm long and wide, and just under 6cm tall, including the VESA mount), it is still a pretty small computer.
I picked it up around along with 2 sticks of
Kingston 16GB DDR4 2400MHz SODIMM non-ECC RAM, (the NUCs are delivered as a "kit" without RAM and storage) and a
Samsung 870 QVO MZ-77Q1T0BW 1TB SATA SSD. Assembling it was really easy, four screws and and a couple of minutes later the RAM and SSD were in, and the NUC POST revealed it found all 32GB RAM and also the SSD. It refused to boot from the FreeBSD installer USB stick though. This was because secure boot was enabled in the BIOS. After disabling it the FreeBSD install was absolutely standard, no issues at all.
This was in the beginning of October 2020 so the latest
12-STABLE was something like
12.2-BETA3 or so, it doesn't really matter since I upgrade to latest
12-STABLE after installing anyway. As always I used the auto-ZFS option in the installer so I can use
bectl to make nice boot environments for easy rollbacks in case of problems when upgrading.
pkg install xorg slim fluxbox firefox and a bit of fiddling later I was looking at a very smoothly running Grafana dashboard in X. The rest of this blogpost is about the configuration of FreeBSD, X, Slim, Fluxbox and Firefox for an ideal wall-mounted Grafana screen. Much of this is done in Ansible, but not all of it. I've documented everything here for the sake of completeness, but a lot of basic configuration is done by my Ansible roles, and is not that relevant for this blogpost. Stuff like configuring syslog, monitoring, SSH keys and all the other sysadminy stuff that all machines need. This blogpost just focuses on what to do to go from a fresh FreeBSD machine to a well oiled Grafana wall display.
Finally: I realize that comparing a NUC to a Raspberry PI is not even close to a fair comparison, for starters the NUC is ten times more expensive. It has a powerful
CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz (2712.12-MHz K8-class CPU) and 32GB memory, so obviously it is going to perform a lot better. But overkill was the whole point of this exercise! I had spent far too much time faffing about with insufficient hardware and at this point I was perfectly happy to throw money at the problem until it went away. YMMV.
First off I added a dedicated user to show the graphs, since I intend to enable autologin for this setup. This user is not going to have
sudo access, it really only has to be able to start X and Firefox. I also have another user with my regular SSH key which I used for debugging and whatever else is needed (rarely sysadm stuff though, since that is handled by Ansible).
On modern FreeBSD X mostly configures itself, but the
i7-8559U inside the
NUC8i7BEH is a Gen9 Intel Coffee Lake CPU, the integrated
Iris Plus Graphics 655 GPU inside it is too new to be supported by the
i915kms module in FreeBSD base. This is the output from
pciconf -lv concerning the GPU:
vgapci0@pci0:0:2:0: class=0x030000 card=0x20748086 chip=0x3ea58086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Iris Plus Graphics 655' class = display subclass = VGA
Fortunately there is a meta-port called
drm-fbsd12.0-kmod (which also works on 12.2 it seems) which installs a newer
i915kms.ko which supports the GPU. Since building the driver requires kernel sources it is not available from the official FreeBSD package builders, and since I needed to upgrade anyway I started a
buildworld with the following beauty of a oneliner (run as root, not with sudo):
time (make -j$(sysctl -n hw.ncpu) buildworld && make -j$(sysctl -n hw.ncpu) kernel && mergemaster -pFUi && make installworld && mergemaster -FUi && make BATCH_DELETE_OLD_FILES=yes delete-old && make BATCH_DELETE_OLD_FILES=yes delete-old-libs) && date. Note that a reboot before the first
mergemaster is recommended, but for small upgrades I usually don't bother. If
installworld fails with weird errors try again after a reboot :)
Including a few minutes for some
mergemaster fun near the end it took just 59 minutes before I had a newly built world and kernel. Then I could build the driver from ports, and add the line
/etc/rc.conf and reboot. After the reboot X started with no issues at all. For reference this is a
dmesg(8) from the NUC after the new driver was enabled (I've highlighed the GPU related stuff):
[tykling@nuc1 ~]$ sudo cat /var/run/dmesg.boot ---<
>--- Copyright (c) 1992-2020 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 12.2-STABLE r366461 GENERIC amd64 FreeBSD clang version 10.0.1 (email@example.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2) VT(efifb): resolution 1920x1080 CPU: Intel(R) Core(TM) i7-8559U CPU @ 2.70GHz (2712.12-MHz K8-class CPU) Origin="GenuineIntel" Id=0x806ea Family=0x6 Model=0x8e Stepping=10 Features=0xbfebfbff Features2=0x7ffafbbf AMD Features=0x2c100800 AMD Features2=0x121 Structured Extended Features=0x29c67af Structured Extended Features3=0x9c002400 XSAVE Features=0xf VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33204260864 (31666 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads random: unblocking device. ioapic0 irqs 0-119 on motherboard Launching APs: 1 3 4 6 7 5 2 Timecounter "TSC-low" frequency 1356058849 Hz quality 1000 random: entropy device external interface kbd0 at kbdmux0 000.000023  netmap_init netmap: loaded module [ath_hal] loaded module_register_init: MOD_LOAD (vesa, 0xffffffff81115e40, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" nexus0 efirtc0: on motherboard efirtc0: registered as a time-of-day clock, resolution 1.000000s cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 24000000 Hz quality 950 Event timer "HPET" frequency 24000000 Hz quality 550 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0 acpi_ec0: port 0x62,0x66 on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 vgapci0: port 0x4000-0x403f mem 0xbf000000-0xbfffffff,0x80000000-0x8fffffff at device 2.0 on pci0 vgapci0: Boot video device xhci0: mem 0x404ac00000-0x404ac0ffff at device 20.0 on pci0 xhci0: 32 bytes context size, 64-bit DMA usbus0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 pci0: at device 20.2 (no driver attached) pci0: at device 20.3 (no driver attached) pci0: at device 22.0 (no driver attached) ahci0: port 0x4090-0x4097,0x4080-0x4083,0x4060-0x407f mem 0xc0a24000-0xc0a25fff,0xc0a27000-0xc0a270ff,0xc0a26000-0xc0a267ff at device 23.0 on pci0 ahci0: AHCI v1.31 with 1 6Gbps ports, Port Multiplier not supported ahcich2: at channel 2 on ahci0 pcib1: at device 28.0 on pci0 pci1: on pcib1 pcib2: at device 28.4 on pci0 pcib2: [GIANT-LOCKED] pcib3: at device 29.0 on pci0 pcib3: [GIANT-LOCKED] pcib4: at device 29.6 on pci0 pcib4: [GIANT-LOCKED] pci2: on pcib4 pci2: at device 0.0 (no driver attached) isab0: at device 31.0 on pci0 isa0: on isab0 hdac0: mem 0xc0a20000-0xc0a23fff,0x404ab00000-0x404abfffff at device 31.3 on pci0 pci0: at device 31.5 (no driver attached) em0: mem 0xc0a00000-0xc0a1ffff at device 31.6 on pci0 em0: Using 1024 TX descriptors and 1024 RX descriptors em0: Using an MSI interrupt em0: Ethernet address: 1c:69:7a:0e:0a:e4 em0: netmap queues/slots: TX 1/1024, RX 1/1024 acpi_button0: on acpi0 acpi_button1: on acpi0 acpi_tz0: on acpi0 acpi_syscontainer0: on acpi0 acpi_tz1: on acpi0 acpi_tz1: _HOT value is absurd, ignored (-73.1C) atrtc0: at port 0x70 irq 8 on isa0 atrtc0: Warning: Couldn't map I/O. atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 atrtc0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12. uart0: at port 0x3f8 irq 4 flags 0x10 on isa0 uart0: non-PNP ISA device will be removed from GENERIC in FreeBSD 12. est0: on cpu0 ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec acpi_tz1: _TMP value is absurd, ignored (-263.1C) hdacc0: at cad 0 on hdac0 hdaa0: at nid 1 on hdacc0 pcm0: at nid 33 and 25 on hdaa0 hdacc1: at cad 2 on hdac0 hdaa1: at nid 1 on hdacc1 pcm1: at nid 3 on hdaa1 Trying to mount root from zfs:zroot/ROOT/12-STABLE-366461 ... Root mount waiting for: usbus0 CAM ugen0.1: <0x8086 XHCI root HUB> at usbus0 uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 ada0 at ahcich2 bus 0 scbus0 target 0 lun 0 ada0: ACS-4 ATA SATA 3.x device ada0: Serial Number S5SVNG0N730381M ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes) ada0: Command Queueing enabled ada0: 953869MB (1953525168 512 byte sectors) uhub0: 18 ports with 18 removable, self powered Root mount waiting for: usbus0 ugen0.2: at usbus0 ukbd0 on uhub0 ukbd0: on usbus0 kbd1 at ukbd0 Root mount waiting for: usbus0 ugen0.3: at usbus0 GEOM_ELI: Device ada0p3.eli created. GEOM_ELI: Encryption: AES-XTS 128 GEOM_ELI: Crypto: software drmn0: on vgapci0 vgapci0: child drmn0 requested pci_enable_io vgapci0: child drmn0 requested pci_enable_io [drm] Unable to create a private tmpfs mount, hugepage support will be disabled(-19). [drm] Found 128MB of eDRAM Failed to add WC MTRR for [0x80000000-0x8fffffff]: -22; performance may suffer [drm] Got stolen memory base 0x7c000000, size 0x4000000 [drm] Supports vblank timestamp caching Rev 2 (21.10.2013). [drm] Driver supports precise vblank timestamp query. [drm] Connector eDP-1: get mode from tunables: [drm] - kern.vt.fb.modes.eDP-1 [drm] - kern.vt.fb.default_mode [drm] failed to retrieve link info, disabling eDP [drm] Connector DP-1: get mode from tunables: [drm] - kern.vt.fb.modes.DP-1 [drm] - kern.vt.fb.default_mode [drm] Connector DP-2: get mode from tunables: [drm] - kern.vt.fb.modes.DP-2 [drm] - kern.vt.fb.default_mode [drm] Connector HDMI-A-1: get mode from tunables: [drm] - kern.vt.fb.modes.HDMI-A-1 [drm] - kern.vt.fb.default_mode [drm] Initialized i915 1.6.0 20171222 for drmn0 on minor 0 VT: Replacing driver "efifb" with new "fb". start FB_INFO: type=11 height=1080 width=1920 depth=32 cmsize=16 size=8294400 pbase=0x80040000 vbase=0xfffff80080040000 name=drmn0 flags=0x0 stride=7680 bpp=32 cmap=0 cmap=7f0000 cmap=7f00 cmap=c4a000 end FB_INFO drmn0: fb0: inteldrmfb frame buffer deviceiwm0: mem 0x404ac10000-0x404ac13fff at device 20.3 on pci0 iwm0: hw rev 0x310, fw ver 34.3125811985.0, address 94:e6:f7:e8:3f:00 drmn0: successfully loaded firmware image with name: i915/kbl_dmc_ver1_04.bin [drm] Finished loading DMC firmware i915/kbl_dmc_ver1_04.bin (v1.4) lo0: link state changed to UP em0: link state changed to UP ums0 on uhub0 ums0: on usbus0 ums0: 6 buttons and [XYZT] coordinates ID=1 [tykling@nuc1 ~]$
I did very little actual configuration of X (apart from disabling the screensaver, so the monitor doesn't turn off after 10 minutes of inactivity). This is the entire
# disable screensaver xset s off xset -dpms # dk keyboard setxkbmap dk # Fluxbox exec /usr/local/bin/startfluxbox
I started out by following the FreeBSD Handbook and enabling
XDM to show a graphical login window on boot. This turned out to be a bad idea since
XDM doesn't support autologin and the authors don't intend to support it. Since I'd like my wall display to start up in X and login and start the browser and show the graphs automatically after reboot autologin is a required feature for this setup.
XDM I went with
SLIM which is the
Simple LogIn Manager. It is very lightweight and has few dependencies. It was very easy to install and enable:
[tykling@nuc1 ~]$ pkg info | grep slim slim-1.3.6_21 Graphical login manager for X11, derived from Login.app slim-themes-1.0.1_2 Theme pack for SLiM login app [tykling@nuc1 ~]$ grep slim /etc/rc.conf slim_enable="YES" [tykling@nuc1 ~]$
I only changed 2 lines in the default
/usr/local/etc/slim.conf file. One was the
default_user setting which I set to
walldisplay which is the username I've created to show the graphs. The other one is
auto_login which unsurprisingly needs to be set to
After these changes I rebooted and
SLIM autologged in the user and started Fluxbox just as I wanted. Lovely!
Fluxbox used to be my window manager of choice back when I still used FreeBSD on the laptop. It is lightweight and simple, perfect for this setup. It doesn't really need any configuration for this, but I did want it to start Firefox when X starts up.
The configuration for
Fluxbox is in
~/.fluxbox/ folder, so for this I needed to edit the file
/usr/home/walldisplay/.fluxbox/startup. I simply added the following line just before the
exec fluxbox line:
This means Firefox will start up every time X starts Fluxbox, and since X starts up on boot and
SLIM is configured to auto-login, this means the machine now starts X, Fluxbox and Firefox automatically on boot.
Since I don't plan on having a keyboard connected to this system permanently, I will need VNC access to the X display so I can add a new tab or change things as new needs develop. I installed the
x11vnc package, and rather than running it all the time as a system service I simply run it over SSH and forward port 5900 to my local system when I need it. This means I don't need to worry about encryption or authentication of the VNC connection. It also means that the VNC server only runs when I need it, which is nice.
To connect to VNC on my wall display computer I first run the following command to start
x11vnc on the walldisplay computer:
$ ssh -t -L5900:localhost:5900 walldisplay@nuc1 'x11vnc -localhost -display :0 -ncache_cr'
x11vnc command outputs some scary sounding warnings about VNC running without a password, it doesn't matter since it is only exposed on localhost. Note that I SSH in as the
walldisplay user, since that is the user logged into the physical X display. I leave the SSH terminal where I started
x11vnc running and open a new terminal tab where I run:
$ vncviewer -encodings 'hextile' localhost:0
My client OS is run Debian 10 inside Qubes OS, the VNC client I chose is called
xtightvncviewer. It works great, the connection is as smooth as I would expect for a LAN VNC connection.
unclutter can be installed with
pkg and it has a simple job: hide the mouse cursor when the mouse is idle, so it doesn't sit in the middle of a graph being annoying. After installing
unclutter I enable it in
/usr/home/walldisplay/.fluxbox/startup by adding the following line just before the
The only thing remaining now is to get Firefox to act right and I should be done!
I wanted to change a few things about Firefox to make it suitable for a wall mounted Grafana dashboard browser: I want it to start in fullscreen (as if
F11 had been pressed), I want it to start with some specific tabs open, and I want it to cycle between the tabs.
Firefox doesn't have an option to make the browser automatically go fullscreen (as if
F11 had been pressed) after stating, but fortunately there is a plugin which does just that. It is called Auto Fullscreen (Github) and it does what it says on the box.
Firefox has an option in
Preferences -> General -> Startup -> Restore previous session which does what I want. I then opened up all the Grafana dashboards I want to show and close the browser, making it save it to the session. Next time (and all times after that) when I open the browser it will open all my Grafana tabs automatically.
To make Firefox cycle through the open tabs I used another addon called Tab cycle Quantum (Github). The only configurable option is the amout of time between cycles, which defaults to 5 seconds. It works as advertised, only gripe I have is that it doesn't start as enabled - I have to press a button in the toolbar to enable it :( I've opened an issue to make it possible to configure the addon to enable tab cycling on start, but since it has been more than 3 years since the last commit I don't have high hopes that it will be fixed. But since the addon is like 10-20 lines of JS I might just fork it and combine it with the above fullscreen addon and make
Tyks Kiosk Addon combining the two.
Finally I installed the Tabliss (Homepage) addon, which shows pretty pictures of nature and stuff when a new tab is opened before a URL is loaded. In my tab rotation I currently have 9 Grafana dashboards plus a single tab where I haven't opened any URL. This means that the tab cycler will show a random pretty picture instead of a Grafana tab about 10% of the time. Highly recommended (also for your day-to-day browser)!
The final result is very nice. It boots from off to Grafana in less than 1 minute, meaning I can just turn it off when I am sleeping or out of the house. Grafana is fast and smooth on it. Since the machine is easily powerful enough to do more than it does already I might enable it for remote X rendering for additional wall mounted displays. But for now I am happy working on perfecting my Grafana dashboards knowing that I can watch them on my wall for years to come. Happy graphing!
I've recently signed up for Github Sponsors meaning it is now easy to sponsor me and my work. If this post or some of my other writing, software or services have helped you then you can consider becoming a sponsor.