05. nov 2020 11:34 UTC
I've been playing with mobile modems recently, and found myself in need of a good way to monitor the signal strength of the modems. Standard AT commands can be used to query each modem about its (perceived) signal strength. I want to get these numbers periodically and put them into Prometheus so I can make myself a nice Grafana dashboard showing how my modems are doing.
This innocent sounding task ended ...
02. nov 2020 22:27 UTC
I recently had to configure a computer for environmental monitoring in the 20" shipping container we use for BornHack storage. The container sits outside in the wind and rain all year. This means we've had to fit it with a dehumidifier to make sure the moisture doesn't destroy all our stuff. Being nerds we want graphs to tell us about the temperature and humidity levels in the container, but since it is sitting in a place with no Internet the first challenge was to get online.
10. oct 2020 16:22 UTC
I've recently added a wall mounted 30" monitor for Grafana in my home. I can highly recommend doing the same, especially in a world where more work from home is becoming the norm.
Having metrics visible at all times can be incredibly helpful in spotting trends and issues. This is the reason we all have wall mounted Grafanas in our workplaces! Since we are all going to be working from home for the foreseeable future it makes sense to have visible metrics at home as well. Also, muc...
27. apr 2020 21:33 UTC
Webserver log analysis has been a sore point in the opensource world for years. People just use
Google Analytics or whatever so noone is working on tools for this, and I got properly fed up with the old perl monsters like AWStats. So I haven't really done any serious attempts at analyzing the logfiles from my webservers for some years, but recently I started playing around with goaccess which is pretty nice.
However, analyzing logfil...
27. apr 2020 19:49 UTC
This post is about the way
newsyslog(8) names rotated logfiles on
FreeBSD. It explains how to reconfigure
newsyslog(8) to timestamp rotated logs rather than numbering them sequentially. It also explains how I renamed the existing logfiles.
newsyslog(8) takes care of rotating logs based on time or size restrictions defined in
/etc/newsyslog.conf or its included files. The default
04. mar 2020 22:09 UTC
Today I set out to make my mailserver a bit better privacy-wise. I wanted to remove certain headers from client-submitted email (meaning ports 465 (smtps) or 587 (submission), authenticated email from clients, not MX mail from other servers).
The headers I wanted to remove were primarily the first
Received: header (which contains the clients IP, both before and after NAT), and the
X-Originating-IP header (if added by the MUA) which c...
24. jan 2020 13:00 UTC
Sometimes a VM is low on disk and I add some more in the hypervisor. After resizing the block device I still need to resize the
GPT partition with
gpart and I need to tell
ZFS about it. This blog post explains how to extend the partition as well as the
ZFS VDEV. This is mostly relevant in virtual machines, but the
zpool online -e trick can also be used when replacing physical disks with larger ones in a real server.
18. nov 2019 19:10 UTC
This morning I decided to write a
gstat_exporter to make
gstat data available to
Prometheus is a wonderful timeseries database, monitoring tool, and software ecosystem really. It is still new but it is quickly gaining traction all over. For example, FreeBSD 12 got a
prometheus_sysctl_exporter(8) in base, which can output all the values from sysctl in a Prometheus friendly format, and it can even ...
17. nov 2019 16:32 UTC
When database tables grow big it can help a lot to introduce table partitioning. I have been playing around with it and wanted to record my procedure for future reference.
partman in my Poudfiere, but I have not yet had the need for it. The script I use to create and attach partitions can be found in my
Ansible Read More
23. oct 2019 23:15 UTC
I use the code code-formatter Black to format code in some of my repositories, for example Certgrinder and SocialRating. I also use flake8 as a linter/code quality checker. This post is about automating running
flake8 in a git pre-commit hook, so I d...
09. may 2019 21:44 UTC
I've used Hetzner for years as one of the places I run jailhosts. Their physical servers are cheap and stable and their support is great especially when considering the price.
I usually use a server for around two years before I start considering retiring it and switcing to a new one. Then it takes me another 1-2 years to actually do something about it. This particular servers was running an old 10.3-STABLE which is EOL long ago, so it was high time to kill it off.
17. jan 2019 14:17 UTC
Today I was testing out https support on a partner API. They've been using a selfsigned certificate during development, and since they will move to production soon the certificate has been replaced by a CA signed certificate from GoDaddy.
I was still getting certificate errors though, and I suspected that they'd forgotten to include the intermediate certificate from the issuer. I wanted to check locally which intermediate to use so I could send an easily acti...
19. apr 2018 13:30 UTC
FreeBSD got UEFI support not too long ago, including in the installer. This means you can install on servers without BIOS or where boot mode is set to UEFI. Legacy booting is full of surprises and UEFI will hopefully turn out to be a nice replacement. The only problem I've encountered has been recreating the UEFI partition after replacing a disk, which is what this blogpost is about.
The installer takes care of creating the initial partitions, the default layout looks like this: ...
23. mar 2018 09:45 UTC
I have never been a big fan of software using
SYSV IPC shared memory and semaphores. I love PostgreSQL but over the years its use of
SYSV IPC has caused various issues for me. I use Zabbix as well, which makes heavy use of shared memory.
I run all my stuff in FreeBSD jails which used to make things even more complicated, because
SYSV IPC stuff wasn't names...
25. feb 2018 13:34 UTC
SSHFP records has been around for a long time. They were first defined in RFC4255 and conceptually they are very similar to
DANE/TLSA. It is basically a way to pin a hash of the SSH public key in the DNS, to allow clients to verify the public key they see when connecting. This means that I can avoid the unhelpful SSH host fingerprint message that we are all used to seeing when we SSH to a new host.
So instead of seein...
25. feb 2018 13:10 UTC
The nature of Ansible is that it connects to the hosts in its
inventory files over SSH. This opens up some new possibilities since I now have a place which has SSH access to the rest of the infrastructure, which I can use to automate various monitoring scripts.
I've recently added this script to my Ansible roles and wanted to post it here. It is very simple, nothing fancy about it, it just loops over the hostnames found in the Ansible inventory files, SSHs into each ...
05. dec 2017 11:25 UTC
I've been working to replace my
OwnCloud installation with something else. I use the Calendar part of
OwnCloud a lot, and this post is about replacing the
CalDAV bits of
OwnCloud with Radicale, a Python based
supervisord so I will start out by showing the config for those before getting deeper into
02. nov 2017 14:47 UTC
This is the story of a bug I found in
Daphne HTTP and Websocket terminating server, where I had to work around the bug for months while waiting for the fix to make it into a release.
Daphne is part of the
Channels project which is
Djangos cool websocket thing that recently got adopted as a part of the official
Django project. We've been using
Channels for a while on the the Schedule...
19. sep 2017 19:59 UTC
Today I had the pleasure of trying out my new apu2c4. Hit a few snags here and there so I am documenting my experiences for future reference. I have an
apu3 as well, but I haven't played with it yet, so that will have to wait for another blog post.
Qubes installation on my laptop is using the
sys-usb vm thing so the first thing I needed to do was to assign the...
07. jun 2017 05:54 UTC
I am in the process of implementing Certgrinder on all my servers, and as a part of that I am publishing TLSA records for all my services. I haven't been able to do so before, because the normal LetsEncrypt procedure means rolling cert and keys every three months. With Certgrinder I am not rolling the keys when renewing, so I can pin the public keys instead of the certificates in the TLSA records.
Most examples ...
30. apr 2017 12:46 UTC
Like many people I've been switching to LetsEncrypt for my certificate signing needs. I recently changed a bunch of LE related things. This post documents my new method of using the LetsEncrypt
certbot client from a central location, with the certificate consumers (webservers etc) getting their certificates over SSH using a standard CSR. Much like when we were using commercial CAs.
This has a couple of important advantages over my old setup:...
19. nov 2016 13:35 UTC
Earlier this week I was pretty surprised to see some weird permissions on some
nginx config files on my servers. The servers are managed by Ansible so I suspected some changes I made to my ansible roles a few days prior. I only made syntax changes so I didn't expect anything to change. But sometimes the rabbit hole goes deeper than you imagined :)
So I looked at the Ansible task that creates ...
02. oct 2016 12:22 UTC
I spotted a listen queue overflow error message in /var/log/messages on one of my jailhosts today.
I have no idea what could be causing this. This jailhost is busy so it could be a lot of things. Seems to happen about every hour, perhaps some scheduled job?
Most results when you search for the error mention TCP services that can't keep up with the connection rate to the TCP port. But since I didn't know which TCP port I had some detective work to do.
17. mar 2016 10:48 UTC
Djangos database migrations system greatly simplifies the task of keeping multiple databases in sync (the schema, not the data) as the schema evolves over time. Formerly known as South, Djangos migrations system is also a good example of how Django embraces and includes 3rd party packages when it makes sense.
Over the last year I (and colleagues) have been developing a Django-based provisioning system for an ISP. The system has grown rather large with a lot o...
23. jan 2016 20:08 UTC
I was asked to provide a replica of a postgresql server running on FreeBSD. The replica was for reporting purposes, so it must be possible to run read-only queries on it, and it needs to be up-to-date at all times. In postgres land this is called a Hot Standby. Almost the same as a warm standby, except that it accepts connections and read-only queries. The replication had to be done to an Ubuntu 14.04.2 LTS machine running on Amazon. Both postgres servers are version 9...