Tyklings blog Witty but descriptive text goes here

Updating Certgrinder

by Tykling

03. oct 2023 04:37 UTC


As you may know I maintain Certgrinder, a kind of LetsEncrypt SSH proxy thing written in Python. There is a Certgrinder client and a Certgrinderd server and together they make it possible to get LetsEncrypt certificates in places and configurations where it would otherwise be tricky, such as tightly isolated environments.

You can read more about the Certgrinder client and server in Read More


Missing Blog Posts

by Tykling

03. oct 2023 04:28 UTC


In the beginning of 2021, after living my whole life in the city, I (quite literally) bought the farm and moved to the countryside about an hour from Copenhagen. It is great, and I should have done it years ago, but as you might imagine the move has taken up a lot of my time and energy the last couple of years. Since both time and energy are needed for writing blogposts the move has had an unfortunate effect on my blogging frequency. As a consequence it has been way too long since my last po…

Read More


FreeBSD and I2C

by Tykling

06. feb 2021 15:54 UTC


I2C is a communication protocol which was originally designed and used for inter-board communications between peripherals, like the processor talking to an EEPROM in a system. It is used extensively in modern electronics, including laptops, cars etc. These days it is also used for short (well, short to medium - I occationally use 50cm cables) distance communication with sensors - not what it was designed for, but it works surprisingly well anyway. Wikipedia has this to say about…

Read More


Going NUTs: Using a UPS on FreeBSD

by Tykling

25. nov 2020 13:54 UTC


I recently configured an unattended remote computer for monitoring humidity in the 20" shipping container we use for BornHack storage. It uses mobile modems for uplinks and it can tell us all sorts of info about the environment inside the container.

Any time a remote unattended system stops responding it can be impossible to know if it is related to network issues, problems with the OS or hardware, or a power outage. We wanted our monitoring to be able to differentiate between a po…

Read More


Monitoring Modem Signal Strength with Promethus

by Tykling

05. nov 2020 11:34 UTC


I've been playing with mobile modems recently, and found myself in need of a good way to monitor the signal strength of the modems. Standard AT commands can be used to query each modem about its (perceived) signal strength. I want to get these numbers periodically and put them into Prometheus so I can make myself a nice Grafana dashboard showing how my modems are doing.

This innocent sounding task ended up…

Read More


FreeBSD, Multiple LTE Modems, PPP and Multi-FIB on APU3C4

by Tykling

02. nov 2020 22:27 UTC


I recently had to configure a computer for environmental monitoring in the 20" shipping container we use for BornHack storage. The container sits outside in the wind and rain all year. This means we've had to fit it with a dehumidifier to make sure the moisture doesn't destroy all our stuff. Being nerds we want graphs to tell us about the temperature and humidity levels in the container, but since it is sitting in a place with no Internet the first challenge was to get online.

Th…

Read More


FreeBSD Wall Display Computer

by Tykling

10. oct 2020 16:22 UTC


I've recently added a wall mounted 30" monitor for Grafana in my home. I can highly recommend doing the same, especially in a world where more work from home is becoming the norm.

Having metrics visible at all times can be incredibly helpful in spotting trends and issues. This is the reason we all have wall mounted Grafanas in our workplaces! Since we are all going to be working from home for the foreseeable future it makes sense to have visible metrics at home as well. Also, much …

Read More


Structured JSON Logging in Nginx

by Tykling

27. apr 2020 21:33 UTC


Webserver log analysis has been a sore point in the opensource world for years. People just use Google Analytics or whatever so noone is working on tools for this, and I got properly fed up with the old perl monsters like AWStats. So I haven't really done any serious attempts at analyzing the logfiles from my webservers for some years, but recently I started playing around with goaccess which is pretty nice.

However, analyzing logfiles…

Read More


Making newsyslog Timestamp Rotated Logs

by Tykling

27. apr 2020 19:49 UTC


This post is about the way newsyslog(8) names rotated logfiles on FreeBSD. It explains how to reconfigure newsyslog(8) to timestamp rotated logs rather than numbering them sequentially. It also explains how I renamed the existing logfiles.

newsyslog(8) takes care of rotating logs based on time or size restrictions defined in /etc/newsyslog.conf or its included files. The default /etc/newsyslog.conf looks …

Read More


Postfix and Privacy

by Tykling

04. mar 2020 22:09 UTC


Background

Today I set out to make my mailserver a bit better privacy-wise. I wanted to remove certain headers from client-submitted email (meaning ports 465 (smtps) or 587 (submission), authenticated email from clients, not MX mail from other servers).

The headers I wanted to remove were primarily the first Received: header (which contains the clients IP, both before and after NAT), and the X-Originating-IP header (if added by the MUA) which con…

Read More


Expanding Disk Space

by Tykling

24. jan 2020 13:00 UTC


Sometimes a VM is low on disk and I add some more in the hypervisor. After resizing the block device I still need to resize the GPT partition with gpart and I need to tell ZFS about it. This blog post explains how to extend the partition as well as the ZFS VDEV. This is mostly relevant in virtual machines, but the zpool online -e trick can also be used when replacing physical disks with larger ones in a real server.

T…

Read More


All in a days work: Prometheus gstat_exporter and Grafana Dashboard

by Tykling

18. nov 2019 19:10 UTC


Background

This morning I decided to write a gstat_exporter to make gstat data available to Prometheus.

Prometheus is a wonderful timeseries database, monitoring tool, and software ecosystem really. It is still new but it is quickly gaining traction all over. For example, FreeBSD 12 got a prometheus_sysctl_exporter(8) in base, which can output all the values from sysctl in a Prometheus friendly format, and it can even ru…

Read More


Playing with Postgres Partitioning

by Tykling

17. nov 2019 16:32 UTC


Background

When database tables grow big it can help a lot to introduce table partitioning. I have been playing around with it and wanted to record my procedure for future reference.

Partitioning Script

I have partman in my Poudfiere, but I have not yet had the need for it. The script I use to create and attach partitions can be found in my Ansible Read More


Git Hook for Code Formatting with Black and flake8

by Tykling

23. oct 2019 23:15 UTC


I use the code code-formatter Black to format code in some of my repositories, for example Certgrinder and SocialRating. I also use flake8 as a linter/code quality checker. This post is about automating running Black and flake8 in a git pre-commit hook, so I don…

Read More


Retiring a Hetzner Server

by Tykling

09. may 2019 21:44 UTC


I've used Hetzner for years as one of the places I run jailhosts. Their physical servers are cheap and stable and their support is great especially when considering the price.

I usually use a server for around two years before I start considering retiring it and switcing to a new one. Then it takes me another 1-2 years to actually do something about it. This particular servers was running an old 10.3-STABLE which is EOL long ago, so it was high time to kill it off.

Usuall…

Read More


Testing an Intermediate Certificate Locally

by Tykling

17. jan 2019 14:17 UTC


Background

Today I was testing out https support on a partner API. They've been using a selfsigned certificate during development, and since they will move to production soon the certificate has been replaced by a CA signed certificate from GoDaddy.

I was still getting certificate errors though, and I suspected that they'd forgotten to include the intermediate certificate from the issuer. I wanted to check locally which intermediate to use so I could send an easily action…

Read More


FreeBSD and UEFI Boot

by Tykling

19. apr 2018 13:30 UTC


FreeBSD got UEFI support not too long ago, including in the installer. This means you can install on servers without BIOS or where boot mode is set to UEFI. Legacy booting is full of surprises and UEFI will hopefully turn out to be a nice replacement. The only problem I've encountered has been recreating the UEFI partition after replacing a disk, which is what this blogpost is about.

The installer takes care of creating the initial partitions, the default layout looks like this: Read More


FreeBSD, Jails and SYSV IPC

by Tykling

23. mar 2018 09:45 UTC


I have never been a big fan of software using SYSV IPC shared memory and semaphores. I love PostgreSQL but over the years its use of SYSV IPC has caused various issues for me. I use Zabbix as well, which makes heavy use of shared memory.

I run all my stuff in FreeBSD jails which used to make things even more complicated, because SYSV IPC stuff wasn't namespa…

Read More


Getting Serious About SSHFP

by Tykling

25. feb 2018 13:34 UTC


SSHFP records has been around for a long time. They were first defined in RFC4255 and conceptually they are very similar to DANE/TLSA. It is basically a way to pin a hash of the SSH public key in the DNS, to allow clients to verify the public key they see when connecting. This means that I can avoid the unhelpful SSH host fingerprint message that we are all used to seeing when we SSH to a new host.

So instead of seeing:…

Read More


Running pkg audit From Ansible

by Tykling

25. feb 2018 13:10 UTC


The nature of Ansible is that it connects to the hosts in its inventory files over SSH. This opens up some new possibilities since I now have a place which has SSH access to the rest of the infrastructure, which I can use to automate various monitoring scripts.

I've recently added this script to my Ansible roles and wanted to post it here. It is very simple, nothing fancy about it, it just loops over the hostnames found in the Ansible inventory files, SSHs into each (u…

Read More


Radicale on FreeBSD

by Tykling

05. dec 2017 11:25 UTC


I've been working to replace my OwnCloud installation with something else. I use the Calendar part of OwnCloud a lot, and this post is about replacing the CalDAV bits of OwnCloud with Radicale, a Python based CalDAV and CardDAV server.

I use nginx and supervisord so I will start out by showing the config for those before getting deeper into Read More


Working Around Bugs

by Tykling

02. nov 2017 14:47 UTC


This is the story of a bug I found in Djangos Daphne HTTP and Websocket terminating server, where I had to work around the bug for months while waiting for the fix to make it into a release.

Daphne is part of the Channels project which is Djangos cool websocket thing that recently got adopted as a part of the official Django project. We've been using Channels for a while on the the Schedule p…

Read More


FreeBSD on the apu2c4

by Tykling

19. sep 2017 19:59 UTC


Today I had the pleasure of trying out my new apu2c4. Hit a few snags here and there so I am documenting my experiences for future reference. I have an apu3 as well, but I haven't played with it yet, so that will have to wait for another blog post.

Serial console on Qubes

My Qubes installation on my laptop is using the sys-usb vm thing so the first thing I needed to do was to assign the U…

Read More


Generating TLSA Records With OpenSSL

by Tykling

07. jun 2017 05:54 UTC


I am in the process of implementing Certgrinder on all my servers, and as a part of that I am publishing TLSA records for all my services. I haven't been able to do so before, because the normal LetsEncrypt procedure means rolling cert and keys every three months. With Certgrinder I am not rolling the keys when renewing, so I can pin the public keys instead of the certificates in the TLSA records.

Most examples of…

Read More


Introducing Certgrinder, a LetsEncrypt SSH Proxy

by Tykling

30. apr 2017 12:46 UTC


Background

Like many people I've been switching to LetsEncrypt for my certificate signing needs. I recently changed a bunch of LE related things. This post documents my new method of using the LetsEncrypt certbot client from a central location, with the certificate consumers (webservers etc) getting their certificates over SSH using a standard CSR. Much like when we were using commercial CAs.

This has a couple of important advantages over my old setup:

<…

Read More