by Tykling
03. oct 2023 04:37 UTC
As you may know I maintain Certgrinder, a kind of LetsEncrypt SSH proxy thing written in Python. There is a Certgrinder client and a Certgrinderd server and together they make it possible to get LetsEncrypt certificates in places and configurations where it would otherwise be tricky, such as tightly isolated environments.
You can read more about the Certgrinder client and server in the documentation on ReadTheDocs.
The Certgrinder client and server both have a few dependencies, one of the primary ones being Cryptography which is used to parse and validate certificates and keys and such. As with most things Cryptography is a moving target and occationally they change something which means Certgrinder breaks. I also have to switch from building with distutils to building with PEP517. This is all part of being a maintainer and it is to be expected when you maintain software over time. Nothing ever stays the same for long.
The following sections describe the changes I made to certgrinder and certgrinderd before releasing v0.18.0.
The certgrinder and certgrinderd Python packages have both been at version 0.17.2 for a while. Until yesterday they were based on the old distutils build system. Most of the Python world has moved to pep517 for building Python packages, and distutils is deprecated and scheduled to be removed in Python 3.12 (which is being released today). Time to switch Certgrinder to pep517!
The main change when switching to PEP517 is the switch from setup.py to pyproject.toml. The pyproject.toml file defines the build system settings but it can also contain settings for other tools like linters and such. While toml is not a very familiar format to me I still really like the idea of putting everything in one file.
Runtime dependencies are now in the dependencies list which looks like this:
dependencies = [
"dnspython == 2.4.2",
"PyYAML == 6.0.1",
"cryptography == 41.0.4",
"pid == 3.0.4",
]
It is also still possible to specify extras which makes it possible to install the package with extra dependencies if needed. I define three extras in pyproject.toml:
[project.optional-dependencies] dev = ["twine == 4.0.2", "pre-commit == 3.4.0", "setuptools-scm == 8.0.3"] test = ["pytest == 7.4.2", "pytest-cov==4.1.0", "tox == 4.11.3"] docs = ["Sphinx==7.2.6", "sphinx-rtd-theme==1.3.0", "sphinx-argparse==0.4.0"]
To use extras I instruct pip to install them like so if I have the repo checked out:
(venv) user@privat-dev:~/devel/certgrinder/client$ pip install .[dev] Processing /home/user/devel/certgrinder/client Installing build dependencies ... done Getting requirements to build wheel ... done Installing backend dependencies ... done Preparing metadata (pyproject.toml) ... done .....output snipped.....
If I want to install from PyPi I just use the name of the package instead:
(venv) user@privat-dev:~/devel/certgrinder/client$ pip install certgrinder[dev] .....output snipped.....
Building the packages with PEP517 is as easy as it was with distutils. Using the build module it is straightforward:
All software projects have versions. Typically the version number lives in Git tags as well as multiple different files and it can be a bit of a faff to keep them all in sync. Single-sourcing the version has been the dream for a while and having tried a few of the methods described here I absolutely prefer the last suggestion on that page, using setuptools-scm. setuptools-scm extracts Python package versions from Git (or hg) metadata instead of declaring them as the version argument or in a Git managed file. This means the version of my Python package is the latest Git tag plus some extra if there a commits and changes after the latest tag.
Introducing setuptools-scm made my RELEASE.md file is now considerably shorter because I don't have to change version numbers in a bunch of files when doing a release. Yay! Highly recommended.
I have an extensive testsuite for Certgrinder so updating dependencies is usually trivial. I just update to the latest and run my tests and if they all work then it is probably safe to update that dependency. Writing the tests for 100% coverage was a lot of work, but it is really worth it down the line with all the time saved when updating!
To actually update the dependencies I do pip install --upgrade foo and then note the version and update it in the build files. Before PEP517 it was in setup.py and now after PEP517 it is in pyproject.toml. Githubs Dependabot helps a lot by creating the PRs so I just have to merge them and cut a release if something needs updating.
One of the primary dependencies of both certgrinder and certgrinderd is Cryptography which is used for parsing, reading and converting keys and certificates. It is a great project offering a nice level of abstraction for working with crypto stuff in Python. As time goes by stuff inevitably changes in a project with the size and complexity of Cryptography and sometimes that means consumers have to change too. In this case the ed25519.Ed25519PrivateKey stuff moved from cryptography.hazmat.backends.openssl to cryptography.hazmat.primitives so I change the imports accordingly.
The pre-commit package is a very popular framework for running linters as pre-commit hooks in Git. I use it in all my projects and lately I have been leaning towards letting pre-commit handle updating the versions of linters too. This means I can keep them out of pyproject.toml and it also means I get just 1 PR from the pre-commit bot instead of many (one per linter which needs updating) from Dependabot.
I already used pre-commit in this project so this change was as simple as removing all the linters from my dev dependencies. I did this while converting to pyproject.toml so there is no seperate commit to show. After the change my dev deps look nice and clean - just twine for uploading packages to PyPi, pre-commit it self, and setuptools-scm now:
[project.optional-dependencies] dev = ["twine == 4.0.2", "pre-commit == 3.4.0", "setuptools-scm == 8.0.3"]
My pre-commit config looks like this:
repos:
- repo: "https://github.com/asottile/pyupgrade"
rev: "v3.13.0"
hooks:
- id: "pyupgrade"
args: ["--py38-plus"]
- repo: "https://github.com/ambv/black"
rev: "23.9.1"
hooks:
- id: "black"
language_version: "python3.9"
- repo: "https://github.com/pycqa/flake8"
rev: "6.1.0"
hooks:
- id: "flake8"
- repo: "https://github.com/pre-commit/mirrors-mypy"
rev: 'v1.5.1'
hooks:
- id: "mypy"
additional_dependencies: ["types-cryptography", "types-requests", "types-PyYAML"]
name: "mypy (client/certgrinder)"
args: ["--strict"]
files: ^client/
- id: "mypy"
additional_dependencies: ["types-cryptography", "types-requests", "types-PyYAML"]
name: "mypy (server/certgrinderd)"
args: ["--strict"]
files: ^server/
- repo: "https://github.com/pre-commit/mirrors-isort"
rev: "v5.10.1"
hooks:
- id: "isort"
- repo: "https://github.com/pycqa/pydocstyle"
rev: "6.3.0"
hooks:
- id: "pydocstyle"
- repo: "local"
hooks:
- id: "sphinx-build-manpages"
name: "sphinx manpage build"
entry: "make --directory docs/ man"
language: "system"
pass_filenames: False
- id: "sphinx-copy-certgrinder-manpage"
name: "sphinx certgrinder.8 manpage copy"
entry: "cp docs/_build/man/certgrinder.8 client/man/"
language: "system"
pass_filenames: False
- id: "sphinx-copy-certgrinderd-manpage"
name: "sphinx certgrinderd.8 manpage copy"
entry: "cp docs/_build/man/certgrinderd.8 server/man/"
language: "system"
pass_filenames: False
- id: "sphinx-git-add-manpages"
name: "sphinx manpage git add"
entry: "git add client/man server/man"
language: "system"
pass_filenames: False
Note: The last bit under repo: local is for building manpages and has nothing to do with linting.
Now that everything is updated I am ready to tag the new v0.18.0 release. After tagging I need to build and upload the new packages to PyPi.
Under PEP517 I use the build package to build packages. Building is as simple as running python -m build in the folder containing pyproject.toml. The output when I was building v0.18.0 showed a problem though:
(venv) user@privat-dev:~/devel/certgrinder/client$ python -m build
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools, setuptools-scm)
* Getting build dependencies for sdist...
running egg_info
writing certgrinder.egg-info/PKG-INFO
writing dependency_links to certgrinder.egg-info/dependency_links.txt
writing entry points to certgrinder.egg-info/entry_points.txt
writing requirements to certgrinder.egg-info/requires.txt
writing top-level names to certgrinder.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'certgrinder.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing certgrinder.egg-info/PKG-INFO
writing dependency_links to certgrinder.egg-info/dependency_links.txt
writing entry points to certgrinder.egg-info/entry_points.txt
writing requirements to certgrinder.egg-info/requires.txt
writing top-level names to certgrinder.egg-info/top_level.txt
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'certgrinder.egg-info/SOURCES.txt'
running check
creating certgrinder-0.18.0
creating certgrinder-0.18.0/certgrinder
creating certgrinder-0.18.0/certgrinder.egg-info
creating certgrinder-0.18.0/certgrinder/tests
creating certgrinder-0.18.0/man
copying files to certgrinder-0.18.0...
copying LICENSE -> certgrinder-0.18.0
copying MANIFEST.in -> certgrinder-0.18.0
copying README.md -> certgrinder-0.18.0
copying pyproject.toml -> certgrinder-0.18.0
copying certgrinder/__init__.py -> certgrinder-0.18.0/certgrinder
copying certgrinder/_version.py -> certgrinder-0.18.0/certgrinder
copying certgrinder/certgrinder.conf.dist -> certgrinder-0.18.0/certgrinder
copying certgrinder/certgrinder.py -> certgrinder-0.18.0/certgrinder
copying certgrinder.egg-info/PKG-INFO -> certgrinder-0.18.0/certgrinder.egg-info
copying certgrinder.egg-info/SOURCES.txt -> certgrinder-0.18.0/certgrinder.egg-info
copying certgrinder.egg-info/dependency_links.txt -> certgrinder-0.18.0/certgrinder.egg-info
copying certgrinder.egg-info/entry_points.txt -> certgrinder-0.18.0/certgrinder.egg-info
copying certgrinder.egg-info/requires.txt -> certgrinder-0.18.0/certgrinder.egg-info
copying certgrinder.egg-info/top_level.txt -> certgrinder-0.18.0/certgrinder.egg-info
copying certgrinder/tests/__init__.py -> certgrinder-0.18.0/certgrinder/tests
copying certgrinder/tests/test_certgrinder.py -> certgrinder-0.18.0/certgrinder/tests
copying man/certgrinder.8 -> certgrinder-0.18.0/man
Writing certgrinder-0.18.0/setup.cfg
Creating tar archive
removing 'certgrinder-0.18.0' (and everything under it)
* Building wheel from sdist
* Creating virtualenv isolated environment...
* Installing packages in isolated environment... (setuptools, setuptools-scm)
* Getting build dependencies for wheel...
running egg_info
writing certgrinder.egg-info/PKG-INFO
writing dependency_links to certgrinder.egg-info/dependency_links.txt
writing entry points to certgrinder.egg-info/entry_points.txt
writing requirements to certgrinder.egg-info/requires.txt
writing top-level names to certgrinder.egg-info/top_level.txt
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file 'certgrinder.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'certgrinder.egg-info/SOURCES.txt'
* Installing packages in isolated environment... (wheel)
* Building wheel...
running bdist_wheel
running build
running build_py
creating build
creating build/lib
creating build/lib/certgrinder
copying certgrinder/_version.py -> build/lib/certgrinder
copying certgrinder/certgrinder.py -> build/lib/certgrinder
copying certgrinder/__init__.py -> build/lib/certgrinder
running egg_info
writing certgrinder.egg-info/PKG-INFO
writing dependency_links to certgrinder.egg-info/dependency_links.txt
writing entry points to certgrinder.egg-info/entry_points.txt
writing requirements to certgrinder.egg-info/requires.txt
writing top-level names to certgrinder.egg-info/top_level.txt
ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any
reading manifest file 'certgrinder.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'certgrinder.egg-info/SOURCES.txt'
/tmp/build-env-lkrf4mfl/lib/python3.9/site-packages/setuptools/command/build_py.py:204: _Warning: Package 'certgrinder.tests' is absent from the `packages` configuration.
!!
********************************************************************************
############################
# Package would be ignored #
############################
Python recognizes 'certgrinder.tests' as an importable package[^1],
but it is absent from setuptools' `packages` configuration.
This leads to an ambiguous overall configuration. If you want to distribute this
package, please make sure that 'certgrinder.tests' is explicitly added
to the `packages` configuration field.
Alternatively, you can also rely on setuptools' discovery methods
(for example by using `find_namespace_packages(...)`/`find_namespace:`
instead of `find_packages(...)`/`find:`).
You can read more about "package discovery" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
If you don't want 'certgrinder.tests' to be distributed and are
already explicitly excluding 'certgrinder.tests' via
`find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
you can try to use `exclude_package_data`, or `include-package-data=False` in
combination with a more fine grained `package-data` configuration.
You can read more about "package data files" on setuptools documentation page:
- https://setuptools.pypa.io/en/latest/userguide/datafiles.html
[^1]: For Python, any directory (with suitable naming) can be imported,
even if it does not contain any `.py` files.
On the other hand, currently there is no concept of package data
directory, all directories are treated like packages.
********************************************************************************
!!
check.warn(importable)
copying certgrinder/certgrinder.conf.dist -> build/lib/certgrinder
creating build/lib/certgrinder/tests
copying certgrinder/tests/__init__.py -> build/lib/certgrinder/tests
copying certgrinder/tests/test_certgrinder.py -> build/lib/certgrinder/tests
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/certgrinder
copying build/lib/certgrinder/_version.py -> build/bdist.linux-x86_64/wheel/certgrinder
copying build/lib/certgrinder/certgrinder.py -> build/bdist.linux-x86_64/wheel/certgrinder
creating build/bdist.linux-x86_64/wheel/certgrinder/tests
copying build/lib/certgrinder/tests/__init__.py -> build/bdist.linux-x86_64/wheel/certgrinder/tests
copying build/lib/certgrinder/tests/test_certgrinder.py -> build/bdist.linux-x86_64/wheel/certgrinder/tests
copying build/lib/certgrinder/certgrinder.conf.dist -> build/bdist.linux-x86_64/wheel/certgrinder
copying build/lib/certgrinder/__init__.py -> build/bdist.linux-x86_64/wheel/certgrinder
running install_egg_info
Copying certgrinder.egg-info to build/bdist.linux-x86_64/wheel/certgrinder-0.18.0-py3.9.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/certgrinder-0.18.0.dist-info/WHEEL
creating '/home/user/devel/certgrinder/client/dist/.tmp-saz0mezc/certgrinder-0.18.0-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'certgrinder/__init__.py'
adding 'certgrinder/_version.py'
adding 'certgrinder/certgrinder.conf.dist'
adding 'certgrinder/certgrinder.py'
adding 'certgrinder/tests/__init__.py'
adding 'certgrinder/tests/test_certgrinder.py'
adding 'certgrinder-0.18.0.dist-info/LICENSE'
adding 'certgrinder-0.18.0.dist-info/METADATA'
adding 'certgrinder-0.18.0.dist-info/WHEEL'
adding 'certgrinder-0.18.0.dist-info/entry_points.txt'
adding 'certgrinder-0.18.0.dist-info/top_level.txt'
adding 'certgrinder-0.18.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built certgrinder-0.18.0.tar.gz and certgrinder-0.18.0-py3-none-any.whl
(venv) user@privat-dev:~/devel/certgrinder/client$
Including unit tests in packages is not recommended, so rather than including certgrinder.tests I want to exclude it.
Package discovery can be a bit fiddly. Setuptools has both auto-discovery and its related options, as well as different ways to specify packages manually. The issue in this case was the certgrinder.tests package being included as a sub-package when building the certgrinder package.
Both the certgrinder and the certgrinderd packages use what setuptools calls a flat layout (where the package directory is in the same directory as pyproject.toml) as opposed to an src layout (where the package directory is inside an src directory in the same directory as pyproject.toml). Both these layouts are supported by setuptools autodiscovery, which comes preloaded with an exclude list which includes tests and much more. This means that by simply switching to setuptools autodiscovery I get my unit tests excluded automatically. I did have to exclude the man directory where the man-pages live, but that was simple enough:
[tool.setuptools.packages.find] exclude = ["man*"]
After fixing this the build is less complain-y:
(venv) user@privat-dev:~/devel/certgrinder/client$ python -m build * Creating virtualenv isolated environment... * Installing packages in isolated environment... (setuptools, setuptools-scm) * Getting build dependencies for sdist... running egg_info writing certgrinder.egg-info/PKG-INFO writing dependency_links to certgrinder.egg-info/dependency_links.txt writing entry points to certgrinder.egg-info/entry_points.txt writing requirements to certgrinder.egg-info/requires.txt writing top-level names to certgrinder.egg-info/top_level.txt reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'certgrinder.egg-info/SOURCES.txt' * Building sdist... running sdist running egg_info writing certgrinder.egg-info/PKG-INFO writing dependency_links to certgrinder.egg-info/dependency_links.txt writing entry points to certgrinder.egg-info/entry_points.txt writing requirements to certgrinder.egg-info/requires.txt writing top-level names to certgrinder.egg-info/top_level.txt reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'certgrinder.egg-info/SOURCES.txt' running check creating certgrinder-0.18.1.dev7+g822fbd5 creating certgrinder-0.18.1.dev7+g822fbd5/certgrinder creating certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info creating certgrinder-0.18.1.dev7+g822fbd5/certgrinder/tests creating certgrinder-0.18.1.dev7+g822fbd5/man copying files to certgrinder-0.18.1.dev7+g822fbd5... copying LICENSE -> certgrinder-0.18.1.dev7+g822fbd5 copying MANIFEST.in -> certgrinder-0.18.1.dev7+g822fbd5 copying README.md -> certgrinder-0.18.1.dev7+g822fbd5 copying pyproject.toml -> certgrinder-0.18.1.dev7+g822fbd5 copying certgrinder/__init__.py -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder copying certgrinder/_version.py -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder copying certgrinder/certgrinder.conf.dist -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder copying certgrinder/certgrinder.py -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder copying certgrinder.egg-info/PKG-INFO -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info copying certgrinder.egg-info/SOURCES.txt -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info copying certgrinder.egg-info/dependency_links.txt -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info copying certgrinder.egg-info/entry_points.txt -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info copying certgrinder.egg-info/requires.txt -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info copying certgrinder.egg-info/top_level.txt -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder.egg-info copying certgrinder/tests/__init__.py -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder/tests copying certgrinder/tests/test_certgrinder.py -> certgrinder-0.18.1.dev7+g822fbd5/certgrinder/tests copying man/certgrinder.8 -> certgrinder-0.18.1.dev7+g822fbd5/man Writing certgrinder-0.18.1.dev7+g822fbd5/setup.cfg Creating tar archive removing 'certgrinder-0.18.1.dev7+g822fbd5' (and everything under it) * Building wheel from sdist * Creating virtualenv isolated environment... * Installing packages in isolated environment... (setuptools, setuptools-scm) * Getting build dependencies for wheel... running egg_info writing certgrinder.egg-info/PKG-INFO writing dependency_links to certgrinder.egg-info/dependency_links.txt writing entry points to certgrinder.egg-info/entry_points.txt writing requirements to certgrinder.egg-info/requires.txt writing top-level names to certgrinder.egg-info/top_level.txt ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any reading manifest file 'certgrinder.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'certgrinder.egg-info/SOURCES.txt' * Installing packages in isolated environment... (wheel) * Building wheel... running bdist_wheel running build running build_py creating build creating build/lib creating build/lib/certgrinder copying certgrinder/_version.py -> build/lib/certgrinder copying certgrinder/certgrinder.py -> build/lib/certgrinder copying certgrinder/__init__.py -> build/lib/certgrinder creating build/lib/certgrinder/tests copying certgrinder/tests/__init__.py -> build/lib/certgrinder/tests copying certgrinder/tests/test_certgrinder.py -> build/lib/certgrinder/tests running egg_info writing certgrinder.egg-info/PKG-INFO writing dependency_links to certgrinder.egg-info/dependency_links.txt writing entry points to certgrinder.egg-info/entry_points.txt writing requirements to certgrinder.egg-info/requires.txt writing top-level names to certgrinder.egg-info/top_level.txt ERROR setuptools_scm._file_finders.git listing git files failed - pretending there aren't any reading manifest file 'certgrinder.egg-info/SOURCES.txt' reading manifest template 'MANIFEST.in' adding license file 'LICENSE' writing manifest file 'certgrinder.egg-info/SOURCES.txt' copying certgrinder/certgrinder.conf.dist -> build/lib/certgrinder installing to build/bdist.linux-x86_64/wheel running install running install_lib creating build/bdist.linux-x86_64 creating build/bdist.linux-x86_64/wheel creating build/bdist.linux-x86_64/wheel/certgrinder copying build/lib/certgrinder/_version.py -> build/bdist.linux-x86_64/wheel/certgrinder copying build/lib/certgrinder/certgrinder.py -> build/bdist.linux-x86_64/wheel/certgrinder creating build/bdist.linux-x86_64/wheel/certgrinder/tests copying build/lib/certgrinder/tests/__init__.py -> build/bdist.linux-x86_64/wheel/certgrinder/tests copying build/lib/certgrinder/tests/test_certgrinder.py -> build/bdist.linux-x86_64/wheel/certgrinder/tests copying build/lib/certgrinder/certgrinder.conf.dist -> build/bdist.linux-x86_64/wheel/certgrinder copying build/lib/certgrinder/__init__.py -> build/bdist.linux-x86_64/wheel/certgrinder running install_egg_info Copying certgrinder.egg-info to build/bdist.linux-x86_64/wheel/certgrinder-0.18.1.dev7+g822fbd5-py3.9.egg-info running install_scripts creating build/bdist.linux-x86_64/wheel/certgrinder-0.18.1.dev7+g822fbd5.dist-info/WHEEL creating '/home/user/devel/certgrinder/client/dist/.tmp-9w4umatk/certgrinder-0.18.1.dev7+g822fbd5-py3-none-any.whl' and adding 'build/bdist.linux-x86_64/wheel' to it adding 'certgrinder/__init__.py' adding 'certgrinder/_version.py' adding 'certgrinder/certgrinder.conf.dist' adding 'certgrinder/certgrinder.py' adding 'certgrinder/tests/__init__.py' adding 'certgrinder/tests/test_certgrinder.py' adding 'certgrinder-0.18.1.dev7+g822fbd5.dist-info/LICENSE' adding 'certgrinder-0.18.1.dev7+g822fbd5.dist-info/METADATA' adding 'certgrinder-0.18.1.dev7+g822fbd5.dist-info/WHEEL' adding 'certgrinder-0.18.1.dev7+g822fbd5.dist-info/entry_points.txt' adding 'certgrinder-0.18.1.dev7+g822fbd5.dist-info/top_level.txt' adding 'certgrinder-0.18.1.dev7+g822fbd5.dist-info/RECORD' removing build/bdist.linux-x86_64/wheel Successfully built certgrinder-0.18.1.dev7+g822fbd5.tar.gz and certgrinder-0.18.1.dev7+g822fbd5-py3-none-any.whl (venv) user@privat-dev:~/devel/certgrinder/client$
Note setuptools-scm doing good work here, latest tag on this branch is v0.18.0 so setuptools-scm bumps the patch version and also includes the information that there has been 7 commits since the v0.18.0 tag, and the latest has the hash g822fbd5. Good stuff!
After fixing this I tagged v0.18.1 with the fixes and uploaded it to PyPi.
After building the packages need to be made available on PyPi. I use twine to upload, it usually works great. Since I used it last they made a change so accounts with 2fa activated need to use an API token for uploads. Or maybe I just activated 2fa on my account since last time. Anyway, I went and logged into PyPi and was able to create two new API tokens for certgrinder and certgrinderd.
Uploading to PyPi looks like this:
(venv) user@privat-dev:~/devel/certgrinder/client$ twine upload dist/certgrinder-0.18.1.tar.gz dist/certgrinder-0.18.1-py3-none-any.whl Uploading distributions to https://upload.pypi.org/legacy/ Enter your username: __token__ Enter your password: Uploading certgrinder-0.18.1-py3-none-any.whl 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.0/41.0 kB • 00:00 • 158.0 MB/s Uploading certgrinder-0.18.1.tar.gz 100% ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.8/53.8 kB • 00:00 • 185.4 MB/s View at: https://pypi.org/project/certgrinder/0.18.1/ (venv) user@privat-dev:~/devel/certgrinder/client$
I repeat the same for the certgrinderd package and then the Python part of the job is done.
The FreeBSD ports for certgrinder and certgrinderd are maintained outside the official ports tree. I would like to get them into the official tree but for now it hasn't happened.
FreeBSD porting is centered around Makefiles, and the ports collection is so vast that examples for almost anything are readily available. In case of questions I consult the FreeBSD Porters Handbook which has plenty of answers. For Python specific questions there is also #freebsd-python on Libera (IRC) which is always helpful.
The changes needed to switch a FreeBSD port from building with distutils to PEP517 are pretty trivial. I changed the USE_PYTHON line to include pep517 instead of distutils and of course update the actual versions of the package and dependencies. I also update the distinfo file which contains size and checksums for the source distribution files.
The complete diff looks like this:
(venv) user@privat-dev:~/devel/tykports$ PAGER= git diff 0f9b41f4ac4041104cedf8025c81225eb82c5d0c..324fae9c4a0cfa6cc673333b2e3eca69e4bfda17
diff --git a/sysutils/py-certgrinder/Makefile b/sysutils/py-certgrinder/Makefile
index 95b5a39..01f24e8 100644
--- a/sysutils/py-certgrinder/Makefile
+++ b/sysutils/py-certgrinder/Makefile
@@ -2,10 +2,9 @@
# $FreeBSD$
PORTNAME= certgrinder
-PORTVERSION= 0.17.2
-PORTREVISION= 16
+PORTVERSION= 0.18.1
CATEGORIES= sysutils python
-MASTER_SITES= CHEESESHOP
+MASTER_SITES= PYPI
PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX}
MAINTAINER= thomas@gibfest.dk
@@ -14,14 +13,17 @@ COMMENT= Client for getting LetsEncrypt certificates from a Certgrinderd server
LICENSE= BSD3CLAUSE
LICENSE_FILE= ${WRKSRC}/LICENSE
-RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}cryptography>=3.3.2:security/py-cryptography@${PY_FLAVOR} \
- ${PYTHON_PKGNAMEPREFIX}dnspython>=1.16.0:dns/py-dnspython@${PY_FLAVOR} \
+BUILD_DEPENDS= ${PYTHON_PKGNAMEPREFIX}setuptools>0:devel/py-setuptools@${PY_FLAVOR} \
+ ${PYTHON_PKGNAMEPREFIX}wheel>0:devel/py-wheel@${PY_FLAVOR} \
+ ${PYTHON_PKGNAMEPREFIX}setuptools_scm>0:devel/py-setuptools_scm@${PY_FLAVOR}
+
+RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}cryptography>=41.0.4:security/py-cryptography@${PY_FLAVOR} \
+ ${PYTHON_PKGNAMEPREFIX}dnspython>=2.4.2:dns/py-dnspython@${PY_FLAVOR} \
${PYTHON_PKGNAMEPREFIX}pid>=3.0.4:devel/py-pid@${PY_FLAVOR} \
- ${PYTHON_PKGNAMEPREFIX}yaml>=5.4.1:devel/py-yaml@${PY_FLAVOR}
+ ${PYTHON_PKGNAMEPREFIX}yaml>=6.0:devel/py-yaml@${PY_FLAVOR}
-# 3.7, 3.8, 3.9, 3.10
-USES= python:3.7+
-USE_PYTHON= autoplist distutils
+USES= python:3.8+
+USE_PYTHON= autoplist pep517
USERS= certgrinder
GROUPS= certgrinder
diff --git a/sysutils/py-certgrinder/distinfo b/sysutils/py-certgrinder/distinfo
index 953bca6..f3af624 100644
--- a/sysutils/py-certgrinder/distinfo
+++ b/sysutils/py-certgrinder/distinfo
@@ -1,3 +1,3 @@
-TIMESTAMP = 1638043424
-SHA256 (certgrinder-0.17.2.tar.gz) = b96590e7e9c5909e6ef35fdc8a16e8be4bee6dc8c12852947172ebedde9cfa33
-SIZE (certgrinder-0.17.2.tar.gz) = 35267
+TIMESTAMP = 1697010638
+SHA256 (certgrinder-0.18.1.tar.gz) = 9df6315f6f19ee40d06325244183769ba947bbdc4a323b43ad35206c8cd26654
+SIZE (certgrinder-0.18.1.tar.gz) = 47602
diff --git a/sysutils/py-certgrinderd/Makefile b/sysutils/py-certgrinderd/Makefile
index 990811a..4de5b14 100644
--- a/sysutils/py-certgrinderd/Makefile
+++ b/sysutils/py-certgrinderd/Makefile
@@ -2,10 +2,9 @@
# $FreeBSD$
PORTNAME= certgrinderd
-PORTVERSION= 0.17.2
-PORTREVISION= 8
+PORTVERSION= 0.18.1
CATEGORIES= sysutils python
-MASTER_SITES= CHEESESHOP
+MASTER_SITES= PYPI
PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX}
MAINTAINER= thomas@gibfest.dk
@@ -14,12 +13,15 @@ COMMENT= Server for getting LetsEncrypt certificates for Certgrinder clients
LICENSE= BSD3CLAUSE
LICENSE_FILE= ${WRKSRC}/LICENSE
+BUILD_DEPENDS= ${PYTHON_PKGNAMEPREFIX}setuptools>0:devel/py-setuptools@${PY_FLAVOR} \
+ ${PYTHON_PKGNAMEPREFIX}wheel>0:devel/py-wheel@${PY_FLAVOR} \
+ ${PYTHON_PKGNAMEPREFIX}setuptools_scm>0:devel/py-setuptools_scm@${PY_FLAVOR}
+
RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}certbot>=${ACME_VERSION},1:security/py-certbot@${PY_FLAVOR} \
- ${PYTHON_PKGNAMEPREFIX}yaml>=5.4.1:devel/py-yaml@${PY_FLAVOR}
+ ${PYTHON_PKGNAMEPREFIX}yaml>=6.0:devel/py-yaml@${PY_FLAVOR}
-# 3.7, 3.8, 3.9, 3.10
-USES= python:3.7+
-USE_PYTHON= autoplist distutils
+USES= python:3.8+
+USE_PYTHON= autoplist pep517
USERS= certgrinderd
GROUPS= certgrinderd
diff --git a/sysutils/py-certgrinderd/distinfo b/sysutils/py-certgrinderd/distinfo
index ffe9c6e..839f924 100644
--- a/sysutils/py-certgrinderd/distinfo
+++ b/sysutils/py-certgrinderd/distinfo
@@ -1,3 +1,3 @@
-TIMESTAMP = 1638043512
-SHA256 (certgrinderd-0.17.2.tar.gz) = 5a4254ace06dbda5b19ab4d2eb9de2006b2bb6c3b72d294c4376aa9215508062
-SIZE (certgrinderd-0.17.2.tar.gz) = 22499
+TIMESTAMP = 1697010646
+SHA256 (certgrinderd-0.18.1.tar.gz) = 7838f144d37db5c8d7b520e0160cc7bda69aa9cf3b3093c0c8e62b45785d899b
+SIZE (certgrinderd-0.18.1.tar.gz) = 27939
(venv) user@privat-dev:~/devel/tykports$
After a test build I committed the changes and started the build on my Poudriere box. I have since updated my own infrastructure and both the client and the server port work as intended.
I really like the way the Python ecosystem has been maturing over the last few years. PEP517 and getting the build and packaging stuff streamlined was way overdue. Using setuptools-scm is lovely. New linters are also coming out, I am looking at switching to Ruff which can replace the linters I am currently using, do more stuff, and is way faster. Win/win.
That is it. I think I've covered everything. In some ways this was a trivial change, it was basically just an import which moved. But it was also time to get the PEP517 conversion done. All in all it was a bit of work but not too bad. Hopefully it will be a while before I have to update again.
I've recently signed up for Github Sponsors meaning it is now easy to sponsor me and my work. If this post or some of my other writing, software or services have helped you then you can consider becoming a sponsor.