I have been running my MicroK8s cluster a few years now with a few applications without much issues. It is hosting my web exposed photo galleries and runs the distributed backup solution for example.
Until now it has been sufficient to use public images from Ducker Hub and whatever tweaks or additional packages have been applied without much effort post pod initialization. Now I have containirized a solution I developed for exposing an arbitrary YouTube channel via RSS and noticed that installing the right packages (for example FFmpeg which brings a bunch of mandatory dependencies with it which one can not opt out of) took some time and I did not want to add this overhead to each cron job execution (yes, I run these updates and conversions as Kubernetes CronJobs).
My cluster is running on 6 Raspberry Pi 4s stacked in a tower, but the master node is running on a Ryzer server (and I have some other nodes on Ryzen servers which are labelled according so that they could pick up heavier load if neeed) so that means that the images I use should be available for both arm64 and amd64 architecture.
I was choosing between Docker and Podman for building but since I had Docker installed on the machine I was building on, I went with Docker and more specifically docker buildx. To host the images locally I use the built-in MicroK8s registry addon which can be enabled easily (and allocating 20 GB) with: microk8s enable registry:size=20G
My image requirements are fairly simple, Debian Stable has what I need except Python 3 and some media packages (and yes, procps is for that nice process signaling utility pkill):
FROM debian:stable
RUN <<EOF
apt-get update && apt-get install -y ffmpeg mediainfo wget ca-certificates python3 procps
EOF
Buidling with Docker is typically as easy as “docker build .” but due to the cross-platform image needs I used the following to build for amd64/arm64, tag the image, export as OCI image and upload to my local registry on localhost (I did the build on the same server as where the MicroK8s master node runs and use the default registry port 32000). In order to be able to reach the local registry I had to create a custom builder mybuilder with relaxed security constraints (“docker buildx create --use --name mybuilder --driver-opt network=host --buildkitd-flags '--allow-insecure-entitlement“) and then build using that builder:
docker buildx build -f podcast-stable-image -t localhost:32000/mydebianstable:registry --platform linux/amd64,linux/arm64 . --push --builder mybuilder
In the CronJob description it is then possible to refer to the image with:image: localhost:32000/mydebianstable:registry
The loading is then a matter of seconds instead of minutes.
In order to not upset Google it makes sense to use a sidecar container with vpn connection so you use different IP addresses when accessing their servers.