Projects
Projects

Custom BootC OS Images Project

Linux
AlmaLinux
bootc
Podman
KVM
ZFS
CI/CD
Secure Boot
DevOps

Developed custom container-native operating system images based on the bootc standard for hyperconverged infrastructure, workstations, and desktop environments.

A server rack with various network and storage devices, representing hyperconverged infrastructure.

Project Overview

This project builds customized, bootc-compatible container images for container-native infrastructure. By utilizing the bootc standard, the operating system is treated as a container image, allowing declarative configuration, automated updates, and atomic rollbacks.

The custom images target different infrastructure cases: general-purpose servers, hyperconverged infrastructure (HCI) hosts, workstation/laptop setups, docker hosts, and systemd-managed desktop environments.

Technical Architecture & Design Principles

Container-Native OS & bootc

Instead of traditional package-based OS deployments, this project uses a container image as the system source.

  • Atomic OS Lifecycle: System updates are performed via container registry fetches and atomic OS-tree updates. If an update fails, a system rollback is performed on the next boot.
  • Declarative Configuration: File configurations are defined in the build steps and copied into the final root filesystem overlay.

Out-of-Tree Kernel Modules & Secure Boot

Deploying copy-on-write filesystems like ZFS requires custom kernel modules that are built out-of-tree.

  • MOK Key Signing: The project generates a Machine Owner Key (MOK) pair under keys/mok/ to sign kernel modules. This preserves Secure Boot compatibility.
  • Custom Module Compilation: The kmods.Containerfile compiles OpenZFS kernel modules for the exact kernel version matching the target base image.

Image Variants & Features

1. Base AlmaLinux bootc

The general-purpose server image is built on AlmaLinux 10 (Containerfile).

  • Base Stack: Includes Tailscale VPN, Cockpit web management interfaces, and backup utilities like Borgmatic, Rclone, and Rsync.
  • Security Shield: Hardens the system with Firewalld, Wazuh security agent monitoring, and CrowdSec intrusion prevention with nftables firewall bouncers.

2. Hyperconverged Infrastructure (HCI)

Built on the base AlmaLinux bootc image (hci.Containerfile).

  • Virtualization Stack: Installs QEMU/KVM virtualization packages, libvirt daemon, and cockpit-machines for managing virtual machines.
  • Storage Integration: Incorporates Btrfs tools and OpenZFS module loading for high-density storage pools. It integrates the 45Drives cockpit-zfs-manager, cockpit-identities, and cockpit-file-sharing plugins.

3. Workstation & Laptop

Built on Bazzite (workstation.Containerfile and laptop.Containerfile).

  • Base Stack: Leverages Bazzite 44 for workstation features.
  • Services: Includes preconfigured developer mode tools, Tailscale, Firewalld, and automated Borgmatic backup tasks.

4. Docker Host

Provides a dedicated container host environment (dockerhost.Containerfile).

  • Base Stack: Built on the base AlmaLinux bootc image.
  • Services: Preconfigures Docker CE, Docker Compose, and integrates a Cockpit-based Docker management panel.

5. Webtop Containerized Desktop

A Fedora KDE desktop environment inside a container (webtop.Containerfile).

  • WebRTC Streaming: Runs Pixelflux Wayland (Selkies) to stream the desktop to a web browser on port 3000.
  • systemd Orchestration: Runs systemd as process ID 1 inside the container to manage user sessions and portal services.
  • Unprivileged Mode: Configured to run without the --privileged container flag by removing restricted kernel capabilities, ensuring isolation.
  • Quadlet Integration: Managed via systemd Quadlets, enabling the container to run automatically as a native service.

Automation & Build Orchestration

The workspace includes a justfile containing recipes to automate development and testing workflows.

graph TD
    A[just build-disk-image] --> B[bootc-image-builder]
    B --> C[qcow2 / ISO Output]
    C --> D[just test-vm]
    D --> E[QEMU/KVM Test VM]
    E --> F[VNC Console Validation]
  • regen-mok: Generates new Machine Owner Keys for Secure Boot module signing.
  • build-disk-image: Utilizes bootc-image-builder to generate a bootable QEMU raw disk image (qcow2) based on the custom container image.
  • build-iso: Creates a bootable installation ISO installer with a Btrfs root filesystem.
  • test-vm: Automates testing by creating an ephemeral virtual machine inside virt-install using the built qcow2 image. It opens a VNC terminal for validation and removes the virtual machine on exit.

Continuous Integration & Delivery

The project utilizes GitHub Actions workflows to automate multi-architecture container builds.

  • Kernel Verification: The build workflow checks for kernel mismatches between the upstream base image and the compiled ZFS modules. If a mismatch is detected, it triggers the module rebuild workflow.
  • Multi-Architecture Builds: Build processes compile amd64 and arm64 image variants concurrently using runner machines.
  • Manifest Creation: Combines the compiled images into a single multi-arch manifest.
  • Cryptographic Signing: Uses cosign to sign image manifests before pushing them to the container registry (ghcr.io).

Key Project Accomplishments

  • Container-Native Infrastructure: Standardized operating system packaging and deployment using container standards.
  • Secure Boot Support: Developed automated module compilation and signing procedures using custom MOK certificates.
  • Intrusion Prevention: Hardened server deployments with default firewall rules, Wazuh endpoint agent integration, and active CrowdSec tracking.
  • Automated Image Compilations: Implemented continuous integration workflows to compile and sign multi-arch system images.