🔥 A revolutionary git-ops-based, multi-tenant hypervisor platform built on systemd technologies for secure, isolated, and declaratively managed virtualization environments.
BBOS is a fully customizable immutable distribution implementing the concepts described in Fitting Everything Together.
The crucial difference that makes BBOS unique compared to other immutable distributions is that users build the BBOS image themselves and sign it with their own keys instead of installing vendor signed images. This allows configuring the image to your liking by having full control over which distribution is used as the base and which packages are installed into the image.
BitBuilder Hypervisor is a next-generation hypervisor system that leverages the full power of systemd's virtualization and containerization capabilities to provide secure, isolated multi-tenant environments. Each tenant's configuration is version-controlled in Git repositories, enabling true infrastructure-as-code workflows with automatic provisioning and updates at boot time.
| Feature | Description | Benefits |
|---|---|---|
| 📚 Git-Ops Native | All configurations stored in Git repositories, pulled at boot time | Version control, atomic deployments, rollback capability |
| 🏠 Multi-Tenant Isolation | Complete separation between tenants using systemd's security features | Zero trust architecture, cryptographic boundaries |
| 🛡️ Immutable Host OS | Read-only host system with all changes applied through git-controlled overlays | Eliminates configuration drift, enhanced security |
| 🎯 Declarative Configuration | Everything defined as code, no manual configuration required | Infrastructure as Code, automated provisioning |
| 🔐 Zero-Trust Architecture | Each tenant runs in completely isolated environments | Hardware-backed security, namespace isolation |
| ⚡ Dynamic Provisioning | Automatic tenant setup and teardown based on git repository state | Scalable operations, automated lifecycle management |
| Component | Purpose | Features |
|---|---|---|
| 🚀 systemd-boot | UEFI boot manager for secure boot chain | Secure boot, UKI support, boot management |
| ⬇️ systemd-import-generator | Automatic EFI image download and boot management | DDI support, automatic updates, verification |
| 🖥️ systemd-vmspawn | Lightweight VM spawning for tenant isolation | Hardware virtualization, resource isolation |
| 📦 systemd-nspawn | Container spawning for application isolation | Namespace isolation, security boundaries |
| 🌐 systemd-networkd | Network configuration and isolation | Software-defined networking, tenant isolation |
| 🏠 systemd-homed | User home directory management with encryption | Encrypted homes, hardware-backed auth |
| 🎁 Portable Services | Self-contained service images | Application portability, dependency isolation |
| 📋 sysext/confext | System and configuration extensions via overlayfs | Layered filesystems, immutable updates |
| 🔗 Varlink | IPC protocol for service communication | Type-safe IPC, service discovery |
BitBuilder Hypervisor implements and adheres to the Linux Userspace API Specifications:
| Specification | Purpose | Implementation |
|---|---|---|
| 📄 Configuration Files Specification | Standardized locations for configuration management | /etc, /usr/lib, /run hierarchy |
| 💽 Discoverable Partitions Specification (DPS) | Auto-discovery of partition semantics | GPT partition type GUIDs |
| 📀 Discoverable Disk Image (DDI) | Self-describing system images | Unified disk images with metadata |
| 🔧 Extension Image | Layered extensions to base images | Overlay filesystem extensions |
| ├── sysext | System Extension Images overlaid on /usr/ and /opt/ |
Runtime system extensions |
| └── confext | Configuration Extension Images overlaid on /etc/ |
Runtime configuration extensions |
| 🚀 Unified Kernel Image (UKI) | Combined kernel, initrd, and boot configuration | Single-file bootable images |
| 🥾 Boot Loader Specification (BLS) | Standardized boot configuration | systemd-boot integration |
| Component | Description | Technology |
|---|---|---|
| 🛡️ Immutable Host | The host OS boots from an immutable image downloaded via systemd-import-generator | DDI images, systemd-import |
| 🏠 Tenant Isolation | Each tenant runs in isolated systemd-vmspawn/nspawn instances | Hardware virtualization, namespaces |
| 📚 Git-Based Configuration | All tenant configurations pulled from dedicated Git repositories | Git-ops, declarative config |
| 🗂️ Dynamic Mount Generation | Custom systemd generators create tenant-specific mount points | systemd generators, overlay mounts |
| 🔧 Layered Extensions | System capabilities extended through sysext/confext layers | Extension images, overlayfs |
Each tenant in the BitBuilder Hypervisor follows a complete lifecycle management approach:
graph LR
A[📚 Git Repository] --> B[🔍 Discovery]
B --> C[⚡ Provisioning]
C --> D[🖥️ VM/Container]
D --> E[🔄 Updates]
E --> A
style A fill:#e3f2fd
style B fill:#f3e5f5
style C fill:#e8f5e8
style D fill:#fff3e0
style E fill:#fce4ec
| Step | Process | Details |
|---|---|---|
| 1️⃣ | Dedicated Git Repository | Each tenant has its own configuration repository |
| 2️⃣ | Automatic Provisioning | Boot-time provisioning via setup-tenant@<tenantname>.service |
| 3️⃣ | Isolated Execution | Runs in systemd-vmspawn or systemd-nspawn instances |
| 4️⃣ | Custom Extensions | Can have tenant-specific sysext/confext layers |
| 5️⃣ | Network Isolation | Complete network isolation via systemd-networkd |
| 6️⃣ | Automatic Updates | Updates automatically when Git repository changes |
sequenceDiagram
participant UEFI as 🔌 UEFI Boot
participant Boot as 🚀 systemd-boot
participant Import as ⬇️ systemd-import-gen
participant Host as 🖥️ Host OS
participant Git as 📚 Git Sync
participant Discovery as 🔍 Tenant Discovery
participant Provision as ⚡ Tenant Provisioning
UEFI->>Boot: Load boot configuration
Boot->>Import: Download and verify EFI image
Import->>Host: Start immutable host OS
Host->>Git: Pull system configurations
Git->>Discovery: Scan for tenant configs
Discovery->>Provision: For each tenant:
Note over Provision: • Clone/pull tenant repo<br/>• Mount tenant directories<br/>• Apply sysext/confext layers<br/>• Start VM/container<br/>• Execute provisioning units
| Phase | Component | Action | Result |
|---|---|---|---|
| 1️⃣ | 🔌 UEFI Boot | systemd-boot loads the boot configuration | Secure boot validation |
| 2️⃣ | ⬇️ Image Import | systemd-import-generator downloads and verifies the host EFI image | DDI image verification |
| 3️⃣ | 🖥️ Host Boot | Immutable host OS starts with minimal services | Base system ready |
| 4️⃣ | 📚 Git Sync | System-level configurations pulled from Git repositories | Configuration sync |
| 5️⃣ | 🔍 Tenant Discovery | Scan for tenant configurations in Git | Tenant enumeration |
| 6️⃣ | ⚡ Tenant Provisioning | For each discovered tenant: provision resources | Multi-tenant setup |
/
├── /boot/ # Boot partition (read-only)
│ └── efi/ # EFI System Partition
├── /etc/ # Host configuration (overlayfs from confext)
│ ├── systemd/
│ │ ├── system/ # System units
│ │ │ ├── tenant@.service # Tenant instantiation template
│ │ │ ├── tenant-infra@.service # Infrastructure manager template
│ │ │ └── tenant-network@.service # Network manager template
│ │ ├── network/ # Network configuration templates
│ │ └── system-generators/ # Custom systemd generators
│ │ ├── tenant-generator # Discovers and instantiates tenants
│ │ └── mount-generator # Generates mount units for tenants
│ └── voa/ # Verification of OS Artifacts
│ └── bitbuilder/image/default/openpgp/
├── /usr/ # Vendor OS (read-only, overlayfs from sysext)
│ ├── lib/
│ │ ├── extensions/ # System-wide extension images
│ │ │ ├── base-tools.sysext.raw
│ │ │ ├── monitoring.sysext.raw
│ │ │ └── security.confext.raw
│ │ ├── systemd/
│ │ │ ├── system/ # Vendor systemd units
│ │ │ └── network/ # Vendor network templates
│ │ └── bitbuilder/
│ │ ├── templates/ # Tenant template repositories
│ │ │ ├── tenant-infra/
│ │ │ ├── tenant-machine/
│ │ │ ├── tenant-container/
│ │ │ └── tenant-homed/
│ │ └── generators/ # BitBuilder systemd generators
│ └── share/
│ └── voa/ # Vendor verification keys
├── /var/ # Persistent variable data
│ ├── lib/
│ │ ├── machines/ # systemd-nspawn containers
│ │ ├── tenants/ # Tenant persistent storage
│ │ │ └── <tenant-id>/
│ │ │ ├── infra/ # Infrastructure VM
│ │ │ │ ├── rootfs/ # Infrastructure manager root
│ │ │ │ └── config.git/ # Cloned infra config repo
│ │ │ ├── machines/ # Tenant VMs (systemd-vmspawn)
│ │ │ │ └── <machine-id>/
│ │ │ │ ├── disk.raw # VM disk image
│ │ │ │ └── config/ # Machine-specific config
│ │ │ ├── containers/ # Tenant containers (systemd-nspawn)
│ │ │ │ └── <container-id>/
│ │ │ │ └── rootfs/ # Container root filesystem
│ │ │ ├── extensions/ # Tenant-specific extensions
│ │ │ │ ├── sysext/ # System extensions
│ │ │ │ └── confext/ # Configuration extensions
│ │ │ ├── network/ # Network namespace configs
│ │ │ └── homed/ # User home directories
│ │ └── extensions.mutable/ # Mutable overlay configuration
│ │ ├── etc/ # -> /var/lib/tenants/<id>/overlay/etc
│ │ └── usr/ # -> /var/lib/tenants/<id>/overlay/usr
│ └── cache/
│ └── tenants/ # Cached tenant images
└── /run/ # Runtime data (tmpfs)
├── systemd/
│ └── generator/ # Generated units at boot
└── tenants/ # Runtime tenant data| Security Layer | Technology | Implementation | Benefit |
|---|---|---|---|
| 🏗️ Immutable Infrastructure | DDI, overlayfs | Host OS cannot be modified at runtime | Eliminates configuration drift |
| 📚 Git-Only Changes | Git-ops workflow | All modifications through Git commits | Audit trail, rollback capability |
| 🏠 Tenant Isolation | systemd security | Complete separation using multiple boundaries | Defense in depth |
| 🔐 Encrypted Storage | systemd-homed, LUKS | Optional encryption for tenant data | Data protection at rest |
| 🚀 Secure Boot | UEFI, TPM | Full secure boot chain support | Boot integrity verification |
graph TB
subgraph "🏠 Tenant A"
VM1[🖥️ VM Instance]
NS1[📦 Container]
end
subgraph "🏠 Tenant B"
VM2[🖥️ VM Instance]
NS2[📦 Container]
end
subgraph "🛡️ Security Boundaries"
PID[🔒 PID Namespaces]
NET[🌐 Network Isolation]
MOUNT[💾 Mount Namespaces]
USER[👤 User Namespaces]
CGROUP[⚙️ Resource Limits]
CAP[🔧 Capabilities]
MAC[🛡️ SELinux/AppArmor]
end
VM1 --- PID
VM2 --- PID
NS1 --- NET
NS2 --- NET
VM1 --- MOUNT
VM2 --- USER
NS1 --- CGROUP
NS2 --- CAP
VM1 --- MAC
bitbuilder-system/
├── .gitops/
│ └── config.yaml # Git-ops configuration
├── generators/ # Custom systemd generators
│ ├── tenant-generator # Tenant discovery and instantiation
│ └── mount-generator # Extension mount generation
├── units/ # System service units
│ ├── tenant@.service # Tenant instantiation template
│ ├── tenant-infra@.service # Infrastructure manager template
│ └── tenant-network@.service # Network setup template
├── network/ # Network configuration templates
│ ├── 10-tenant-bridge.netdev
│ ├── 10-tenant-bridge.network
│ ├── 20-wg-tenant.netdev
│ ├── 20-wg-tenant.network
│ ├── 30-vxlan-tenant.netdev
│ └── 30-vxlan-tenant.network
├── extensions/ # System-wide extensions
│ ├── base-tools.sysext.raw
│ ├── monitoring.sysext.raw
│ └── security.confext.raw
└── tenants/ # Tenant registry
└── registry.json # Active tenants listtenant-<name>/
├── metadata.json # Tenant metadata and requirements
├── infrastructure/ # Infrastructure manager config
│ ├── config.yaml # Infra manager configuration
│ └── resource-limits.yaml # Resource quotas
├── network/ # Network configuration
│ ├── topology.yaml # Network topology definition
│ ├── wireguard/ # VPN configurations
│ └── firewall/ # Firewall rules
├── machines/ # VM definitions
│ └── <machine-name>/
│ ├── config.yaml # VM configuration
│ └── cloud-init/ # Cloud-init data
├── containers/ # Container definitions
│ └── <container-name>/
│ ├── config.yaml # Container configuration
│ └── rootfs-overlay/ # Filesystem overlays
├── extensions/ # Tenant-specific extensions
│ ├── sysext/ # System extensions
│ │ └── *.sysext.raw
│ └── confext/ # Configuration extensions
│ └── *.confext.raw
├── services/ # Portable service definitions
│ └── *.portable.raw
├── users/ # User definitions
│ └── <username>/
│ └── homed-config.json # systemd-homed configuration
└── scripts/
├── pre-provision.sh # Pre-provisioning script
├── post-provision.sh # Post-provisioning script
└── health-check.sh # Health monitoring scriptBitBuilder provides standard templates for creating new tenants and resources:
| Template | Purpose | Technology | Use Case |
|---|---|---|---|
🏗️ tenant-infra-template |
Infrastructure manager VM template | systemd-vmspawn | Tenant orchestration |
🖥️ tenant-machine-template |
VM instance template | systemd-vmspawn | Application VMs |
📦 tenant-container-template |
Container template | systemd-nspawn | Containerized apps |
🏠 tenant-homed-template |
User home directory template | systemd-homed | User environments |
🔧 sysext-template |
System extension template | Extension images | System augmentation |
⚙️ confext-template |
Configuration extension template | Extension images | Config management |
| Advantage | Description | Impact |
|---|---|---|
| 📚 Version Control | All infrastructure changes tracked in Git | Complete change history, diff capabilities |
| 🔄 Rollback Capability | Easy reversion to previous configurations | Zero-downtime rollbacks, safety net |
| 📋 Audit Trail | Complete history of all changes | Compliance, security, accountability |
| 🎯 Declarative Management | Define desired state, not procedures | Reduced complexity, predictable outcomes |
| 📈 Scalability | Add/remove tenants by updating Git repositories | Horizontal scaling, resource optimization |
| ⚡ Consistency | Identical deployments from the same Git commits | Reproducible infrastructure, dev/prod parity |
| 🧪 Testing | Test configurations in staging before production | Risk reduction, validation workflows |
| Use Case | Industry | Benefits | Technology Stack |
|---|---|---|---|
| 🏢 Multi-Tenant Hosting | Cloud Providers | Secure isolation for multiple customers | systemd-vmspawn, networking |
| 👨💻 Development Environments | Software Development | Reproducible developer workspaces | systemd-nspawn, git-ops |
| 🌐 Edge Computing | IoT, CDN | Lightweight virtualization for edge deployments | Minimal footprint, secure boot |
| 🔄 CI/CD Infrastructure | DevOps | Dynamic test environments | Automated provisioning, git integration |
| 🔒 Application Isolation | Security | Run untrusted workloads safely | Namespace isolation, capabilities |
| 📋 Compliance Environments | Finance, Healthcare | Auditable, immutable infrastructure | Immutable OS, audit trails |
| Component | Minimum | Recommended | Purpose |
|---|---|---|---|
| 🔌 UEFI Firmware | UEFI 2.0+ | UEFI 2.8+ with Secure Boot | Boot integrity, hardware security |
| ⚙️ systemd Version | 258+ | 260+ | All virtualization features enabled |
| 📚 Git Client | 2.30+ | Latest stable | Repository operations, authentication |
| 🌐 Network Connectivity | Basic | High-bandwidth | Git repository access, updates |
| 💾 Resources | 8GB RAM, 100GB storage | 32GB RAM, 1TB NVMe SSD | Tenant VMs/containers |
The BBOS image is built using mkosi. You will need to install the current main branch of mkosi to build current BBOS images.
First, configure the variant you'd like to build in mkosi.local.conf. For a
desktop system, you'll want the desktop profile and either the gnome or the
kde profile.
[Distribution]
Distribution=arch
[Config]
Profiles=desktop,kde
To build the image, run mkosi -B -f from the BBOS repository. Currently
arch, fedora and debian are supported distributions. Implementing support for a
new distribution (that's already supported in mkosi) is as simple as writing the
necessary config files to install the required packages for that distribution.
To update the system after installation, you clone the BBOS repository
or your fork of it, make sure mkosi.local.conf is configured to your liking and
run mkosi -B -ff sysupdate -- update --reboot which will update the system using
systemd-sysupdate and then reboot.
Sometimes BBOS adopts systemd features as soon as they get merged into
systemd without waiting for an official release. That's why we recommend
enabling the obs profile to enable the systemd repositories on OBS
(https://software.opensuse.org//download.html?project=system%3Asystemd&package=systemd)
containing systemd packages which are built every day from systemd's git main
branch.
To enable the obs profile, add the following to mkosi.local.conf:
[Config]
Profiles=obs
As an alternative to using the obs profile, you can build systemd from source:
git clone https://github.com/systemd/systemd
cd systemd
mkosi -f sandbox -- meson setup build
mkosi -f sandbox -- meson compile -C build
mkosi -t none -fThen write the following to mkosi.local.conf in the BBOS repository to
use the artifacts from the systemd repository built by mkosi in BBOS:
[Content]
VolatilePackageDirectories=../systemd/build/mkosi.builddir/<distribution>~<release>~<arch>
[Build]
ExtraSearchPaths=../systemd/build
Make sure the distribution and release in mkosi.local.conf are identical in the
systemd checkout and the bbos checkout.
To build a newer systemd, run git pull in the systemd repository followed by
mkosi -f sandbox -- meson compile -C build and mkosi -t none.
BBOS images are signed for Secure Boot with the user's keys. To generate a new key,
run mkosi genkey. The key must be stored safely, it will be required to sign updates.
The key can be stored in a smartcard. Then you have to set the key in mkosi.local.conf:
[Validation]
SecureBootKey=pkcs11:object=Private key 1;type=private
SecureBootKeySource=provider:pkcs11
#SignExpectedPcrKey=pkcs11:object=Private key 1;type=private
#SignExpectedPcrKeySource=provider:pkcs11
VerityKey=pkcs11:object=Private key 1;type=private
VerityKeySource=provider:pkcs11
Before installing BBOS, make sure that Secure Boot is in setup mode on the
target system. The Secure Boot mode can be configured in the UEFI firmware
interface of the target system. If there's an existing Linux installation on the
target system already, run systemctl reboot --firmware-setup to reboot into
the UEFI firmware interface. At the same time, make sure the UEFI firmware
interface is password protected so an attacker cannot just disable Secure Boot
again.
To install BBOS with a USB drive, first build the image on an existing
Linux system as described above. Then, burn it to the USB drive with
mkosi burn /dev/<usb>. Once burned to the USB drive, plug the USB drive into
the system onto which you'd like to install BBOS and boot into the USB
drive via the firmware. Then, boot into the "Installer" UKI profile. When you
end up in the root shell, run
systemd-repart --dry-run=no --empty=force --defer-partitions=swap,root,home /dev/<drive>
to install BBOS to the system's drive. Finally, reboot into the target
drive (not the USB) and the regular profile (not the installer one) to complete
the installation.
systemd doesn't support adding a recovery key to a partition enrolled with a token
only (tpm/fido2). It is possible to use cryptenroll to add a recovery password
to the root partition: cryptsetup luksAddKey --token-type systemd-tpm2 /dev/<id>
Only firmwares that are dependencies of a kernel module are included, but some
modules don't declare their dependencies properly. Dependencies of a module can be
found with modinfo. If you experience missing firmwares, you should report
this to the module maintainer. FirmwareInclude= can be added in mkosi.local.conf
to include the firmware regardless of whether a module depends on it.
After installing BBOS and logging into your systemd-homed managed user, run the following to configure systemd-homed for the best experience:
homectl update \
--auto-resize-mode=off \
--disk-size=max \
--luks-discard=on"Disabling the auto resize mode avoids slow system boot and shutdown. Enabling LUKS discard makes sure the home directory doesn't become inaccessible because systemd-homed is unable to resize the home directory.
If you boot BBOS in a virtual machine using mkosi vm, the root password
is automatically set to bbos and a default user bbos with password
bbos is created as well.
This project is licensed under the GNU Lesser General Public License v2.1 (LGPL-2.1). See the LICENSE file for details.