VM Configuration

The “VM Layer” of the Podplane Architecture is responsible for getting Pods scheduled on a VM. This is essentially Netsy + core Kubernetes + containerd, configured by vmconfig - a minimal configuration system designed for Debian-based Linux VMs, written in Bash.

How It Works

The user-data script which invokes the vmconfig entrypoint is responsible for supplying the package dependencies, which is defined by the “kind” of VMs you want vmconfig to configure:

1. knd creates a Kubernetes Data Plane / Worker node, which runs kubelet, containerd, and supporting services.

2. knc creates a Kubernetes Control Plane node, which is essentially a base of knd + adds Netsy (as an etcd alternative), kube-apiserver, kube-scheduler, kube-controller-manager, and a stateless container registry.

Deployed Clusters

The flow for clusters you create via Podplane CLI is:

1. CLI generates and deploys infrastructure-as-code
2. Nstance auto-scales VMs using the Podplane-generated cloud-init user-data script
3. Each VM’s user-data script downloads the relevant dependencies, extracts the vmconfig archive, and runs its install/configure scripts
4. vmconfig sets up all required services for the VM’s kind (knd = data plane, knc = control plane)
5. Control Plane nodes run Netsy backed by object storage, replacing the need for etcd and on-disk state

Local Clusters

For local VMs run via the Podplane CLI, the same vmconfig configuration is used to run a single knc VM per local cluster.

The CLI itself is responsible for downloading/caching dependencies and serving them to the VM via a webserver it runs in the background via the local start command. The same webserver also hosts a fake OIDC server and fake S3 server for local clusters. You can also use local server to run the webserver directly.

Container Registry

VMs run a Zot Registry backed by the configured registry object-storage bucket.

Podplane supports component and template image “mirroring” whereby the registry bucket stores a copy of all required container images if mirroring is enabled.

To use this mirror, components render explicit image references such as <registry-hostname>/mirror/<original-registry>/<repository>:<tag>. This differs to user-pushed app images which live under <registry-hostname>/apps/.... Podplane intentionally does not make zot a transparent containerd pull-through cache for all image pulls - while that would have made configuring image references easier without the <registry-hostname>/ prefix, the decision to use explicit references was because:

1. it is obvious from the rendered Kubernetes manifest when an image is using the Podplane built-in registry, even though components and templates need to do more work to render the correct image references (e.g. you have to override an off-the-shelf Helm chart values file to change the images used)
2. user workloads keep native Kubernetes registry authentication behavior, including per-namespace and per-service-account imagePullSecrets; a transparent zot pull-through cache would require zot to authenticate to upstream registries itself and would not naturally receive the pod’s upstream registry credentials

Templates and components should therefore use canonical upstream image references by default, and only render mirrored references intentionally when Podplane owns the image selection and registry mirror behavior.

Package Dependencies

Data Plane & Control Plane VMs:

• nstance-agent to register VMs
• fluent-bit for log forwarding
  • libpq5 runtime dependency for fluent-bit
• kube2iam for providing IAM Roles to pods
• containerd the container runtime
• runc for containerd to spawn OCI containers
• kubelet for running node pods
  • uidmap runtime dependency of kubelet for getsubids
  • libsubid5 runtime dependency of kubelet for getsubids
• cni-plugins the reference CNI plugins

Control Plane VMs:

• netsy as an etcd alternative
• kube-apiserver the Kubernetes API server
• kube-scheduler the Kubernetes scheduler
• kube-controller-manager the core Kubernetes control loops
• distribution the stateless container registry

Env Vars & Service Configuration

While most of the configuration for each VM kind is the same, there are a set of variable inputs to the vmconfig package, propagated to each service via environment variable .env files:

1. /opt/podplane/etc/user-data.env: immutable, created on first-boot by the cloud-init user-data.sh script, read only by the /opt/podplane/bin/configure.sh script. Contains for example the instance ID, Nstance nonce JWT, and OIDC issuer URL.

2. /opt/podplane/etc/detected.env: immutable, created during the first run of the /opt/podplane/bin/bootstrap.sh script, and read only by the /opt/podplane/bin/configure.sh script. Contains for example the detected instance hostname and IPv4 / IPv6 addresses.

3. /opt/podplane/etc/mutable.env: mutable, created during the first run of the /opt/podplane/bin/bootstrap.sh script, able to be updated by the /opt/podplane/bin/update-mutable-env.sh script, and read only by the /opt/podplane/bin/configure.sh script. Contains for example the OIDC issuer URL.

4. /opt/env/<service>.env: mutable, updated only by the /opt/podplane/bin/configure.sh script, combining the outputs of user-data.env and mutable.env files.

The systemd services configured by vmconfig then source their respective /opt/env/<service>.env files to specify configuration; some are able to use env vars only, others have a helper script which passes those through to configuration flags.

Example Configuration Flow for AWS

When you deploy a new cluster, configuration is propagated to each service like so:

podplane cluster create
└─ writes podplane.cluster.jsonc
   └─ generates podplane.*.tf
      └─ calls Nstance Terraform modules
         └─ configures nstance-server groups/templates/files
         └─ AWS ASG scales up a VM which runs the nstance-server
            └─ nstance-server schedules new Podplane VMs e.g. of `knc` kind
               └─ cloud-init runs `user-data.sh` generated by Podplane
                  └─ user-data downloads and verifies all deps
                  └─ user-data extracts the /opt/deps/vmconfig.tar.gz archive
                  └─ user-data runs `install.sh` from /opt/podplane/bin
                  └─ user-data writes /opt/podplane/etc/user-data.env
                  └─ user-data runs `configure.sh` from /opt/podplane/bin
                     └─ runs `/opt/podplane/bin/bootstrap.sh` once
                        └─ creates /opt/podplane/etc/detected.env
                     └─ creates /opt/podplane/etc/mutable.env
                     └─ configure.sh generates /opt/env/*.env
                        ├─ /opt/env/nstance-agent.env
                        ├─ /opt/env/kubelet.env
                        ├─ /opt/env/kube-apiserver.env
                        ├─ /opt/env/netsy.env
                        └─ other component env files
                     └─ runs `/opt/podplane/bin/restart.sh` to start systemd services
               └─ systemd services source their /opt/env/<component>.env files   

When you update a cluster, for example to change the OIDC issuer, configuration is propagated using the lower half of the same pipeline:

nstance-server runtime config/files change, sends a new `mutable.env` file to each node
└─ nstance-agent receives the `mutable.env` file into /opt/nstance-agent/recv
   └─ nstance-recv-watch.sh detects write-files.last
      ├─ update-mutable-env.sh validates + merges recv/mutable.env into /opt/podplane/etc/mutable.env
      ├─ copies delivered certs/keys/config files into their runtime locations
      ├─ runs configure.sh to regenerate /opt/env/*.env
      └─ restarts the affected services

Mutable Configuration Vars

The following variables are able to be live-updated without having Nstance rotate VMs, via the mutable.env file as described above.

These variables can be set/overriden alongside the Podplane-generated .tf configuration:

The Podplane .tf configuration derives the following variables which can also be propagated via the mutable.env file:

In the OpenTofu/Terraform configuration generated by the cluster create command, these values come from local.mutable_env.