Troubleshooting

Each section below is a refusal the CLI prints, and what it asks you to change. The transcripts are executed against the binary, so the wording here is the wording you get. The last section collects development-environment problems that come from third-party toolchains.

Error format

A space error gives the cause on one line and the fix on the next, with a pointer to deeper evidence where one exists. A refusal names the command that resolves it:

space run: payload initramfs not found: _build/spaceos/payload.cpio
Run `space build` first to produce the run artifacts.

$ space build . -t app:release --release --signing-profile release
space build: a --release build requires an explicit --rollback-index of 1 or more; 0 is the development domain's index. Pass the index this release publishes: it must be above the one the target node has already admitted, or the node refuses the launch.
[1]

The local runner is the background supervisor that tracks running workloads. A read command never starts one: space status reports that no runner is connected and exits 0, and space logs says the same and exits non-zero. space run is what starts a workload, and space stop starts the runner on demand when it has one to stop.

Build errors

Release-looking tag refused

A tag such as :release, :prod, or :production is refused unless the build passes --release. The build stops without it.

Release build refused for its signing identity

$ space build . -t app:release --release --signing-profile release --rollback-index 1
space build: a --release build will not mint signing identity "release"; a release key is generated where the release is authorized, not where it is built. Name an identity `space keys ls` already lists, or pass the `--signing-key-ed25519` and `--signing-key-mldsa65` pair
[1]

A --release build signs with an identity that existed before the build asked for it. Naming a profile that does not exist is how one is created on a development build; on a release it is the build minting its own authority, so it is refused. space keys ls lists what you have.

Two neighboring refusals share the reason. --signing-profile dev is refused by name, because that name occupies the development trust anchor on every node that admits the build. A --signing-key-ed25519 and --signing-key-mldsa65 pair pointing at the development identity is refused by fingerprint, because renaming those keys does not replace them.

Release build refused for its rollback index

A --release build requires --rollback-index 1 or more. The index defaults to 0, the development domain, and a release published at 0 would never outrank its predecessor, so the check runs at build time where you can see it. Pass the index this release publishes; it must be above the one the node has already admitted. A --release build also requires an explicit non-development signing profile, and rejects --keystore dev.

Service dependency cycle

service dependency cycle at eo-classifier

The depends_on graph in build.yaml contains a cycle. depends_on defines a strict launch order; remove one edge to break the cycle. A depends_on naming a service that does not exist fails with unknown service dependency.

Runtime and isolation disagree

$ space sbom runtime-mismatch.yaml
space: runtime-mismatch.yaml: build service "eo-classifier" runtime mirage does not support isolation runc (a unikernel needs spt/hvt/firecracker, an oci service needs runc, a host service needs host)
at index 0 of
array<build_service>
in member services of
build_partition object
in member payload of
object
in member partitions of
build object
[1]

The trailer names the path through the document that reached the fault. The service’s declared runtime and its isolation backend are incompatible. This is a build-time error: fix either field in build.yaml (or the override in run.yaml) so the pair is consistent, or leave isolation as auto and let the runtime derive it.

OCI image pull failure

An image reference in build.yaml that does not resolve fails the build. Check that the name and tag are correct, that you are authenticated to the registry, and that the image has a variant for the target architecture (docker manifest inspect <image>).

Kernel not found

A local kernel path (as opposed to an OCI reference) must exist relative to the directory where space build runs. Values starting with ./ or / and bare filenames are treated as local paths.

Run errors

run.yaml rejected

The space.run schema is strict. Unknown fields anywhere in the document are rejected: a mistyped field name fails the load instead of being silently dropped. The relay and ground blocks default when omitted. A run.yaml that declares no partitions at all is refused, because there is nothing to default the relay from.

Unknown isolation backend

$ space run _build/spaceos --runtime unknown-backend.yaml --dry-run
space: unknown-backend.yaml: unknown isolation backend: vm
in member isolation of
run_service object
at index 0 of
array<run_service object>
in member services of
run_partition object
in member payload of
object
in member partitions of
run object
[1]

The isolation field must be one of auto, runc, spt, hvt, firecracker, qemu, or host. The selected target must provide the host features required by the chosen backend.

Service name mismatch

Service names in run.yaml must match the names declared in build.yaml for the same partition. A placement entry for a name the composition does not contain is rejected when the run is validated. Use space run ... --dry-run to check placement without booting.

General

Stale cache

If a build produces unexpected results after an image has been updated upstream, use an immutable image digest or a new tag, then rebuild the composition. Avoid relying on a mutable :latest tag when debugging.

Verbosity and machine output

space build, space run, space publish, space deploy, space verify, space sbom, space status and space logs take --json for structured output, -v for more log detail (twice for debug, three times for protocol traces), and --log=SPEC for per-source levels. space run’s volume flag is spelled --volume in full and has no short form, so -v there is verbosity like everywhere else. To rehearse without booting, use space run ... --dry-run, space deploy ... --plan, and space target apply --plan, which prints the provider plan and stops.

Development environment

Unikraft build fails with `flex` flags

flex: Can't use -f or -F with -l option

On some distributions, /usr/bin/lex contains a call to flex -l; on others, /usr/bin/lex is a symlink to flex. Force the build to call flex directly:

$ export LEX=flex

Unikraft 9pfs access does not work

ERR: [libvirtio_bus] Failed to find the driver for the virtio device 0x92020 (id:9)
ERR: [libvirtio_pci] Failed to register the virtio device: -14

The Unikraft 9p filesystem driver is gated behind an opt-in build flag. Use the development Unikraft release and enable 9pfs explicitly:

opam pin -ny https://github.com/mirage/ocaml-unikraft.git
opam install ocaml-unikraft-default-x86_64 \
             ocaml-unikraft-option-9pfs

(Substitute arm64 for the target architecture as appropriate.)

QEMU bridge access denied

access denied by acl file, bridge helper failed

QEMU’s bridge helper runs setuid and consults an ACL file before attaching a tap interface to a bridge. Allow the current user:

echo "allow all" | sudo tee /etc/qemu/${USER}.conf
echo "include /etc/qemu/${USER}.conf" \
  | sudo tee --append /etc/qemu/bridge.conf
sudo chown root:${USER} /etc/qemu/${USER}.conf
sudo chmod 640 /etc/qemu/${USER}.conf

Packets not received on non-standard ports

Strict firewall configurations (typical on recent Fedora / RHEL / openSUSE) drop packets to non-standard ports even when no iptables rule is visible, because the rule lives in the nftables table managed by firewall-cmd. Inspect both:

iptables -L
nft list ruleset

Add the bridge to a firewall zone and open the relevant ports through firewall-cmd, marking the rule persistent with --persistent.

C++ linking fails on Unikraft

undefined reference to __cxa_call_terminate

This appears when compiling C++ with a recent GCC; Unikraft’s bundled libcxxabi is several versions behind. Use these workarounds in order:

Build inside an older Debian image with a compatible toolchain; e.g. docker run --rm -v "$PWD":/src -w /src debian:11 bash -c 'apt-get install -y build-essential && kraft build'.
Use clang++ instead of g++. The two settings below select the clang toolchain, and Unikraft’s CPP setting names the C++ compiler.
Drop to an older gcc (GCC 14 or earlier).

kraft system set toolchain.CC="clang"
kraft system set toolchain.CPP="clang++"

Incomplete unikernel stack traces

Wrap the unikernel’s entry point in a try ... with and explicitly print the backtrace. Failures remain visible even when the generated entry point does not print the full exception by default.

`tap` interface drops out of the bridge

After qemu-bridge-helper attaches a tap interface to a bridge, the interface disappears from the bridge a short while later, and unikernel networking stops working. On systems running systemd-networkd, this is typically because the default 80-wired.network rule matches the tap device and attempts to manage it.

Override the match with a higher-precedence file that tells systemd-networkd to ignore tap interfaces:

[Match]
Kind=tun
Name=tap*

[Link]
Unmanaged=yes

Reload with sudo networkctl reload. Unikernel networking now survives across QEMU restarts.

Upstream systemd has a fix that scopes 80-wired.network to physical interfaces only; recent distributions may not need this workaround.