Troubleshooting

Error format

Every space error is one line of cause, then the fix, then a pointer to deeper evidence. A refusal names the command that unlocks it:

space run: payload initramfs not found: _build/spaceos/payload.cpio
Run `space build` first to produce the run artifacts.

space build: refusing to build a release-looking tag with the dev identity
pass --signing-profile release or --signing-key KEY

When a read command cannot reach the local engine, the background supervisor that tracks running workloads, the error names the engine log and the remedy: start it with space up.

Build errors

Release-looking tag refused

A tag such as :release, :prod, or :production is refused unless the build passes --release, and a --release build requires an explicit non-development signing profile. The build stops without one.

Service dependency cycle

service dependency cycle at eo-classifier

The depends_on graph in build.yaml contains a cycle. depends_on defines a strict launch order; remove one edge to break the cycle. A depends_on naming a service that does not exist fails with unknown service dependency.

Runtime and isolation disagree

runtime mirage does not support isolation runc (a unikernel needs
spt/hvt/firecracker, an oci service needs runc, a host service needs host)

The service’s declared runtime and its isolation backend are incompatible. This is a build-time error: fix either field in build.yaml (or the override in run.yaml) so the pair is consistent, or leave isolation as auto and let the runtime derive it.

OCI image pull failure

An image reference in build.yaml that does not resolve fails the build. Check that the name and tag are correct, that you are authenticated to the registry, and that the image has a variant for the target architecture (docker manifest inspect <image>).

Kernel not found

A local kernel path (as opposed to an OCI reference) must exist relative to the directory where space build runs. Values starting with ./ or / and bare filenames are treated as local paths.

Run errors

run.yaml rejected

The space.run schema is strict: unknown fields anywhere in the document are rejected, so a mistyped field name fails the load instead of being silently dropped. The relay and ground blocks default when omitted; a run.yaml that declares no partitions at all is refused because there is nothing to default the relay from.

Unknown isolation backend

unknown isolation backend: vm

The isolation field must be one of auto, runc, spt, hvt, firecracker, qemu, or host. The selected target must provide the host features required by the chosen backend.

Service name mismatch

Service names in run.yaml must match the names declared in build.yaml for the same partition. A placement entry for a name the composition does not contain is rejected when the run is validated. Use space run ... --dry-run to check placement without booting.

General

Stale cache

If a build produces unexpected results after an image has been updated upstream, use an immutable image digest or a new tag, then rebuild the composition. Avoid relying on a mutable :latest tag when debugging.

Verbosity and machine output

Read commands take --json for structured output. Log verbosity follows the standard log-filter convention through the CLI’s logging flags. On space run, -v is the volume flag, not verbose mode. To rehearse without booting, use space run ... --dry-run and space deploy ... --plan.

Development environment

Unikraft build fails with `flex` flags

flex: Can't use -f or -F with -l option

On some distributions, /usr/bin/lex contains a call to flex -l; on others, /usr/bin/lex is a symlink to flex. Force the build to call flex directly:

export LEX=flex

Unikraft 9pfs access does not work

ERR: [libvirtio_bus] Failed to find the driver for the virtio device 0x92020 (id:9)
ERR: [libvirtio_pci] Failed to register the virtio device: -14

The Unikraft 9p filesystem driver is gated behind an opt-in build flag. Use the development Unikraft release and enable 9pfs explicitly:

opam pin -ny https://github.com/mirage/ocaml-unikraft.git
opam install ocaml-unikraft-default-x86_64 \
             ocaml-unikraft-option-9pfs

(Substitute arm64 for the target architecture as appropriate.)

QEMU bridge access denied

access denied by acl file, bridge helper failed

QEMU’s bridge helper runs setuid and consults an ACL file before attaching a tap interface to a bridge. Allow the current user:

echo "allow all" | sudo tee /etc/qemu/${USER}.conf
echo "include /etc/qemu/${USER}.conf" \
  | sudo tee --append /etc/qemu/bridge.conf
sudo chown root:${USER} /etc/qemu/${USER}.conf
sudo chmod 640 /etc/qemu/${USER}.conf

Packets not received on non-standard ports

Strict firewall configurations (typical on recent Fedora / RHEL / openSUSE) drop packets to non-standard ports even when no iptables rule is visible, because the rule lives in the nftables table managed by firewall-cmd. Inspect both:

iptables -L
nft list ruleset

Add the bridge to a firewall zone and open the relevant ports through firewall-cmd, marking the rule persistent with --persistent.

C++ linking fails on Unikraft

undefined reference to __cxa_call_terminate

This appears when compiling C++ with a recent GCC; Unikraft’s bundled libcxxabi is several versions behind. Use these workarounds in order:

Build inside an older Debian image with a compatible toolchain; e.g. docker run --rm -v "$PWD":/src -w /src debian:11 bash -c 'apt-get install -y build-essential && kraft build'.
Use clang++ instead of g++:
Terminal window
```
kraft system set toolchain.CC="clang"
kraft system set toolchain.CPP="clang++"
```
(Unikraft uses CPP for the C++ compiler, not the preprocessor.)
Drop to an older gcc (GCC 14 or earlier).

Incomplete unikernel stack traces

Wrap the unikernel’s entry point in a try ... with and explicitly print the backtrace. Failures remain visible even when the generated entry point does not print the full exception by default.

`tap` interface drops out of the bridge

After qemu-bridge-helper attaches a tap interface to a bridge, the interface disappears from the bridge a short while later, and unikernel networking stops working. On systems running systemd-networkd, this is typically because the default 80-wired.network rule matches the tap device and attempts to manage it.

Override the match with a higher-precedence file that tells systemd-networkd to ignore tap interfaces:

[Match]
Kind=tun
Name=tap*

[Link]
Unmanaged=yes

Reload with sudo networkctl reload. Unikernel networking now survives across QEMU restarts.

Upstream systemd has a fix that scopes 80-wired.network to physical interfaces only; recent distributions may not need this workaround.