Troubleshooting

When space refuses, it says why on one line and what to do on the next. Each section below is one refusal and the change it asks of you. The last section collects problems that third-party toolchains cause.

Error format

A space error gives the cause on one line and the fix on the next, with a pointer to deeper evidence where one exists. A refusal names the command that resolves it:

space run: payload initramfs not found: _build/spaceos/payload.cpio
Run `space build` first to produce the run artifacts.

$ space build . -t app:release --release --signing-key release
space build: a --release build will not mint signing identity "release"
  Name an identity `space keys ls` already lists, or pass
  `--signing-key-ed25519 PATH` and `--signing-key-mldsa65 PATH`.
[123]

The local runner is the background supervisor that tracks running workloads. A read command never starts one: space status reports that no runner is connected and exits 0, and space logs says the same and exits non-zero. space run is what starts a workload. space run stop starts nothing either: a runner that is not listening is running nothing, so the command says that nothing is running and has nothing to stop.

Build errors

Release-looking tag refused

A tag such as :release, :prod, or :production is refused unless the build passes --release. The build stops without it.

Release build refused for its signing identity

$ space build . -t app:release --release --signing-key release --rollback-index 1
space build: a --release build will not mint signing identity "release"; a
  release key is generated where the release is authorized, not where it is
  built. Name an identity `space keys ls` already lists, or pass the
  `--signing-key-ed25519` and `--signing-key-mldsa65` pair
[123]

The signing identity is checked after the runtime bundle and the source checkout are resolved, so a machine with neither meets the SpaceOS runtime has not been released first and never reaches this line.

A --release build signs with an identity that existed before the build asked for it. Naming an identity that does not exist is how one is created on a development build; on a release it is the build minting its own authority, so it is refused. space keys ls lists what you have.

Two related refusals follow from the same rule. --signing-key dev is refused by name, because a node holds the development identity's public key under the anchor name dev, and a release signed under that name could not be told apart from a development build. A --signing-key-ed25519 and --signing-key-mldsa65 pair pointing at the development identity is refused by fingerprint, because renaming those keys does not replace them.

Release build refused for its rollback index

A --release build requires --rollback-index 1 or more. The index defaults to 0, the development domain, and a release published at 0 would never outrank its predecessor, so the check runs at build time where you can see it. Pass the index this release publishes; it must be above the one the node has already admitted. A --release build also requires an explicit non-development signing identity, and rejects --keystore dev.

Service dependency cycle

service dependency cycle at eo-classifier

The depends_on graph in space.yaml contains a cycle. depends_on defines a strict launch order; remove one edge to break the cycle. A depends_on naming a service that does not exist fails with unknown service dependency.

Runtime and isolation disagree

$ space show --sbom runtime-mismatch/space.yaml
space show: runtime-mismatch/space.yaml:8.11-11.25:
  run.partitions.payload.services.[0]: build service "eo-classifier" runtime
  mirage does not support isolation runc (it supports spt, hvt, firecracker or
  qemu)
[123]

The refusal opens on the path through the document that reached the fault. The service's declared runtime and its isolation backend are incompatible. This is a build-time error: fix either field so the pair is consistent, or leave isolation as auto and let the runtime derive it.

OCI image pull failure

An image reference in space.yaml that does not resolve fails the build. Check that the name and tag are correct, that you are authenticated to the registry, and that the image has a variant for the target architecture (docker manifest inspect <image>).

Kernel not found

The -k or --kernel option of space build and space run boots a kernel other than the one the product supplies, for that run only; a kernel: field in space.yaml is refused. A local kernel path given there (as opposed to an OCI reference) must exist relative to the directory where the command runs. Values starting with ./ or / and bare filenames are treated as local paths.

Run errors

space.yaml rejected

The space.project schema is strict. Unknown fields anywhere in the document are rejected: a mistyped field name fails the load instead of being silently dropped. The relay and ground blocks default when omitted. A project that declares no partitions at all is refused, because there is nothing to default the relay from. A kernel: or a target_profile: is refused by name, because both describe a machine and belong in targets/NAME.yaml.

Unknown isolation backend

$ space run -C unknown-backend --dry-run
space run: space.yaml:12.22-23: run.partitions.payload.services.[0].isolation:
  unknown isolation backend: vm
[123]

The isolation field must name one of the isolation backends, and the selected target must provide what that backend needs.

Service sends to a name the partition does not know

A service's send_to must name a peer service of the same partition or an endpoint the partition lists under requires.frame_bus. A name in neither is read as a misspelling and refused, and the refusal names both the sender and the partition. Use space run --dry-run to check the placement without booting.

General

Stale cache

If a build produces unexpected results after an image has been updated upstream, use an immutable image digest or a new tag, then rebuild the composition. Avoid relying on a mutable :latest tag when debugging.

Verbosity and machine output

space build, space run, space publish, space deploy, space image verify, space show --sbom, space status and space logs take --json for structured output, -v for more log detail (twice for debug, three times for protocol traces), and --log=SPEC for per-source levels. To rehearse without booting, use space run --dry-run, space deploy --plan, and space target apply --plan, which prints the provider plan and stops.

Development environment

Unikraft build fails with `flex` flags

flex: Can't use -f or -F with -l option

On some distributions, /usr/bin/lex contains a call to flex -l; on others, /usr/bin/lex is a symlink to flex. Force the build to call flex directly:

$ export LEX=flex

Unikraft 9pfs access does not work

ERR: [libvirtio_bus] Failed to find the driver for the virtio device 0x92020 (id:9)
ERR: [libvirtio_pci] Failed to register the virtio device: -14

The Unikraft 9p filesystem driver is gated behind an opt-in build flag. Use the development Unikraft release and enable 9pfs explicitly:

opam pin -ny https://github.com/mirage/ocaml-unikraft.git
opam install ocaml-unikraft-default-x86_64 \
             ocaml-unikraft-option-9pfs

(Substitute arm64 for the target architecture as appropriate.)

QEMU bridge access denied

access denied by acl file, bridge helper failed

QEMU's bridge helper runs setuid and consults an ACL file before attaching a tap interface to a bridge. Allow the current user:

echo "allow all" | sudo tee /etc/qemu/${USER}.conf
echo "include /etc/qemu/${USER}.conf" \
  | sudo tee --append /etc/qemu/bridge.conf
sudo chown root:${USER} /etc/qemu/${USER}.conf
sudo chmod 640 /etc/qemu/${USER}.conf

Packets not received on non-standard ports

Strict firewall configurations (typical on recent Fedora / RHEL / openSUSE) drop packets to non-standard ports even when no iptables rule is visible, because the rule lives in the nftables table managed by firewall-cmd. Inspect both:

iptables -L
nft list ruleset

Add the bridge to a firewall zone and open the relevant ports through firewall-cmd, marking the rule persistent with --persistent.

C++ linking fails on Unikraft

undefined reference to __cxa_call_terminate

This appears when compiling C++ with a recent GCC; Unikraft's bundled libcxxabi is several versions behind. Use these workarounds in order:

Build inside an older Debian image with a compatible toolchain; e.g. docker run --rm -v "$PWD":/src -w /src debian:11 bash -c 'apt-get install -y build-essential && kraft build'.
Use clang++ instead of g++. The two settings below select the clang toolchain, and Unikraft's CPP setting names the C++ compiler.
Drop to an older gcc (GCC 14 or earlier).

kraft system set toolchain.CC="clang"
kraft system set toolchain.CPP="clang++"

Incomplete unikernel stack traces

Wrap the unikernel's entry point in a try ... with and explicitly print the backtrace. Failures remain visible even when the generated entry point does not print the full exception by default.

`tap` interface drops out of the bridge

After qemu-bridge-helper attaches a tap interface to a bridge, the interface disappears from the bridge a short while later, and unikernel networking stops working. On systems running systemd-networkd, this is typically because the default 80-wired.network rule matches the tap device and attempts to manage it.

Override the match with a higher-precedence file that tells systemd-networkd to ignore tap interfaces:

[Match]
Kind=tun
Name=tap*

[Link]
Unmanaged=yes

Reload with sudo networkctl reload. Unikernel networking now survives across QEMU restarts.

Upstream systemd has a fix that scopes 80-wired.network to physical interfaces only; recent distributions may not need this workaround.