# Run microVMs in a KVM sandbox

URL: https://briq.cloud/docs/guides/run-microvms-in-a-kvm-sandbox

> Give the agent a sandbox with /dev/kvm, to run a hypervisor of its own such as Firecracker and the microVMs it starts.

Recipe:

- [Nested virtualization](https://briq.cloud/cookbook/kvm-sandbox.md): A sandbox with /dev/kvm, to run a hypervisor of your own, for example Firecracker, and the microVMs it starts. The same as the sandbox recipe: briq_exec for commands, `steps` at boot. Slower to wake: every start is a cold boot, with no memory kept. Default size large (4 CPUs, 8 GB); Pro and Team plans.

An agent develops or tests something that starts virtual machines: a microVM runtime, a
Firecracker setup, a test suite that boots guests. The [sandbox](https://briq.cloud/docs/guides/run-code-in-a-sandbox)
has no `/dev/kvm`. The `kvm-sandbox` recipe is the same sandbox with it, and with Firecracker in
`/usr/local/bin`. First, [connect your agent](https://briq.cloud/docs/getting-started).

> Check that this sandbox can run Firecracker, then run the project's tests in it.

```text
briq_run({ recipe: "kvm-sandbox" })
→ bq_3m8w1x5k7qzd sandbox (ghcr.io/briq-cloud/sandbox:kvm) running · cost so far … · sleeps at 18:42 UTC unless used
  outputs:
    WORKDIR=/work

briq_exec({ briq: "bq_3m8w1x5k7qzd", cmd: ["ls", "-l", "/dev/kvm"] })
→ crw… /dev/kvm

briq_exec({ briq: "bq_3m8w1x5k7qzd", cmd: ["firecracker", "--version"] })
→ Firecracker v…
```

1. `briq_run` starts a `large` microVM (4 CPUs, 8 GB) with a 40 GB volume at `/work`.
2. The boot runs `kvm-ready` before the `steps`: udev, the ZFS module, the cgroup controllers.
   It logs one line, `kvm-ready: /dev/kvm, kernel …, zfs 2.4.1, cgroup controllers: …`. If it
   fails, the start fails and `briq_logs` says why.
3. `briq_exec` runs commands as in the sandbox. `/dev/kvm` and `firecracker` are there for them.

## Boot a microVM

A guest needs a kernel and a root filesystem. The sandbox brings neither: use your own, or follow
Firecracker's
[getting started](https://github.com/firecracker-microvm/firecracker/blob/main/docs/getting-started.md),
which starts a guest from `/usr/local/bin/firecracker`. Put the files on `/work`: the volume is kept
while the sandbox sleeps.

## Run a test suite

A suite that boots guests takes minutes. Run it with `async: true` and read the result when it is
done ([Run a long command](https://briq.cloud/docs/guides/run-a-long-command)):

```text
briq_exec({ briq: "bq_…", cmd: ["sh", "-c", "cd /work/app && cargo test"], async: true })
→ ex_5t2n8k1w9qrd running since 10:02 UTC
briq_exec_result({ briq: "bq_…", exec: "ex_5t2n8k1w9qrd" })
→ ex_5t2n8k1w9qrd exited 0 after 14m
```

A running async command counts as use, so the sandbox does not sleep under it. Work briq cannot
see (a daemon you started, a guest left running) needs a longer `sleep`, `"3h"` or `"never"`:

```text
briq_run({ recipe: "kvm-sandbox", sleep: "3h" })
```

## Set up at boot

Clone and build in `steps`, as in the sandbox. They run at every start, and every start is a cold boot:

```text
briq_run({
  recipe: "kvm-sandbox",
  steps: [{ run: "git clone --depth 1 https://github.com/acme/vmm ." }]
})
```

## Limits

* **x86\_64 only.**
* **Not everywhere.** `briq_catalog` lists the recipe only where `/dev/kvm` is available.
* **The agent's policy has to allow the size.** Otherwise `policy_violation`: see `allowedSizes` in
  [Keys and policy](https://briq.cloud/docs/concepts/keys-and-policy).
* **Every start is a cold boot.** No memory is kept while the sandbox sleeps. A guest running
  inside is gone after a sleep, and so is every process: start them again. Files on `/work` stay.
* **A long run needs `sleep`.** A longer value, or `"never"`, keeps the sandbox up.
* **Pro and Team plans.** A Hobby team gets a `plan_limit` error.
* **`large` by default.** `xlarge` for a full test suite or many VMs at once; see [Cost](https://briq.cloud/docs/cost).
* **Kernel modules come from the host.** The ZFS module matches the host's kernel, not the
  image; another module the kernel does not have cannot be loaded.
* **A fork copies the disk only.** The guests it runs are not in it.

## Run real systemd units

`systemd` is in the image, but it is not the init. For code that needs real units, start it as PID 1
of namespaces of its own, with `/` shared as on a server:

```text
briq_exec({ briq: "bq_…", cmd: ["unshare", "--pid", "--fork", "--mount", "--cgroup", "--mount-proc", "sh", "-c",
  "mount -t cgroup2 none /sys/fs/cgroup; mount --make-rshared /; exec env container=lxc /usr/lib/systemd/systemd"], async: true })
```

Commands enter them with `nsenter -t <pid of systemd> -a`. Without the shared `/`, a sandboxed unit
keeps copies of mounts made after it starts, and their unmount answers `busy`. A complete setup also
gives systemd its own `/run` and masks the units a container has no use for.

## Example: a microVM runtime's test suites

A Rust runtime that runs microVMs on Firecracker. Its suites need `/dev/kvm`, ZFS, nftables, systemd
units and real guests. All of them ran in one sandbox, with a long `sleep` because they ran detached,
where the briq cannot see them.

Measured on one `xlarge`, AMD Ryzen host, 2026-11-05, stage times to within 15 s:

| Stage                                            | Result            | Time           |
| ------------------------------------------------ | ----------------- | -------------- |
| Release build, from nothing                      |                   | 95 s           |
| Install: systemd, then the daemon under its unit |                   | 30 s           |
| Start a guest inside, image already pulled       |                   | 1.02 s, 0.83 s |
| `cargo test --workspace`                         | 1434 passed       | 197 s          |
| Fleet smoke                                      | 33 passed         | 61 s           |
| End-to-end                                       | 294 passed        | 561 s          |
| Stress, with crashes                             | 635 checks passed | 501 s          |
| Crashes, isolation, a full disk                  | 134 checks passed | 611 s          |
| Alerts fire and resolve                          | 45 checks passed  | 880 s          |