# Nerdctl Spike Evaluation

Date: 2026-03-12

## Goal

Evaluate whether per-tenant rootless `containerd` + `nerdctl` is a better fit than rootless Podman for treehouse tenant app hosting, with emphasis on:

- tenant-local storage
- rootless multi-service private networking
- Dockerfile build reliability
- systemd user-service fit
- public port publishing for Caddy

## Host changes for the spike

Committed in the repo:

- `21f39be` `Add containerd spike dependencies`
- `0e9c707` `Add runc for nerdctl spike`

NixOS now provides:

- `containerd`
- `nerdctl`
- `buildkit`
- `rootlesskit`
- `cni-plugins`
- `slirp4netns`
- `runc`

and enables:

- `virtualisation.containerd.enable = true;`

## Test environment

- Tenant: `podtest`
- UID: `2012`
- Rootless runtime dir: `/run/user/2012`
- Rootless storage:
  - `/home/podtest/.local/share/containerd`
  - `/home/podtest/.local/share/nerdctl`

## What worked

### 1. Rootless containerd can run under `systemd --user`

A transient tenant-local service worked:

```bash
systemd-run --user --unit=nerdctl-spike-containerd \
  -p Delegate=yes \
  -E PATH=... \
  -E HOME=/home/podtest \
  -E XDG_RUNTIME_DIR=/run/user/2012 \
  /tmp/nerdctl-rootless-docs/containerd-rootless.sh
```

Observed:

- service stayed up
- `rootlesskit` and `slirp4netns` started successfully
- `containerd` socket appeared at `/run/user/2012/containerd/containerd.sock`

This is already a meaningful difference from the current Podman failure mode:

- Podman rootless private networking failed in `slirp4netns` sandbox setup
- rootless `containerd-rootless.sh` did not hit that exact sandbox failure

### 2. Tenant-local storage works

Confirmed:

- runtime sockets/state live under `/run/user/2012/...`
- image/layer data lives under `/home/podtest/.local/share/containerd`

Measured after the spike:

- `/home/podtest/.local/share/containerd` about `13M`
- `/home/podtest/.local/share/nerdctl` about `76K`

So the storage-accounting story is good.

### 3. Rootless containers can run

This worked:

```bash
nerdctl --address unix:///run/user/2012/containerd/containerd.sock \
  run --rm --net none hello-world
```

So the stack is not fundamentally broken. Rootless `containerd` + `nerdctl` can launch containers on this host.

### 4. Dockerfile builds can work

Out of the box, `nerdctl build` did not work because it needs a tenant-local `buildkitd`.

A plain tenant user-service `buildkitd --rootless` failed with:

- `rootless mode requires to be executed as the mapped root in a user namespace`

But `buildkitd` did work when launched inside the existing rootless containerd namespace:

```bash
nsenter -U -m -n -t "$(cat /run/user/2012/containerd-rootless/child_pid)" -F \
  buildkitd --rootless \
  --addr unix:///run/user/2012/buildkit-default/buildkitd.sock \
  --oci-worker-rootless \
  --oci-worker-net host
```

With that in place, this Dockerfile built successfully:

- base image: `node:20-alpine`
- includes `RUN npm install --omit=dev`

That matters because it means the Node-build case that broke under Podman can succeed here.

### 5. Compose command exists

Confirmed:

```bash
nerdctl compose version
```

returned:

- `nerdctl Compose version v1.7.7`

So the UX surface exists, even though networking still determines whether it is actually usable for tenant apps.

## What failed

### 1. Rootless default network bridge is broken with the default detached-netns mode

Default `nerdctl run` failed with:

```text
failed to call cni.Setup: plugin type="bridge" failed (add):
failed to create bridge "nerdctl0": could not add "nerdctl0": operation not permitted
```

This affected:

- `nerdctl run hello-world`
- `nerdctl run -p 21050:3000 ...`
- `nerdctl run --net bridge ...`

In the default setup, this blocked:

- multi-service private networking
- published localhost ports

### 2. `--net host` was not a usable escape hatch

Testing with:

```bash
nerdctl run --rm --net host hello-world
```

failed with:

```text
error mounting "sysfs" to rootfs at "/sys" ... operation not permitted
```

So host networking did not provide a clean fallback.

### 3. The upstream setup helper is brittle in this environment

`containerd-rootless-setuptool.sh check` failed with:

```text
newuidmap ... write to uid_map failed: Operation not permitted
```

But the actual runtime could still be started successfully through a direct `systemd-run --user ... containerd-rootless.sh`.

So the helper script is not a reliable signal for this host.

### 4. Builds are not single-daemon simple

For real tenant builds, `nerdctl` needs:

- rootless `containerd`
- rootless `buildkitd`

and `buildkitd` itself had to be launched inside the rootless namespace to work correctly.

That means the stack is more operationally complex than “replace Podman with one daemonless tool”.

## Important debugging results

Manual bridge creation inside the rootless child namespace worked:

```bash
nsenter -U -n -m -t "$CHILD" -F sh -lc 'ip link add nerdctltest0 type bridge'
```

So this does **not** look like a hard kernel limitation or a total inability to do bridges rootlessly.

That pointed more toward:

- the current nerdctl/CNI hook path
- namespace handoff details
- or missing rootless-specific integration/config

In other words:

- the host can do the primitive
- the current default runtime wiring was not doing it correctly

### Follow-up networking spike result

The critical fix candidate was:

```bash
CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=false
```

When rootless `containerd` was launched with detached netns disabled:

- `nerdctl run --rm hello-world` worked on the default bridge network
- published localhost ports worked
- custom bridge networks worked
- service-to-service communication by container name worked

Concrete validation:

```bash
# launch rootless containerd
systemd-run --user ... \
  -E CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=false \
  /tmp/nerdctl-rootless-docs/containerd-rootless.sh

# create private network
nerdctl network create spike-net

# run service on that network
nerdctl run -d --name net-web --network spike-net nerdctl-spike-build:latest

# reach it from another container on the same network
nerdctl run --rm --network spike-net alpine wget -qO- http://net-web:3000

# publish a localhost port for host proxying
nerdctl run -d --name spike-web -p 127.0.0.1:21050:3000 nerdctl-spike-build:latest
curl http://127.0.0.1:21050
```

Observed results:

- inter-container request succeeded and returned `nerdctl spike ok`
- `curl http://127.0.0.1:21050` succeeded
- `nerdctl ps -a` showed the published port mapping

So the networking story changed substantially:

- default `detach-netns=auto/true`: broken bridge/CNI setup
- `detach-netns=false`: working rootless bridge networking on this host

## Comparison to Podman

### Better than current Podman

- tenant-local storage works
- rootless container runtime starts
- rootless Dockerfile builds with `RUN npm install` can be made to work
- does not hit the same `slirp4netns` sandbox mount failure that blocked Podman pods

### Worse / still not solved

- tenant app runtime networking is still not usable for the product yet
- private multi-service networking is not working
- public port publishing is not working
- build path requires extra per-tenant daemon wiring (`buildkitd`)

So this is not a drop-in rescue of the original Podman plan.

## Conclusion

`containerd` + `nerdctl` is a credible runtime candidate for:

- tenant-local image/layer storage
- rootless container execution
- rootless image builds

But it is **not currently a ready replacement** for tenant app hosting on treehouse, because the runtime networking layer still fails in the place that matters most:

- multi-service private networking
- published localhost ports for Caddy

Compared with Podman:

- Podman failed in rootless network helper sandbox creation
- nerdctl/containerd initially failed in the default detached CNI bridge path
- but nerdctl/containerd became workable once `CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=false` was set

That is a much stronger result than the earlier Podman spike.

## Recommendation

Do not migrate tenant app hosting to `containerd` + `nerdctl` immediately, but this is now a credible direction.

The leading candidate design is:

- rootless `containerd` per tenant
- rootless `buildkitd` launched inside the same rootless namespace
- `CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=false`

If we want to continue this line of investigation, the next step should turn the working spike into a deliberate managed configuration:

1. Encode `CONTAINERD_ROOTLESS_ROOTLESSKIT_DETACH_NETNS=false` in the tenant runtime service model.
2. Formalize tenant-local `buildkitd` startup in the same rootless namespace as `containerd`.
3. Re-test the exact platform contract:
   - app build
   - private multi-service networking
   - published localhost ports for Caddy
   - cleanup on tenant deletion
4. Measure any downsides of disabling detached netns:
   - build performance
   - host loopback exposure during builds
   - compatibility with `--net host`

The practical conclusion now is:

- `containerd` + `nerdctl` solves the storage boundary well
- builds can work
- networking can work on this host with the right rootlesskit policy
- this path is more promising than the current Podman approach
