---
name: treehouse-troubleshooting
description: Treehouse troubleshooting — common failure patterns, diagnosis steps, and fixes for nanoclaw, tenant containers, app deployments, Caddy, backups, and SSH access.
allowed-tools: Bash(ssh:*), Bash(sudo:*), Read, Write, Edit
---

# Treehouse Troubleshooting

SSH: `ssh treehouse-host '<command>'`

## Platform Health Check

```bash
treehouse doctor           # human-readable — ✓/⚠/✗ per check, with remediation hints
treehouse doctor --json    # structured JSON, exit 0=ok / 1=warnings / 2=failures
```

Checks run: nanoclaw/caddy/borgmatic services · Caddy config validation · backup recency (<25h) · tenant volume usage (warn ≥85%, fail ≥95%) · tenant containerd runtime health · stuck agent containers (>30 min) · app control-plane drift · buildkit restart churn.

**Always run `treehouse doctor` first** when investigating any platform issue. Each failing check prints a specific remediation command.

## Nanoclaw Service Issues

```bash
treehouse nanoclaw status
treehouse nanoclaw logs --lines 200
sudo journalctl -u nanoclaw.service -f
```

- **Agent not responding:** Check if the group is registered (`treehouse chats list`). Try SIGHUP: `treehouse nanoclaw hup`
- **Container stuck:** `treehouse agents ps` to list running containers; `treehouse agents kill --tenant <id> --all --yes` to force-kill
- **DB locked / startup failure:** Check `messages.db` is not corrupted — `sqlite3 /var/lib/nanoclaw/store/messages.db "PRAGMA integrity_check"`

## Tenant App Deployment Failures

```bash
treehouse apps list --tenant <id>
treehouse apps logs --tenant <id> --app <app>
```

- **Build fails:** Check buildkitd is running — `treehouse runtime ping --tenant <id>`
- **Container won't start:** Check nerdctl logs and `treehouse apps status --tenant <id> --app <app>`; verify `app.yml` uses the supported contract and does not define `ports:`
- **Containerd daemon crashed / container in Unknown state:** Restart the tenant user service: `sudo systemctl restart user@<uid>.service` then delete orphaned container records via `ctr --namespace default containers delete <id>`. **NEVER use `systemctl kill user-<uid>.slice`** — this kills ALL tenant services including running apps.
- **"port is already allocated" or app status disagrees with reality:** Run `treehouse apps reconcile --tenant <id> --fix` first. Then inspect `treehouse apps ports --tenant <id>` and `treehouse apps status --tenant <id> --app <app>`. If the same app still fails after reconcile, use `treehouse apps remove --tenant <id> --app <app> --yes` and re-deploy from `app.yml`.
- **App unreachable:** Check Caddy — `treehouse caddy lint`; verify app container is up with `nerdctl ps` (see `/treehouse-apps`)
- **Filesystem / snapshot corruption for one app:** Use the app-scoped reset path from the tenant side (`runtime-reset`) or admin-side app redeploy/removal flow. Do not use global image pruning as the first response.
- **Disk full:** `treehouse runtime df` — if volume at 100%, tenant needs to free space or volume needs resizing

## Caddy / Routing Issues

```bash
treehouse caddy lint
treehouse caddy reload
sudo journalctl -u caddy.service -n 50
```

- **TLS cert not issuing:** Verify DNS A record points to 89.167.99.240 and port 80 is reachable
- **502 Bad Gateway:** App container is not running or listening on wrong port
- **Caddyfile syntax error:** `treehouse caddy lint` will show the offending line

## Disk / ENOSPC

```bash
treehouse runtime df                  # per-tenant volume usage
df -h /                               # host root disk
```

- **Tenant ENOSPC:** The agent will notify the user. Admin can resize the loopback volume or help the tenant free space
- **Host disk full:** Check `/var/lib/docker` for stale images — `treehouse runtime prune` (with care)
- **Build cache bloat:** `nerdctl system prune` as the tenant user

## SSH Access

- Use `ssh treehouse-host '<command>'` from inside the admin container
- Key: `/home/node/.ssh/id_ed25519` (mounted read-only)
- Known hosts: `/home/node/.ssh/known_hosts` (mounted read-only)
- Do NOT use `ssh_cmd` or `hostbash` — removed

If SSH fails: check the key exists (`ls /home/node/.ssh/`), check known_hosts has the host entry, verify nanoclaw is running (SSH key is mounted by the runner).

## Mission Control Issues

```bash
treehouse mc restart --yes
sudo journalctl -u mission-control.service -n 50
```

- **UI out of date:** UI is a static build — run `treehouse mc restart --build-ui --yes` after any frontend change
- **API errors:** Check backend logs; MC backend is `server.mjs` on `mission-control.service`
- **MC unreachable:** Only accessible via Tailscale at `mission.null` — check Tailscale is connected

## Backups

```bash
treehouse backups status
sudo borgmatic list
```

- **Timer not running:** `systemctl status borgmatic.timer`
- **Connection refused to storage box:** SSH key or storage box config issue — check `/etc/nixos/borgmatic.nix`
- **Integrity check failure:** `sudo borgmatic check` — may require `--repair` if minor corruption
