---
name: treehouse-lcm
description: LCM (Lossless Context Memory) — how the per-tenant conversation memory system works, how to troubleshoot it, and how to develop/extend it. Use when investigating LCM issues, adding features, or understanding how memory compaction and retrieval work.
allowed-tools: Bash(ssh:*), Bash(sudo:*), Read, Write, Edit
---

# LCM — Lossless Context Memory

LCM stores every conversation turn to a per-group SQLite DB, injects recent turns into the agent's context on new sessions, and compacts old turns into summaries via Claude Haiku.

## Architecture

```
nanoclaw/src/lcm.ts          — core: DB, store, assemble, compact, search, embeddings
nanoclaw/src/embedder.ts     — lazy-loaded nomic-embed-text-v1.5 singleton (ONNX, 768-dim)
nanoclaw/src/index.ts        — turn storage hook (onOutput callback) + compaction trigger + embedder warmup
nanoclaw/src/ipc.ts          — IPC handlers: lcm-compact, lcm-search, lcm-semantic-search, lcm-hybrid-search, lcm-expand, lcm-describe
sdk/src/modules/lcm.ts       — SDK: lcmSearch(), lcmSemanticSearch(), lcmHybridSearch(), lcmDescribe()
sdk/cli/tc.js                — `tc memory search [--semantic|--match] <query>` / `tc memory describe [id]`
container/skills-platform/treehouse-rules/SKILL.md  — agent rules for using LCM
container/skills-platform/treehouse-tc/SKILL.md     — tc memory commands doc
```

## Enable/Disable

**Global (all tenants):**
```bash
# Enable
echo "LCM_ENABLED=true" >> /home/admin/treehouse/nanoclaw/.env
sudo systemctl restart nanoclaw

# Disable
sed -i '/LCM_ENABLED/d' /home/admin/treehouse/nanoclaw/.env
sudo systemctl restart nanoclaw
```

**Per-tenant:**
```bash
# Enable
touch /home/<tenant>/nanoclaw/data/.lcm-enabled

# Disable
rm /home/<tenant>/nanoclaw/data/.lcm-enabled
```
Note: global `LCM_ENABLED=true` overrides per-tenant flag files.

## DB Location

```
/home/<tenant>/nanoclaw/data/lcm/<group-folder>.db
```

Examples:
```bash
sudo sqlite3 /home/koala/nanoclaw/data/lcm/general.db ".tables"
sudo sqlite3 /home/koala/nanoclaw/data/lcm/general.db "SELECT COUNT(*) FROM turns;"
sudo sqlite3 /home/koala/nanoclaw/data/lcm/general.db \
  "SELECT role, substr(content,1,80), timestamp FROM turns ORDER BY timestamp DESC LIMIT 10;"
sudo sqlite3 /home/koala/nanoclaw/data/lcm/general.db \
  "SELECT id, depth, substr(content,1,100), created_at FROM summary_nodes ORDER BY created_at DESC LIMIT 5;"
```

## DB Schema

```sql
-- Core tables (schema v1)
turns (id, session_id, seq, role, content, input_tokens, timestamp)
summary_nodes (id, depth, content, source_turn_ids, parent_id, created_at)
context_items (id, assembled_at, item_type, item_id, position)
lcm_meta (key, value)

-- Embeddings: BLOB fallback (schema v2, used when sqlite-vec unavailable)
embeddings (id, item_type, item_id, embedding BLOB, created_at)
  -- embedding BLOB = Float32Array serialized, 768 × 4 bytes = 3072 bytes/row

-- Embeddings: sqlite-vec path (created by ensureVecTables() when SQLITE_VEC_PATH is set)
embedding_meta (vec_rowid INTEGER PK, item_type, item_id UNIQUE, created_at)
vec_embeddings VIRTUAL TABLE USING vec0(embedding float[768])
  -- vec_rowid links embedding_meta → vec_embeddings for KNN joins
```

**sqlite-vec** is loaded from `SQLITE_VEC_PATH` (set in NixOS nanoclaw config to `${pkgs.sqlite-vec}/lib/vec0.so`). If the extension loads successfully, the vec0 path is used; otherwise falls back to BLOB + in-memory cosine similarity.

**Insert order (vec0 path):** vec0 must be inserted without a rowid param (sqlite-vec 0.1.x rejects bound rowid params). Insert `vec_embeddings` first → auto-assigned rowid → store in `embedding_meta.vec_rowid`.

## Turn Storage Flow

1. Agent responds → `onOutput` callback fires with `result.status === 'success'`
2. `storeTurnPair()` stores both user and agent turns immediately
3. **Piped turns** (same container, multiple rounds): `pipedMessageContent` Map holds the actual user message, set at `queue.sendMessage()` call site in the message loop
4. Fallback: if `storeTurnPair` never ran (e.g. container error), post-`runAgent` code stores it

Key variables in `index.ts`:
- `pipedMessageContent: Map<chatJid, string>` — module-level, set when message piped to active container
- `turnStored: boolean` — per-processGroup closure, prevents double-storage
- `sentAssistantTexts: string[]` — accumulates agent text, reset after each piped round

## Context Injection

`assembleLcmContext(groupFolder)` in `lcm.ts` returns a `<lcm_context>` block prepended to the user prompt in `container-runner.ts:811`. Contains:
- **Summary nodes** (oldest to newest, if any)
- **Fresh tail**: last `FRESH_TAIL_TURNS = 32` raw turns

The block includes a comment identifying it as long-term stored memory.

## Compaction

**Thresholds** (in `lcm.ts`):
```typescript
FRESH_TAIL_TURNS = 32          // turns injected verbatim
COMPACTION_FRESH_TAIL = 32     // turns exempt from compaction
COMPACTION_SOFT_THRESHOLD = 200_000 * 0.75  // ~150k tokens to trigger
```

**Trigger**: checked in `onOutput` callback after each successful turn store. Fires async `runCompaction()` if `shouldCompact()` returns true.

**API auth**: uses `ANTHROPIC_API_KEY` if set, falls back to `CLAUDE_CODE_OAUTH_TOKEN` from `.env`. Both `index.ts` and `ipc.ts` read these.

**Compaction model**: `claude-haiku-4-5-20251001`

**DAG structure**: depth-0 summaries cover raw turn groups; depth-1 summaries cover depth-0 groups, etc.

## Search & Retrieval (Agent-Facing)

Three search modes — agents default to hybrid:

```bash
tc memory search <query>             # hybrid (default) — RRF merge of keyword + semantic
tc memory search --semantic <query>  # pure semantic via nomic-embed-text-v1.5 + sqlite-vec KNN
tc memory search --match <query>     # pure keyword/FTS — LIKE match on turns + summary_nodes
```

**Hybrid search IPC flow:**
```
tc memory search → sdk.lcmHybridSearch() → IPC lcm-hybrid-search
→ ipc.ts handler → hybridSearchLcm() → Promise.all([searchLcm(), semanticSearchLcm()])
→ RRF merge (k=60) → sorted by combined score → JSON result file → tc prints
```

**Semantic search IPC flow:**
```
tc memory search --semantic → sdk.lcmSemanticSearch() → IPC lcm-semantic-search
→ ipc.ts handler → semanticSearchLcm() → embed query → vec0 KNN (or in-memory cosine)
→ JSON result file → tc prints
```

**Keyword search IPC flow:**
```
tc memory search --match → sdk.lcmSearch() → IPC lcm-search
→ ipc.ts handler → searchLcm() → LIKE match on turns + summary_nodes → tc prints
```

**Embedding pipeline:**
- Model: `nomic-embed-text-v1.5` (ONNX, 768-dim, 8192-token input, ~274MB cached to `/var/lib/nanoclaw/.cache/transformers`)
- Lazy singleton in `embedder.ts` — loads once on first call, thread-safe via promise deduplication
- Fires async after every `storeTurn` and `storeTurnPair` (non-blocking, best-effort)
- Pre-warmed at nanoclaw startup if LCM is globally enabled

## Troubleshooting

**Turns not being stored:**
- Check if LCM is enabled: `LCM_ENABLED` in NixOS config env or `.lcm-enabled` flag file
- Check nanoclaw logs: `sudo journalctl -u nanoclaw -n 50 | grep -i lcm`
- Verify DB exists and has rows: `sudo sqlite3 /home/<tenant>/nanoclaw/data/lcm/<group>.db "SELECT COUNT(*) FROM turns;"`

**Embeddings not being stored:**
- Check if sqlite-vec is loaded: `sudo sqlite3 /home/<tenant>/nanoclaw/data/lcm/<group>.db "SELECT COUNT(*) FROM embedding_meta;"`
- If `embedding_meta` doesn't exist, sqlite-vec extension failed to load — check `SQLITE_VEC_PATH` env var in nanoclaw service
- If `embedding_meta` has rows but `vec_embeddings` doesn't: old schema bug (pre-fix). The new code auto-detects and drops/recreates the tables on next DB open.
- Embedding is fire-and-forget — errors are silently swallowed. Check embedder logs: `sudo journalctl -u nanoclaw -n 100 | grep -i embed`
- Model download: first use downloads ~274MB to `/var/lib/nanoclaw/.cache/transformers`. Check disk space.

**Semantic search returns no results:**
- Confirm embeddings exist: `SELECT COUNT(*) FROM embedding_meta;` in the DB
- Confirm vec_embeddings matches: `SELECT COUNT(*) FROM vec_embeddings;` — should equal embedding_meta count
- Test sqlite-vec directly: load `SQLITE_VEC_PATH` extension and run a KNN query
- Fallback path: if sqlite-vec unavailable, semantic search uses in-memory cosine (slower but functional)

**Compaction not firing:**
- Needs `ANTHROPIC_API_KEY` or `CLAUDE_CODE_OAUTH_TOKEN` in `/home/admin/treehouse/nanoclaw/.env`
- Threshold is ~150k tokens of old turns — won't fire until there are many turns
- Check logs for: `LCM: compaction failed`
- After code changes, SIGHUP is NOT sufficient — requires `sudo systemctl restart nanoclaw`

**Agent not using tc memory search:**
- Rules are in `container/skills-platform/treehouse-rules/SKILL.md`
- Skills are synced from host at container spawn (no image rebuild needed)
- Kill the active container to force a fresh spawn with updated skills

**Wrong user content stored for piped turns:**
- Check `pipedMessageContent` map logic in `index.ts`
- Map is set at `queue.sendMessage()` call site and consumed in `onOutput`

## Development Notes

- **SIGHUP** only reloads tenant/group config — does NOT reload compiled JS. Always `sudo systemctl restart nanoclaw` after code changes.
- **Skills changes** take effect on next container spawn — no image rebuild needed.
- **Agent-runner changes** require Docker image rebuild: `cd /home/admin/treehouse/nanoclaw/container && sudo docker build -t nanoclaw-agent:latest .`
- **SDK changes**: rebuild with `cd /home/admin/treehouse/sdk && npm run build`, then available immediately (mounted at runtime).
- **Test thresholds**: temporarily lower `COMPACTION_FRESH_TAIL = 2` and `COMPACTION_SOFT_THRESHOLD = 10` in `lcm.ts` to test compaction with just 3 turns. Remember to restore and restart.
