# LCM: Lossless Context Management for Nanoclaw

**Task ID:** mmvbhkvqqlyn
**Created:** 2026-03-18
**Status:** In Progress
**Reference:** ~/lossless-claw (MIT, TypeScript/SQLite), papers.voltropy.com/LCM

---

## Problem

Current nanoclaw context model has critical gaps:

- Only inbound user messages are stored — agent responses are discarded
- Memory depends entirely on native Claude/Codex session resumption — when sessions expire or transcripts are lost, the agent forgets everything with no warning
- In-session compaction is opaque — Claude wipes context silently; nanoclaw archives the transcript to markdown but never feeds it back
- No cross-session search — agents cannot query their own history
- Container teardown (30-minute idle timeout) destroys anything not in the transcript

---

## Goals

1. Every conversation turn (user + agent) persisted to a per-group SQLite DB
2. Context assembled from recent raw turns + summarised history (DAG) on each container start
3. Compaction runs via ephemeral tenant containers (billing attributed correctly)
4. Agents can search and expand history via MCP tools
5. Graceful checkpoint on service restart / planned maintenance
6. Works for both Claude and Codex providers

---

## Architecture

### Key Decisions

- **Feature flag:** `LCM_ENABLED=true` env var — off by default, opt-in per tenant initially
- **DB per group:** `/home/<tenant>/nanoclaw/data/lcm/<folder>.db` — one SQLite DB per conversation thread
  - e.g. `/home/admin/nanoclaw/data/lcm/dm-110785097162035200.db`
  - Natural tenant isolation; lives in tenant home dir so borgmatic backs it up automatically
  - Per-group files avoid SQLite write contention between concurrent group turns
- **Separate from messages.db:** No migration risk; easy to wipe/rebuild; messages.db stays as platform bus and UI source
- **lossless-claw as local dependency:** Core logic (store, compaction, assembler, summarize) reused directly; OpenClaw-specific bridge/engine replaced with nanoclaw-specific wiring in `lcm.ts`

### What Gets Stored Per Turn

- User message text
- Agent response text — concatenation of `sentAssistantTexts` + final result (deduplicated)
- **NOT** progress messages (tool-use status updates — ephemeral UI feedback only)
- Input token count (from Claude usage metadata)
- Timestamp, session ID, sequence number
- **Content cap:** 50,000 characters per field with truncation marker — prevents large tool outputs from bloating the DB

### Storage Layout

Per-group SQLite DB:

```
/home/<tenant>/nanoclaw/data/lcm/<folder>.db
```

Tables:

**turns**
```sql
CREATE TABLE turns (
  id TEXT PRIMARY KEY,
  session_id TEXT NOT NULL,
  seq INTEGER NOT NULL,
  role TEXT NOT NULL,        -- 'user' | 'agent'
  content TEXT NOT NULL,
  input_tokens INTEGER DEFAULT 0,
  timestamp TEXT NOT NULL
);
```

**summary_nodes**
```sql
CREATE TABLE summary_nodes (
  id TEXT PRIMARY KEY,
  depth INTEGER NOT NULL,
  content TEXT NOT NULL,
  source_turn_ids TEXT NOT NULL,  -- JSON array of turn IDs
  parent_id TEXT,
  created_at TEXT NOT NULL,
  FOREIGN KEY (parent_id) REFERENCES summary_nodes(id)
);
```

**context_items** (debug/audit only)
```sql
CREATE TABLE context_items (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  assembled_at TEXT NOT NULL,
  item_type TEXT NOT NULL,   -- 'turn' | 'summary'
  item_id TEXT NOT NULL,
  position INTEGER NOT NULL
);
```

### Context Assembly

In `container-runner.ts`, before container launch:
- Query recent N turns (fresh tail, default 32) as raw text
- Query summary DAG for older context
- Inject assembled context into the initial prompt as a structured block:

```
<lcm_context>
[summary nodes — oldest to newest]

[recent raw turns — user/agent pairs]
</lcm_context>
```

- On new session: inject full assembled context
- On resumed session: inject only if session token count is high

### Compaction

Triggered post-turn when estimated token count exceeds threshold (~75% of model context window):

1. Identify turns outside fresh tail not yet summarised
2. Launch ephemeral tenant container (no platform context, no skills, no IPC tools)
3. Prompt: summarise chunk, return text, exit
4. Write summary node to DB, link to source turn IDs
5. Cascade: if summary count at depth N >= threshold, condense into depth N+1 node

**Summarisation prompts** (ported from lossless-claw/src/summarize.ts):
- System: "You are a context-compaction summarization engine. Follow user instructions exactly and return plain text summary content only."
- Leaf normal: preserve key decisions, rationale, constraints, active tasks. Target 35% of input tokens, max 1200.
- Leaf aggressive: keep only durable facts, TODOs, blockers, decisions. Target 20%, max 640.
- D1 condensed: session-level, ~2000 tokens, include timeline with timestamps.
- D2 condensed: phase-level, trajectory not minutiae, include timeline with dates.
- D3+ condensed: long-term memory, durable context only, brief date-range timeline.
- All end with: "Expand for details about: [what was dropped]"

**Three-level escalation** guarantees convergence:
1. Normal summarisation
2. Aggressive (bullet points, T/2 target)
3. Deterministic truncate to 512 tokens — no LLM, always converges

Compaction container uses the tenant configured model/provider by default. Optional `lcm_model` / `lcm_provider` per-tenant override for cheaper summarisation (e.g. Haiku).

### PreCompact Hook (Claude only)

Existing `createPreCompactHook` fires before Claude wipes in-session context. Extend to:
- Write pre-compact messages to LCM DB if not already captured
- Trigger immediate compaction run

### Graceful Checkpoint (SIGTERM)

On nanoclaw receiving SIGTERM:
- Drop `_checkpoint` sentinel in each group IPC dir
- Wait for containers to flush current transcript to LCM DB
- Then exit

### MCP Retrieval Tools

New tools in `ipc-mcp-stdio.ts`:

- **`lcm_search(pattern)`** — full-text search across turn history and summaries
- **`lcm_expand(summary_id)`** — expand a summary node back to source turns. **Sub-agents only** — main loop returns an error. Prevents context flooding in the primary interaction loop.
- **`lcm_describe(id)`** — describe a time range, session, or summary node

---

## Files Affected

- `nanoclaw/src/lcm.ts` — **New:** LcmStore class, assembleLcmContext(), runCompaction()
- `nanoclaw/src/index.ts` — Turn capture post-turn; SIGTERM checkpoint handler
- `nanoclaw/src/container-runner.ts` — Context injection pre-launch; compaction trigger post-turn
- `nanoclaw/src/hooks-server.ts` — Turn capture post-turn (hooks path)
- `nanoclaw/container/agent-runner/src/index.ts` — Single-shot compaction mode; extend PreCompact hook
- `nanoclaw/container/agent-runner/src/ipc-mcp-stdio.ts` — lcm_search, lcm_expand, lcm_describe MCP tools
- `nanoclaw/nixos/borgmatic.nix` — Include LCM DB paths in backup

---

## Implementation Phases

### Phase 1 — Foundation (turn capture + storage)
- [ ] Design LCM SQLite schema, write migrations
- [ ] Add input_tokens capture from Claude usage metadata per turn
- [ ] Write LcmStore class (thin wrapper around better-sqlite3)
- [ ] Hook turn capture into `index.ts` post-turn
- [ ] Hook turn capture into `hooks-server.ts` post-turn
- [ ] Store agent response alongside user message
- [ ] Feature flag: LCM_ENABLED check wrapping all writes
- [ ] Unit tests: schema, insert, query, content truncation
- [ ] **Git commit**

### Phase 2 — Context injection
- [ ] Add `assembleLcmContext()` — recent turns + summary DAG -> prompt string
- [ ] Inject assembled context in `container-runner.ts` before container launch
- [ ] Handle new session vs resumed session
- [ ] Unit tests: context assembly, empty DB case, large context handling
- [ ] **Git commit**

### Phase 3 — Compaction
- [ ] Add single-shot compaction mode to agent-runner
- [ ] Add `runCompaction()` in `lcm.ts`
- [ ] Wire post-turn compaction trigger (token threshold check)
- [ ] Extend `createPreCompactHook` to flush to LCM DB and trigger compaction
- [ ] Port summarize prompts from lossless-claw with three-level escalation
- [ ] Unit tests: threshold trigger, DAG cascade, escalation fallback
- [ ] **Git commit**

### Phase 4 — Retrieval tools
- [ ] Add `lcm_search` MCP tool
- [ ] Add `lcm_expand` MCP tool (sub-agent only enforcement)
- [ ] Add `lcm_describe` MCP tool
- [ ] Rebuild agent-runner, clear stale copies
- [ ] Unit tests: search, expand, describe
- [ ] **Git commit**

### Phase 5 — Resilience
- [ ] SIGTERM handler to checkpoint active sessions before exit
- [ ] Codex: turn-by-turn capture without PreCompact hook
- [ ] LCM DB paths added to borgmatic backup config
- [ ] Optional `lcm_model` / `lcm_provider` override per tenant
- [ ] Integration test: SIGTERM during active turn, DB integrity check
- [ ] **Git commit**

---

## Testing Strategy

### Unit Tests (nanoclaw/src/lcm.test.ts)

- Schema migrations run clean on fresh DB
- Schema migrations are idempotent (run twice, no error)
- Turn insert: user + agent rows appear with correct fields
- Content truncation: content > 50k chars is truncated with marker
- Progress messages: NOT stored (only user and agent roles)
- assembleLcmContext: returns empty string for empty DB
- assembleLcmContext: returns recent N turns as formatted text
- assembleLcmContext: includes summary nodes above fresh tail
- Three-level escalation: Level 3 always converges
- Summary DAG: cascade triggers when leaf count >= threshold

### Integration Tests

- After a full turn in index.ts: both user and agent rows in LCM DB
- Container launch: assembled context appears in prompt
- Compaction: post-turn trigger fires when token estimate exceeds threshold
- lcm_search: returns matching turns
- lcm_expand (sub-agent): returns source turns
- lcm_expand (main agent): returns error — not permitted

---

## E2E Testing Instructions

### Prerequisites
- `LCM_ENABLED=true` set for the test tenant
- Access to a live Discord/WhatsApp group on the test tenant
- Ability to send `/clear` to reset Claude sessions

### Test 1 — Basic turn capture (Phase 1)
1. Send to the group: "My favourite number is 42"
2. Confirm the agent responds
3. On host: `sqlite3 /home/<tenant>/nanoclaw/data/lcm/<folder>.db "SELECT role, content FROM turns ORDER BY seq"`
4. **Expected:** Two rows — one `user` row with the message, one `agent` row with the response

### Test 2 — Context survives session clear (Phase 2)
1. Send: "My favourite number is 42"
2. Send: `/clear` to reset the native Claude session
3. Send: "What is my favourite number?"
4. **Expected:** Agent replies with 42 — context came from LCM injection, not Claude session memory

### Test 3 — Context survives service restart (Phase 5)
1. Send: "Remember that the project codename is Aurora"
2. On host: `ssh treehouse-host 'sudo systemctl restart nanoclaw'`
3. Send: "What was the project codename?"
4. **Expected:** Agent replies with Aurora

### Test 4 — Compaction (Phase 3)
1. Have a long conversation (~50+ turns) with the agent
2. On host: `sqlite3 /home/<tenant>/nanoclaw/data/lcm/<folder>.db "SELECT id, depth, substr(content,1,100) FROM summary_nodes"`
3. **Expected:** At least one summary_node row exists

### Test 5 — lcm_search (Phase 4)
1. Tell the agent: "The API key prefix is sk-prod"
2. Ask: "Search your memory for API key"
3. **Expected:** Agent uses lcm_search and finds the turn mentioning "sk-prod"

### Test 6 — lcm_expand restriction (Phase 4)
1. Ask the agent directly: "Use lcm_expand on any summary node"
2. **Expected:** Agent receives an error — lcm_expand is only available in sub-agent sessions

---

## Future Work (Post Phase 5)

- **messages.db pruning:** Rolling 30-day retention on `messages` table with scheduled VACUUM once LCM is authoritative history
- **Large file handling:** Exploration Summary for tool outputs > 25k tokens (store by path, not inline)
- **Embedding index:** Semantic search over summary nodes as a complementary retrieval pathway
- **Storage estimate:** ~5-10MB/day platform-wide at current usage. Not a concern at current scale.
