session-viewer/docs/prd-jsonl-first-discovery.md

# PRD: JSONL-First Session Discovery

## Status: Ready for Implementation

## Context

Session viewer relies exclusively on `sessions-index.json` files that Claude Code maintains. These indexes are unreliable — a known, widespread bug with multiple open GitHub issues ([#22030](https://github.com/anthropics/claude-code/issues/22030), [#21610](https://github.com/anthropics/claude-code/issues/21610), [#18619](https://github.com/anthropics/claude-code/issues/18619), [#22114](https://github.com/anthropics/claude-code/issues/22114)).

### Root cause

Claude Code updates `sessions-index.json` only at session end. If a session crashes, is killed, or is abandoned, the JSONL file is written but the index is never updated. Multiple concurrent Claude instances can also corrupt the index (last-write-wins on a single JSON file). There is no reindex command and no background repair process.

### Impact on this system

- **542 unindexed JSONL files** across all projects (87 MB total)
- **48 unindexed in last 7 days** (30.8 MB)
- **13 projects** have JSONL session files but no index at all
- **Zero sessions from today** (Feb 4, 2026) appear in any index
- **3,103 total JSONL files** vs **2,563 indexed entries** = 17% loss rate

### Key insight

The `.jsonl` files are the source of truth. The index is an unreliable convenience cache. The session viewer must treat it that way.

## Requirements

### Must have

1. **All sessions with a `.jsonl` file must appear in the session list**, regardless of whether they're in `sessions-index.json`
2. **Exact message counts** — no estimates, no approximations. Contract: Tier 3 extraction MUST reuse the same line-classification logic as `parseSessionContent` (shared helper), so list counts cannot drift from detail parsing.
3. **Performance**: Warm start (cache exists, few changes) must complete under 1 second. Cold start (no cache) is acceptable up to 5 seconds for first request
4. **Correctness over speed** — never show stale metadata if the file has been modified
5. **Zero config** — works out of the box with no setup or external dependencies

### Should have

6. Session `summary` extracted from the last `type="summary"` line in the JSONL
7. Session `firstPrompt` extracted from the first non-system-reminder user message
8. Session `duration` MUST be derivable without relying on `sessions-index.json` — extract first and last timestamps from JSONL when index is missing or stale
9. Persistent metadata cache survives server restarts

### Won't have (this iteration)

- Real-time push updates (sessions appearing in UI without refresh)
- Background file watcher daemon
- Integration with `cass` as a search/indexing backend
- Rebuilding Claude Code's `sessions-index.json`

## Technical Design

### Architecture: Filesystem-primary with tiered metadata lookup

```
discoverSessions()
  |
  +-- For each project directory under ~/.claude/projects/:
  |     |
  |     +-- fs.readdir() --> list all *.jsonl files
  |     +-- Read sessions-index.json (optional, used as pre-populated cache)
  |     |
  |     +-- Batch stat all .jsonl files (bounded concurrency)
  |     |     Files that disappeared between readdir and stat are silently skipped (TOCTOU race)
  |     |
  |     +-- For each .jsonl file:
  |     |     |
  |     |     +-- Tier 1: Check index
  |     |     |     Entry exists AND normalize(index.modified) matches stat mtime?
  |     |     |     --> Use index content data (messageCount, summary, firstPrompt)
  |     |     |     --> Use stat-derived timestamps for created/modified (always)
  |     |     |
  |     |     +-- Tier 2: Check persistent metadata cache
  |     |     |     path + mtimeMs + size match?
  |     |     |     --> Use cached metadata (fast path)
  |     |     |
  |     |     +-- Tier 3: Extract metadata from JSONL content
  |     |           Read file, lightweight parse using shared line iterator + counting helper
  |     |           --> Cache result for future lookups
  |     |
  |     +-- Collect SessionEntry[] for this project
  |
  +-- Merge all projects
  +-- Sort by modified (descending) — always stat-derived, never index-derived
  +-- Async: persist metadata cache to disk (if dirty)
```

### Tier explanation

| Tier | Source | Speed | When used | Trusts from source |
|------|--------|-------|-----------|--------------------|
| 1 | `sessions-index.json` | Instant (in-memory lookup) | Index exists, entry present, `normalize(modified)` matches actual file mtime | `messageCount`, `summary`, `firstPrompt` only. Timestamps always from stat. |
| 2 | Persistent metadata cache | Instant (in-memory lookup) | Index missing/stale, but file hasn't changed since last extraction (mtimeMs + size match) | All cached fields |
| 3 | JSONL file parse | ~5-50ms/file | New or modified file, not in any cache | Extracted fresh |

Tier 1 reuses Claude's index when it's valid — no wasted work. The index `modified` field (ISO string) is normalized to milliseconds and compared against the real file `stat.mtimeMs`. If the index is missing or corrupt, discovery continues with Tier 2 and 3 without error. Even when Tier 1 is valid, `created` and `modified` timestamps on the `SessionEntry` always come from `fs.stat` — the index is a content cache only.

### Tier 1: Index validation details

The actual `sessions-index.json` format has `created` and `modified` as ISO strings, not a `fileMtime` field. Tier 1 validation must:

1. Map JSONL filename to sessionId: `sessionId := path.basename(jsonlFile, '.jsonl')`
2. Look up `sessionId` in the index `Map<string, IndexEntry>`
3. Compare `new Date(entry.modified).getTime()` against `stat.mtimeMs` — reject if they differ by more than 1000ms (accounts for ISO string → filesystem mtime rounding)
4. If the index entry has no `modified` field, skip Tier 1 (fall through to Tier 2)
5. When Tier 1 is valid, trust only content fields (`messageCount`, `summary`, `firstPrompt`). The `created`/`modified` on the resulting `SessionEntry` must come from `stat.birthtimeMs`/`stat.mtimeMs` respectively — this ensures list ordering is never stale even within the 1s mtime tolerance window.

### Shared line-iteration and counting (parser parity contract)

The biggest correctness risk in this design is duplicating any JSONL processing logic. The real parser in `session-parser.ts` has non-trivial expansion rules:

- User array content: expands `tool_result` and `text` blocks into separate messages
- `system-reminder` detection reclassifies user `text` blocks as `system_message`
- Assistant array content: `thinking`, `text`, and `tool_use` each become separate messages
- `progress`, `file-history-snapshot`, `summary` → 1 message each
- `system`, `queue-operation` → 0 (skipped)

It also has error-handling behavior: malformed/truncated JSON lines are skipped (common when sessions crash mid-write). If the metadata extractor and the full parser handle malformed lines differently, counts will drift.

Rather than reimplementing any of these rules, extract shared helpers at two levels:

```typescript
// In session-parser.ts (or a shared module):

// Level 1: Line iteration with consistent error handling
// Splits content by newlines, JSON.parse each, skips malformed lines identically
// to how parseSessionContent handles them. Returns parse error count for diagnostics.
export function forEachJsonlLine(
  content: string,
  onLine: (parsed: RawLine, lineIndex: number) => void
): { parseErrors: number }

// Level 2: Classification and counting (called per parsed line)
export function countMessagesForLine(parsed: RawLine): number
export function classifyLine(parsed: RawLine): LineClassification
```

Both `extractSessionMetadata()` and `parseSessionContent()` use `forEachJsonlLine()` for iteration, ensuring identical malformed-line handling. Both use `countMessagesForLine()` for counting. This two-level sharing guarantees that list counts can never drift from detail-view counts, regardless of future parser changes or edge cases in error handling.

### Metadata extraction (Tier 3)

A lightweight `extractSessionMetadata()` function reads the JSONL file and extracts only what the list view needs, without building full message content strings:

```typescript
export function extractSessionMetadata(content: string): SessionMetadata
```

Implementation:

1. Iterate lines via `forEachJsonlLine(content, ...)` — the shared iterator with identical malformed-line handling as the main parser
2. Call `countMessagesForLine(parsed)` per line — the shared helper that uses the **same classification rules** as `parseSessionContent` in `session-parser.ts`
3. Extract `firstPrompt`: content of the first user message that isn't a `<system-reminder>`, truncated to 200 characters
4. Extract `summary`: the `summary` field from the last `type="summary"` line
5. Capture first and last `timestamp` fields for duration computation

No string building, no `JSON.stringify`, no markdown processing — just counting, timestamp capture, and first-match extraction. This is exact (matches `parseSessionContent().length` via shared helpers) but 2-3x faster than full parsing.

### Persistent metadata cache

**Location:** `~/.cache/session-viewer/metadata.json`

```typescript
interface CacheFile {
  version: 1;
  entries: Record<string, {  // keyed by absolute file path
    mtimeMs: number;
    size: number;
    messageCount: number;
    firstPrompt: string;
    summary: string;
    created: string;   // ISO string from file birthtime
    modified: string;  // ISO string from file mtime
    firstTimestamp: string;  // ISO from first JSONL line with timestamp
    lastTimestamp: string;   // ISO from last JSONL line with timestamp
  }>;
}
```

Behavior:
- Loaded once on first `discoverSessions()` call
- Entries validated by `(mtimeMs, size)` — if either changes, entry is re-extracted via Tier 3
- Written to disk asynchronously using a dirty-flag write-behind strategy: only when cache has new/updated entries, coalescing multiple discovery passes, non-blocking
- Flush any pending write on process exit (`SIGTERM`, `SIGINT`) and graceful server shutdown — prevents losing cache updates when the server stops before the async write fires
- Corrupt or missing cache file triggers graceful fallback (all files go through Tier 3, cache rebuilt)
- Atomic writes: write to temp file, then rename (prevents corruption from crashes during write)
- Stale entries (file no longer exists on disk) are pruned on save

### Concurrency model

Cold start with 3,103 files requires bounded parallelism to avoid file-handle exhaustion and IO thrash, while still meeting the <5s target:

- **Stat phase**: Batch all `fs.stat()` calls with concurrency limit (e.g., 64). This classifies each file into Tier 1/2 (cache hit) or Tier 3 (needs parse). Files that fail stat (ENOENT from deletion race, EACCES) are silently skipped with a debug log.
- **Parse phase**: Process Tier-3 misses with bounded concurrency (e.g., 8). Each parse reads + iterates via shared `forEachJsonlLine()` + shared counter. With max file size 4.5MB, each parse is ~5-50ms.
- Use a simple async work queue (e.g., `p-limit` or hand-rolled semaphore). No worker threads needed for this IO-bound workload.

### Performance expectations

| Scenario | Estimated time |
|----------|---------------|
| Cold start (no cache, no index) | ~3-5s for 3,103 files (~500MB), bounded concurrency: stat@64, parse@8 |
| Warm start (cache exists, few changes) | ~300-500ms (stat all files at bounded concurrency, in-memory lookups) |
| Incremental (cache + few new sessions) | ~500ms + ~50ms per new file |
| Subsequent API calls within 30s TTL | <1ms (in-memory session list cache) |

### Existing infrastructure leveraged

- **30-second in-memory cache** in `sessions.ts` (`getCachedSessions()`) — unchanged, provides the fast path for repeated API calls
- **`?refresh=1` query parameter** — forces cache invalidation, unchanged
- **Concurrent request deduplication** via `cachePromise` pattern — unchanged
- **Security validations** — path traversal rejection, containment checks, `.jsonl` extension enforcement — applied to filesystem-discovered files identically

## Implementation scope

### Checkpoints

#### CP0 — Parser parity foundations
- Extract `forEachJsonlLine()` shared line iterator from existing parser
- Extract `countMessagesForLine()` and `classifyLine()` shared helpers
- Refactor `extractMessages()` to use these internally (no behavior change to parseSessionContent)
- Tests verify identical behavior on malformed/truncated lines

#### CP1 — Filesystem-first correctness
- All `.jsonl` sessions appear even with missing/corrupt index
- `extractSessionMetadata()` uses shared line iterator + counting helpers; exact counts verified by tests
- Stat-derived `created`/`modified` are the single source for SessionEntry timestamps and list ordering
- Duration computed from JSONL timestamps, not index
- TOCTOU races (readdir/stat, stat/read) handled gracefully — disappeared files silently skipped

#### CP2 — Persistent cache
- Atomic writes with dirty-flag write-behind; prune stale entries
- Invalidation keyed on `(mtimeMs, size)`
- Flush pending writes on process exit / server shutdown

#### CP3 — Index fast path (Tier 1)
- Parse index into Map; normalize `modified` ISO → ms; validate against stat mtime with 1s tolerance
- sessionId mapping: `basename(file, '.jsonl')`
- Tier 1 trusts content fields only; timestamps always from stat

#### CP4 — Performance hardening
- Bounded concurrency for stat + parse phases
- Warm start <1s verified on real dataset

### Modified files

**`src/server/services/session-parser.ts`**

1. Extract `forEachJsonlLine(content, onLine): { parseErrors: number }` — shared line iterator with consistent malformed-line handling
2. Extract `countMessagesForLine(parsed: RawLine): number` — shared counting helper
3. Extract `classifyLine(parsed: RawLine): LineClassification` — shared classification
4. Refactor `extractMessages()` to use these shared helpers internally (no behavior change to parseSessionContent)

**`src/server/services/session-discovery.ts`**

1. Add `extractSessionMetadata(content: string): SessionMetadata` — lightweight JSONL metadata extractor using shared line iterator + counting helper
2. Add `MetadataCache` class — persistent cache with load/get/set/save, dirty-flag write-behind, shutdown flush
3. Rewrite per-project discovery loop — filesystem-first, tiered metadata lookup with bounded concurrency
4. Read `sessions-index.json` as optimization only — parse into `Map<sessionId, IndexEntry>`, normalize `modified` to ms, validate against stat mtime before trusting
5. Register shutdown hooks for cache flush on `SIGTERM`/`SIGINT`

### Unchanged files

- `src/server/routes/sessions.ts` — existing caching layer works as-is
- `src/shared/types.ts` — `SessionEntry` type already has `duration?: number`
- All client components — no changes needed

### New tests

- Unit test: `forEachJsonlLine()` skips malformed lines identically to how `parseSessionContent` handles them
- Unit test: `forEachJsonlLine()` reports parse error count for truncated/corrupted lines
- Unit test: `countMessagesForLine()` matches actual `extractMessages()` output length on sample lines
- Unit test: `extractSessionMetadata()` output matches `parseSessionContent().length` on sample fixtures (including malformed/truncated lines)
- Unit test: Duration extracted from JSONL timestamps matches expected values
- Unit test: SessionEntry `created`/`modified` always come from stat, even when Tier 1 index data is trusted
- Unit test: Tier 1 validation rejects stale index entries (mtime mismatch beyond 1s tolerance)
- Unit test: Tier 1 handles missing `modified` field gracefully (falls through to Tier 2)
- Unit test: Discovery works with no `sessions-index.json` present
- Unit test: Discovery silently skips files that disappear between readdir and stat (TOCTOU)
- Unit test: Cache hit/miss/invalidation behavior (mtimeMs + size)
- Unit test: Cache dirty-flag only triggers write when entries changed

## Edge cases

| Scenario | Behavior |
|----------|----------|
| File actively being written | mtime changes between stat and read. Next discovery pass re-extracts. Partial JSONL handled gracefully (malformed lines skipped via shared `forEachJsonlLine`, same behavior as real parser). |
| Deleted session files | File in cache but gone from disk. Entry silently dropped, pruned from cache on next save. |
| File disappears between readdir and stat | TOCTOU race. Stat failure (ENOENT) silently skipped with debug log. |
| File disappears between stat and read | Read failure silently skipped; file excluded from results. Next pass re-discovers if it reappears. |
| Index entry with wrong mtime | Tier 1 validation rejects it (>1s tolerance). Falls through to Tier 2/3. |
| Index entry with no `modified` field | Tier 1 skips it. Falls through to Tier 2/3. |
| Index `modified` in seconds vs milliseconds | Normalization handles both ISO strings and numeric timestamps. |
| Cache file locked or unwritable | Extraction still works, just doesn't persist. Warning logged to stderr. |
| Very large files | 4.5MB max observed. Tier 3 parse ~50ms. Acceptable. |
| Concurrent server restarts | Cache writes are atomic (temp file + rename). |
| Server killed before async cache write | Shutdown hooks flush pending writes on SIGTERM/SIGINT. Hard kills (SIGKILL) may lose updates — acceptable, cache rebuilt on next cold start. |
| Empty JSONL files | Returns `messageCount: 0`, empty `firstPrompt`, `summary`, and timestamps. Duration: 0. |
| Projects with no index file | Discovery proceeds normally via Tier 2/3. Common case (13 projects). |
| Non-JSONL files in project dirs | Filtered out by `.jsonl` extension check in `readdir` results. |
| File handle exhaustion | Bounded concurrency (stat@64, parse@8) prevents opening thousands of handles. |
| Future parser changes (new message types) | Shared line iterator + counting helper in session-parser.ts means Tier 3 automatically stays in sync. |
| Malformed JSONL lines (crash mid-write) | Shared `forEachJsonlLine()` skips identically in both metadata extraction and full parsing — no count drift. |

## Verification plan

1. Start dev server, confirm today's sessions appear immediately in the session list
2. Compare message counts for indexed sessions: Tier 1 data vs Tier 3 extraction (should match)
3. Verify duration is shown for sessions that have no index entry (JSONL-only sessions)
4. Delete a `sessions-index.json`, refresh — verify all sessions for that project still appear with correct counts and durations
5. Run existing test suite: `npm test`
6. Run new unit tests for shared line iterator, counting helper, `extractSessionMetadata()`, and `MetadataCache`
7. Verify `created`/`modified` in session list come from stat, not index (compare with `ls -l` output)
8. Verify cold start performance: delete `~/.cache/session-viewer/metadata.json`, time the first API request
9. Verify warm start performance: time a subsequent server start with cache in place
10. Verify cache dirty-flag: repeated refreshes with no file changes should not write cache to disk
11. Kill server with SIGTERM, restart — verify cache was flushed (no full re-parse on restart)