session-viewer/docs/prd-jsonl-first-discovery.md

PRD: JSONL-First Session Discovery

Status: Ready for Implementation

Context

Session viewer relies exclusively on sessions-index.json files that Claude Code maintains. These indexes are unreliable — a known, widespread bug with multiple open GitHub issues (#22030, #21610, #18619, #22114).

Root cause

Claude Code updates sessions-index.json only at session end. If a session crashes, is killed, or is abandoned, the JSONL file is written but the index is never updated. Multiple concurrent Claude instances can also corrupt the index (last-write-wins on a single JSON file). There is no reindex command and no background repair process.

Impact on this system

• 542 unindexed JSONL files across all projects (87 MB total)
• 48 unindexed in last 7 days (30.8 MB)
• 13 projects have JSONL session files but no index at all
• Zero sessions from today (Feb 4, 2026) appear in any index
• 3,103 total JSONL files vs 2,563 indexed entries = 17% loss rate

Key insight

The .jsonl files are the source of truth. The index is an unreliable convenience cache. The session viewer must treat it that way.

Requirements

Must have

1. All sessions with a .jsonl file must appear in the session list, regardless of whether they're in sessions-index.json
2. Exact message counts — no estimates, no approximations. Contract: Tier 3 extraction MUST reuse the same line-classification logic as parseSessionContent (shared helper), so list counts cannot drift from detail parsing.
3. Performance: Warm start (cache exists, few changes) must complete under 1 second. Cold start (no cache) is acceptable up to 5 seconds for first request
4. Correctness over speed — never show stale metadata if the file has been modified
5. Zero config — works out of the box with no setup or external dependencies

Should have

6. Session summary extracted from the last type="summary" line in the JSONL
7. Session firstPrompt extracted from the first non-system-reminder user message
8. Session duration MUST be derivable without relying on sessions-index.json — extract first and last timestamps from JSONL when index is missing or stale
9. Persistent metadata cache survives server restarts

Won't have (this iteration)

• Real-time push updates (sessions appearing in UI without refresh)
• Background file watcher daemon
• Integration with cass as a search/indexing backend
• Rebuilding Claude Code's sessions-index.json

Technical Design

Architecture: Filesystem-primary with tiered metadata lookup

discoverSessions()
  |
  +-- For each project directory under ~/.claude/projects/:
  |     |
  |     +-- fs.readdir() --> list all *.jsonl files
  |     +-- Read sessions-index.json (optional, used as pre-populated cache)
  |     |
  |     +-- Batch stat all .jsonl files (bounded concurrency)
  |     |     Files that disappeared between readdir and stat are silently skipped (TOCTOU race)
  |     |
  |     +-- For each .jsonl file:
  |     |     |
  |     |     +-- Tier 1: Check index
  |     |     |     Entry exists AND normalize(index.modified) matches stat mtime?
  |     |     |     --> Use index content data (messageCount, summary, firstPrompt)
  |     |     |     --> Use stat-derived timestamps for created/modified (always)
  |     |     |
  |     |     +-- Tier 2: Check persistent metadata cache
  |     |     |     path + mtimeMs + size match?
  |     |     |     --> Use cached metadata (fast path)
  |     |     |
  |     |     +-- Tier 3: Extract metadata from JSONL content
  |     |           Read file, lightweight parse using shared line iterator + counting helper
  |     |           --> Cache result for future lookups
  |     |
  |     +-- Collect SessionEntry[] for this project
  |
  +-- Merge all projects
  +-- Sort by modified (descending) — always stat-derived, never index-derived
  +-- Async: persist metadata cache to disk (if dirty)

Tier explanation

Tier	Source	Speed	When used	Trusts from source
1	sessions-index.json	Instant (in-memory lookup)	Index exists, entry present, normalize(modified) matches actual file mtime	messageCount, summary, firstPrompt only. Timestamps always from stat.
2	Persistent metadata cache	Instant (in-memory lookup)	Index missing/stale, but file hasn't changed since last extraction (mtimeMs + size match)	All cached fields
3	JSONL file parse	~5-50ms/file	New or modified file, not in any cache	Extracted fresh

Tier 1 reuses Claude's index when it's valid — no wasted work. The index modified field (ISO string) is normalized to milliseconds and compared against the real file stat.mtimeMs. If the index is missing or corrupt, discovery continues with Tier 2 and 3 without error. Even when Tier 1 is valid, created and modified timestamps on the SessionEntry always come from fs.stat — the index is a content cache only.

Tier 1: Index validation details

The actual sessions-index.json format has created and modified as ISO strings, not a fileMtime field. Tier 1 validation must:

1. Map JSONL filename to sessionId: sessionId := path.basename(jsonlFile, '.jsonl')
2. Look up sessionId in the index Map<string, IndexEntry>
3. Compare new Date(entry.modified).getTime() against stat.mtimeMs — reject if they differ by more than 1000ms (accounts for ISO string → filesystem mtime rounding)
4. If the index entry has no modified field, skip Tier 1 (fall through to Tier 2)
5. When Tier 1 is valid, trust only content fields (messageCount, summary, firstPrompt). The created/modified on the resulting SessionEntry must come from stat.birthtimeMs/stat.mtimeMs respectively — this ensures list ordering is never stale even within the 1s mtime tolerance window.

Shared line-iteration and counting (parser parity contract)

The biggest correctness risk in this design is duplicating any JSONL processing logic. The real parser in session-parser.ts has non-trivial expansion rules:

• User array content: expands tool_result and text blocks into separate messages
• system-reminder detection reclassifies user text blocks as system_message
• Assistant array content: thinking, text, and tool_use each become separate messages
• progress, file-history-snapshot, summary → 1 message each
• system, queue-operation → 0 (skipped)

It also has error-handling behavior: malformed/truncated JSON lines are skipped (common when sessions crash mid-write). If the metadata extractor and the full parser handle malformed lines differently, counts will drift.

Rather than reimplementing any of these rules, extract shared helpers at two levels:

// In session-parser.ts (or a shared module):

// Level 1: Line iteration with consistent error handling
// Splits content by newlines, JSON.parse each, skips malformed lines identically
// to how parseSessionContent handles them. Returns parse error count for diagnostics.
export function forEachJsonlLine(
  content: string,
  onLine: (parsed: RawLine, lineIndex: number) => void
): { parseErrors: number }

// Level 2: Classification and counting (called per parsed line)
export function countMessagesForLine(parsed: RawLine): number
export function classifyLine(parsed: RawLine): LineClassification

Both extractSessionMetadata() and parseSessionContent() use forEachJsonlLine() for iteration, ensuring identical malformed-line handling. Both use countMessagesForLine() for counting. This two-level sharing guarantees that list counts can never drift from detail-view counts, regardless of future parser changes or edge cases in error handling.

Metadata extraction (Tier 3)

A lightweight extractSessionMetadata() function reads the JSONL file and extracts only what the list view needs, without building full message content strings:

export function extractSessionMetadata(content: string): SessionMetadata

Implementation:

1. Iterate lines via forEachJsonlLine(content, ...) — the shared iterator with identical malformed-line handling as the main parser
2. Call countMessagesForLine(parsed) per line — the shared helper that uses the same classification rules as parseSessionContent in session-parser.ts
3. Extract firstPrompt: content of the first user message that isn't a <system-reminder>, truncated to 200 characters
4. Extract summary: the summary field from the last type="summary" line
5. Capture first and last timestamp fields for duration computation

No string building, no JSON.stringify, no markdown processing — just counting, timestamp capture, and first-match extraction. This is exact (matches parseSessionContent().length via shared helpers) but 2-3x faster than full parsing.

Persistent metadata cache

Location: ~/.cache/session-viewer/metadata.json

interface CacheFile {
  version: 1;
  entries: Record<string, {  // keyed by absolute file path
    mtimeMs: number;
    size: number;
    messageCount: number;
    firstPrompt: string;
    summary: string;
    created: string;   // ISO string from file birthtime
    modified: string;  // ISO string from file mtime
    firstTimestamp: string;  // ISO from first JSONL line with timestamp
    lastTimestamp: string;   // ISO from last JSONL line with timestamp
  }>;
}

Behavior:

• Loaded once on first discoverSessions() call
• Entries validated by (mtimeMs, size) — if either changes, entry is re-extracted via Tier 3
• Written to disk asynchronously using a dirty-flag write-behind strategy: only when cache has new/updated entries, coalescing multiple discovery passes, non-blocking
• Flush any pending write on process exit (SIGTERM, SIGINT) and graceful server shutdown — prevents losing cache updates when the server stops before the async write fires
• Corrupt or missing cache file triggers graceful fallback (all files go through Tier 3, cache rebuilt)
• Atomic writes: write to temp file, then rename (prevents corruption from crashes during write)
• Stale entries (file no longer exists on disk) are pruned on save

Concurrency model

Cold start with 3,103 files requires bounded parallelism to avoid file-handle exhaustion and IO thrash, while still meeting the <5s target:

• Stat phase: Batch all fs.stat() calls with concurrency limit (e.g., 64). This classifies each file into Tier 1/2 (cache hit) or Tier 3 (needs parse). Files that fail stat (ENOENT from deletion race, EACCES) are silently skipped with a debug log.
• Parse phase: Process Tier-3 misses with bounded concurrency (e.g., 8). Each parse reads + iterates via shared forEachJsonlLine() + shared counter. With max file size 4.5MB, each parse is ~5-50ms.
• Use a simple async work queue (e.g., p-limit or hand-rolled semaphore). No worker threads needed for this IO-bound workload.

Performance expectations

Scenario	Estimated time
Cold start (no cache, no index)	~3-5s for 3,103 files (~500MB), bounded concurrency: stat@64, parse@8
Warm start (cache exists, few changes)	~300-500ms (stat all files at bounded concurrency, in-memory lookups)
Incremental (cache + few new sessions)	~500ms + ~50ms per new file
Subsequent API calls within 30s TTL	<1ms (in-memory session list cache)

Existing infrastructure leveraged

• 30-second in-memory cache in sessions.ts (getCachedSessions()) — unchanged, provides the fast path for repeated API calls
• ?refresh=1 query parameter — forces cache invalidation, unchanged
• Concurrent request deduplication via cachePromise pattern — unchanged
• Security validations — path traversal rejection, containment checks, .jsonl extension enforcement — applied to filesystem-discovered files identically

Implementation scope

Checkpoints

CP0 — Parser parity foundations

• Extract forEachJsonlLine() shared line iterator from existing parser
• Extract countMessagesForLine() and classifyLine() shared helpers
• Refactor extractMessages() to use these internally (no behavior change to parseSessionContent)
• Tests verify identical behavior on malformed/truncated lines

CP1 — Filesystem-first correctness

• All .jsonl sessions appear even with missing/corrupt index
• extractSessionMetadata() uses shared line iterator + counting helpers; exact counts verified by tests
• Stat-derived created/modified are the single source for SessionEntry timestamps and list ordering
• Duration computed from JSONL timestamps, not index
• TOCTOU races (readdir/stat, stat/read) handled gracefully — disappeared files silently skipped

CP2 — Persistent cache

• Atomic writes with dirty-flag write-behind; prune stale entries
• Invalidation keyed on (mtimeMs, size)
• Flush pending writes on process exit / server shutdown

CP3 — Index fast path (Tier 1)

• Parse index into Map; normalize modified ISO → ms; validate against stat mtime with 1s tolerance
• sessionId mapping: basename(file, '.jsonl')
• Tier 1 trusts content fields only; timestamps always from stat

CP4 — Performance hardening

• Bounded concurrency for stat + parse phases
• Warm start <1s verified on real dataset

Modified files

src/server/services/session-parser.ts

1. Extract forEachJsonlLine(content, onLine): { parseErrors: number } — shared line iterator with consistent malformed-line handling
2. Extract countMessagesForLine(parsed: RawLine): number — shared counting helper
3. Extract classifyLine(parsed: RawLine): LineClassification — shared classification
4. Refactor extractMessages() to use these shared helpers internally (no behavior change to parseSessionContent)

src/server/services/session-discovery.ts

1. Add extractSessionMetadata(content: string): SessionMetadata — lightweight JSONL metadata extractor using shared line iterator + counting helper
2. Add MetadataCache class — persistent cache with load/get/set/save, dirty-flag write-behind, shutdown flush
3. Rewrite per-project discovery loop — filesystem-first, tiered metadata lookup with bounded concurrency
4. Read sessions-index.json as optimization only — parse into Map<sessionId, IndexEntry>, normalize modified to ms, validate against stat mtime before trusting
5. Register shutdown hooks for cache flush on SIGTERM/SIGINT

Unchanged files

• src/server/routes/sessions.ts — existing caching layer works as-is
• src/shared/types.ts — SessionEntry type already has duration?: number
• All client components — no changes needed

New tests

• Unit test: forEachJsonlLine() skips malformed lines identically to how parseSessionContent handles them
• Unit test: forEachJsonlLine() reports parse error count for truncated/corrupted lines
• Unit test: countMessagesForLine() matches actual extractMessages() output length on sample lines
• Unit test: extractSessionMetadata() output matches parseSessionContent().length on sample fixtures (including malformed/truncated lines)
• Unit test: Duration extracted from JSONL timestamps matches expected values
• Unit test: SessionEntry created/modified always come from stat, even when Tier 1 index data is trusted
• Unit test: Tier 1 validation rejects stale index entries (mtime mismatch beyond 1s tolerance)
• Unit test: Tier 1 handles missing modified field gracefully (falls through to Tier 2)
• Unit test: Discovery works with no sessions-index.json present
• Unit test: Discovery silently skips files that disappear between readdir and stat (TOCTOU)
• Unit test: Cache hit/miss/invalidation behavior (mtimeMs + size)
• Unit test: Cache dirty-flag only triggers write when entries changed

Edge cases

Scenario	Behavior
File actively being written	mtime changes between stat and read. Next discovery pass re-extracts. Partial JSONL handled gracefully (malformed lines skipped via shared forEachJsonlLine, same behavior as real parser).
Deleted session files	File in cache but gone from disk. Entry silently dropped, pruned from cache on next save.
File disappears between readdir and stat	TOCTOU race. Stat failure (ENOENT) silently skipped with debug log.
File disappears between stat and read	Read failure silently skipped; file excluded from results. Next pass re-discovers if it reappears.
Index entry with wrong mtime	Tier 1 validation rejects it (>1s tolerance). Falls through to Tier 2/3.
Index entry with no modified field	Tier 1 skips it. Falls through to Tier 2/3.
Index modified in seconds vs milliseconds	Normalization handles both ISO strings and numeric timestamps.
Cache file locked or unwritable	Extraction still works, just doesn't persist. Warning logged to stderr.
Very large files	4.5MB max observed. Tier 3 parse ~50ms. Acceptable.
Concurrent server restarts	Cache writes are atomic (temp file + rename).
Server killed before async cache write	Shutdown hooks flush pending writes on SIGTERM/SIGINT. Hard kills (SIGKILL) may lose updates — acceptable, cache rebuilt on next cold start.
Empty JSONL files	Returns messageCount: 0, empty firstPrompt, summary, and timestamps. Duration: 0.
Projects with no index file	Discovery proceeds normally via Tier 2/3. Common case (13 projects).
Non-JSONL files in project dirs	Filtered out by .jsonl extension check in readdir results.
File handle exhaustion	Bounded concurrency (stat@64, parse@8) prevents opening thousands of handles.
Future parser changes (new message types)	Shared line iterator + counting helper in session-parser.ts means Tier 3 automatically stays in sync.
Malformed JSONL lines (crash mid-write)	Shared forEachJsonlLine() skips identically in both metadata extraction and full parsing — no count drift.

Verification plan

1. Start dev server, confirm today's sessions appear immediately in the session list
2. Compare message counts for indexed sessions: Tier 1 data vs Tier 3 extraction (should match)
3. Verify duration is shown for sessions that have no index entry (JSONL-only sessions)
4. Delete a sessions-index.json, refresh — verify all sessions for that project still appear with correct counts and durations
5. Run existing test suite: npm test
6. Run new unit tests for shared line iterator, counting helper, extractSessionMetadata(), and MetadataCache
7. Verify created/modified in session list come from stat, not index (compare with ls -l output)
8. Verify cold start performance: delete ~/.cache/session-viewer/metadata.json, time the first API request
9. Verify warm start performance: time a subsequent server start with cache in place
10. Verify cache dirty-flag: repeated refreshes with no file changes should not write cache to disk
11. Kill server with SIGTERM, restart — verify cache was flushed (no full re-parse on restart)