# Deep Performance Audit Report

**Date:** 2026-02-12
**Branch:** `perf-audit` (e9bacc94)
**Parent:** `039ab1c2` (master, v0.6.1)

---

## Methodology

1. **Baseline** — measured p50/p95 latency for all major commands with warm cache
2. **Profile** — used macOS `sample` profiler and `EXPLAIN QUERY PLAN` to identify hotspots
3. **Golden output** — captured exact numeric outputs before changes as equivalence oracle
4. **One lever per change** — each optimization isolated and independently benchmarked
5. **Revert threshold** — any optimization <1.1x speedup reverted per audit rules

---

## Baseline Measurements (warm cache, release build)

| Command | Latency | Notes |
|---------|---------|-------|
| `who --path src/core/db.rs` (expert) | 2200ms | **Hotspot** |
| `who --active` | 83-93ms | Acceptable |
| `who workload` | 22ms | Fast |
| `stats` | 107-112ms | **Hotspot** |
| `search "authentication"` | 1030ms | **Hotspot** (library-level) |
| `list issues -n 50` | ~40ms | Fast |

---

## Optimization 1: INDEXED BY for DiffNote Queries

**Target:** `src/cli/commands/who.rs` — expert and reviews query paths

**Problem:** SQLite query planner chose `idx_notes_system` (38% selectivity, 106K rows) over `idx_notes_diffnote_path_created` (9.3% selectivity, 26K rows) for path-filtered DiffNote queries. The partial index `WHERE noteable_type = 'MergeRequest' AND type = 'DiffNote'` is far more selective but the planner's cost model didn't pick it.

**Change:** Added `INDEXED BY idx_notes_diffnote_path_created` to all 8 SQL queries across `query_expert`, `query_expert_details`, `query_reviews`, `build_path_query` (probes 1 & 2), and `suffix_probe`.

**Results:**

| Query | Before | After | Speedup |
|-------|--------|-------|---------|
| expert (specific path) | 2200ms | 56-58ms | **38x** |
| expert (broad path) | 2200ms | 83ms | **26x** |
| reviews | 1800ms | 24ms | **75x** |

**Isomorphism proof:** `INDEXED BY` only changes which index the planner uses, not the query semantics. Same rows matched, same ordering, same output. Verified by golden output comparison across 5+ runs.

---

## Optimization 2: Conditional Aggregates in Stats

**Target:** `src/cli/commands/stats.rs`

**Problem:** 12+ sequential `COUNT(*)` queries each requiring a full table scan of `documents` (61K rows). Each scan touched the same pages but couldn't share work.

**Changes:**
- Documents: 5 sequential COUNTs -> 1 query with `SUM(CASE WHEN ... THEN 1 END)`
- FTS count: `SELECT COUNT(*) FROM documents_fts` (virtual table, slow) -> `SELECT COUNT(*) FROM documents_fts_docsize` (shadow B-tree table, 19x faster)
- Embeddings: 2 queries -> 1 with `COUNT(DISTINCT document_id), COUNT(*)`
- Dirty sources: 2 queries -> 1 with conditional aggregates
- Pending fetches: 2 queries -> 1 each (discussions, dependents)

**Results:**

| Metric | Before | After | Speedup |
|--------|--------|-------|---------|
| Warm median | 112ms | 66ms | **1.70x** |
| Cold | 1220ms | ~700ms | ~1.7x |

**Golden output verified:**

```
total:61652, issues:8241, mrs:10018, discussions:43393, truncated:63
fts:61652, embedded:61652, chunks:88161
```

All values match exactly across before/after runs.

**Isomorphism proof:** `SUM(CASE WHEN x THEN 1 END)` is algebraically identical to `COUNT(*) WHERE x`. The FTS5 shadow table `documents_fts_docsize` has exactly one row per FTS document by SQLite specification, so `COUNT(*)` on it equals the virtual table count.

---

## Investigation: Two-Phase FTS Search (REVERTED)

**Target:** `src/search/fts.rs`, `src/cli/commands/search.rs`

**Hypothesis:** FTS5 `snippet()` generation is expensive. Splitting search into Phase 1 (score-only MATCH+bm25) and Phase 2 (snippet for filtered results only) should reduce work.

**Implementation:** Created `fetch_fts_snippets()` that retrieves snippets only for post-filter document IDs via `json_each()` join.

**Results:**

| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| search (limit 20) | 1030ms | 995ms | 3.5% |

**Decision:** Reverted. Per audit rules, <1.1x speedup does not justify added code complexity.

**Root cause:** The bottleneck is not snippet generation but `MATCH` + `bm25()` scoring itself. Profiling showed `strspn` (FTS5 tokenizer) and `memmove` as the top CPU consumers. The same query runs in 30ms on system sqlite3 but 1030ms in rusqlite's bundled SQLite — a ~125x gap despite both being SQLite 3.51.x compiled at -O3.

---

## Library-Level Finding: Bundled SQLite FTS5 Performance

**Observation:** FTS5 MATCH+bm25 queries are ~125x slower in rusqlite's bundled SQLite vs system sqlite3.

| Environment | Query Time | Notes |
|-------------|-----------|-------|
| System sqlite3 (macOS) | 30ms (with snippet), 8ms (without) | Same .db file |
| rusqlite bundled | 1030ms | `features = ["bundled"]`, OPT_LEVEL=3 |

**Profiler data (macOS `sample`):**
- Top hotspot: `strspn` in FTS5 tokenizer
- Secondary: `memmove` in FTS5 internals
- Scaling: ~5ms per result (limit 5 = 497ms, limit 20 = 995ms)

**Possible causes:**
- Bundled SQLite compiled without platform-specific optimizations (SIMD, etc.)
- Different memory allocator behavior
- Missing compile-time tuning flags

**Recommendation for future:** Investigate switching from `features = ["bundled"]` to system SQLite linkage, or audit the bundled compile flags in the `libsqlite3-sys` build script.

---

## Exploration Agent Findings (Informational)

Four parallel exploration agents surveyed the entire codebase. Key findings beyond what was already addressed:

### Ingestion Pipeline
- Serial DB writes in async context (acceptable — rusqlite is synchronous)
- Label ingestion uses individual inserts (potential batch optimization, low priority)

### CLI / GitLab Client
- GraphQL client recreated per call (`client.rs:98-100`) — caches connection pool, minor
- Double JSON deserialization in GraphQL responses — medium priority
- N+1 subqueries in `list` command (`list.rs:408-423`) — 4 correlated subqueries per row

### Search / Embedding
- No N+1 patterns, no O(n^2) algorithms
- Chunking is O(n) single-pass with proper UTF-8 safety
- Ollama concurrency model is sound (parallel HTTP, serial DB writes)

### Database / Documents
- O(n^2) prefix sum in `truncation.rs` — low traffic path
- String allocation patterns in extractors — micro-optimization territory

---

## Opportunity Matrix

| Candidate | Impact | Confidence | Effort | Score | Status |
|-----------|--------|------------|--------|-------|--------|
| INDEXED BY for DiffNote | Very High | High | Low | **9.0** | Shipped |
| Stats conditional aggregates | Medium | High | Low | **7.0** | Shipped |
| Bundled SQLite FTS5 | Very High | Medium | High | 5.0 | Documented |
| List N+1 subqueries | Medium | Medium | Medium | 4.0 | Backlog |
| GraphQL double deser | Low | Medium | Low | 3.5 | Backlog |
| Truncation O(n^2) | Low | High | Low | 3.0 | Backlog |

---

## Files Modified

| File | Change |
|------|--------|
| `src/cli/commands/who.rs` | INDEXED BY hints on 8 SQL queries |
| `src/cli/commands/stats.rs` | Conditional aggregates, FTS5 shadow table, merged queries |

---

## Quality Gates

- All 603 tests pass
- `cargo clippy --all-targets -- -D warnings` clean
- `cargo fmt --check` clean
- Golden output verified for both optimizations