theirongolddev/gitlore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1f36fe6a21a433623c7ac357149a2b56d97302c0
gitlore/SPEC.md
teernisse 1f36fe6a21 more planning
2026-01-21 15:56:11 -05:00

1609 lines
66 KiB
Markdown
Raw Blame History

# GitLab Knowledge Engine - Spec Document
## Executive Summary
A self-hosted tool to extract, index, and semantically search 2+ years of GitLab data (issues, MRs, and discussion threads) from 2 main repositories (~50-100K documents including threaded discussions). The MVP delivers semantic search as a foundational capability that enables future specialized views (file history, personal tracking, person context). Discussion threads are preserved as first-class entities to maintain conversational context essential for decision traceability.
---
## Quick Start
### Prerequisites
| Requirement | Version | Notes |
|-------------|---------|-------|
| Node.js | 20+ | LTS recommended |
| npm | 10+ | Comes with Node.js |
| Ollama | Latest | Optional for semantic search; lexical search works without it |
### Installation
```bash
# Clone and install
git clone https://github.com/your-org/gitlab-inbox.git
cd gitlab-inbox
npm install
npm run build
npm link # Makes `gi` available globally
```
### First Run
1. **Set your GitLab token** (create at GitLab > Settings > Access Tokens with `read_api` scope):
```bash
export GITLAB_TOKEN="glpat-xxxxxxxxxxxxxxxxxxxx"
```
2. **Run the setup wizard:**
```bash
gi init
```
This creates `gi.config.json` with your GitLab URL and project paths.
3. **Verify your environment:**
```bash
gi doctor
```
All checks should pass (Ollama warning is OK if you only need lexical search).
4. **Sync your data:**
```bash
gi sync
```
Initial sync takes 10-20 minutes depending on repo size and rate limits.
5. **Search:**
```bash
gi search "authentication redesign"
```
### Troubleshooting First Run
| Symptom | Solution |
|---------|----------|
| `Config file not found` | Run `gi init` first |
| `GITLAB_TOKEN not set` | Export the environment variable |
| `401 Unauthorized` | Check token has `read_api` scope |
| `Project not found: group/project` | Verify project path in GitLab URL |
| `Ollama connection refused` | Start Ollama or use `--mode=lexical` for search |
---
## Discovery Summary
### Pain Points Identified
1. **Knowledge discovery** - Tribal knowledge buried in old MRs/issues that nobody can find
2. **Decision traceability** - Hard to find *why* decisions were made; context scattered across issue comments and MR discussions
### Constraints
| Constraint | Detail |
|------------|--------|
| Hosting | Self-hosted only, no external APIs |
| Compute | Local dev machine (M-series Mac assumed) |
| GitLab Access | Self-hosted instance, PAT access, no webhooks (could request) |
| Build Method | AI agents will implement; user is TypeScript expert for review |
### Target Use Cases (Priority Order)
1. **MVP: Semantic Search** - "Find discussions about authentication redesign"
2. **Future: File/Feature History** - "What decisions were made about src/auth/login.ts?"
3. **Future: Personal Tracking** - "What am I assigned to or mentioned in?"
4. **Future: Person Context** - "What's @johndoe's background in this project?"
---
## Architecture Overview
```
┌─────────────────────────────────────────────────────────────────┐
│ GitLab API │
│ (Issues, MRs, Notes) │
└─────────────────────────────────────────────────────────────────┘
(Commit-level indexing explicitly post-MVP)
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Data Ingestion Layer │
│ - Incremental sync (PAT-based polling) │
│ - Rate limiting / backoff │
│ - Raw JSON storage for replay │
│ - Dependent resource fetching (notes, MR changes) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Data Processing Layer │
│ - Normalize artifacts to unified schema │
│ - Extract searchable documents (canonical text + metadata) │
│ - Content hashing for change detection │
│ - Build relationship graph (issue↔MR↔note↔file) │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Storage Layer │
│ - SQLite + sqlite-vss + FTS5 (hybrid search) │
│ - Structured metadata in relational tables │
│ - Vector embeddings for semantic search │
│ - Full-text index for lexical search fallback │
└─────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────┐
│ Query Interface │
│ - CLI for human testing │
│ - JSON API for AI agent testing │
│ - Semantic search with filters (author, date, type, label) │
└─────────────────────────────────────────────────────────────────┘
```
### Technology Choices
| Component | Recommendation | Rationale |
|-----------|---------------|-----------|
| Language | TypeScript/Node.js | User expertise, good GitLab libs, AI agent friendly |
| Database | SQLite + sqlite-vss | Zero-config, portable, vector search built-in |
| Embeddings | Ollama + nomic-embed-text | Self-hosted, runs well on Apple Silicon, 768-dim vectors |
| CLI Framework | Commander.js or oclif | Standard, well-documented |
### Alternative Considered: Postgres + pgvector
- Pros: More scalable, better for production multi-user
- Cons: Requires running Postgres, heavier setup
- Decision: Start with SQLite for simplicity; migration path exists if needed
---
## GitLab API Strategy
### Primary Resources (Bulk Fetch)
Issues and MRs support efficient bulk fetching with incremental sync:
```
GET /projects/:id/issues?updated_after=X&order_by=updated_at&sort=asc&per_page=100
GET /projects/:id/merge_requests?updated_after=X&order_by=updated_at&sort=asc&per_page=100
```
### Dependent Resources (Per-Parent Fetch)
Discussions must be fetched per-issue and per-MR. There is no bulk endpoint:
```
GET /projects/:id/issues/:iid/discussions?per_page=100&page=N
GET /projects/:id/merge_requests/:iid/discussions?per_page=100&page=N
```
**Pagination:** Discussions endpoints return paginated results. Fetch all pages per parent to avoid silent data loss.
### Sync Pattern
**Initial sync:**
1. Fetch all issues (paginated, ~60 calls for 6K issues at 100/page)
2. For EACH issue → fetch all discussions (~3K calls)
3. Fetch all MRs (paginated, ~60 calls)
4. For EACH MR → fetch all discussions (~3K calls)
5. Total: ~6,100+ API calls for initial sync
**Incremental sync:**
1. Fetch issues where `updated_after=cursor` (bulk)
2. For EACH updated issue → refetch ALL its discussions
3. Fetch MRs where `updated_after=cursor` (bulk)
4. For EACH updated MR → refetch ALL its discussions
### Critical Assumption (Softened)
We *expect* adding a note/discussion updates the parent's `updated_at`, but we do not rely on it exclusively.
**Mitigations (MVP):**
1. **Tuple cursor semantics:** Cursor is a stable tuple `(updated_at, gitlab_id)`. Ties are handled explicitly - process all items with equal `updated_at` before advancing cursor.
2. **Rolling backfill window:** Each sync also re-fetches items updated within the last N days (default 14, configurable). This ensures "late" updates are eventually captured even if parent timestamps behave unexpectedly.
3. **Periodic full re-sync:** Remains optional as an extra safety net (`gi sync --full`).
The backfill window provides 80% of the safety of full resync at <5% of the API cost.
### Rate Limiting
- Default: 10 requests/second with exponential backoff
- Respect `Retry-After` headers on 429 responses
- Add jitter to avoid thundering herd on retry
- **Separate concurrency limits:**
- `sync.primaryConcurrency`: concurrent requests for issues/MRs list endpoints (default 4)
- `sync.dependentConcurrency`: concurrent requests for discussions endpoints (default 2, lower to avoid 429s)
- Bound concurrency per-project to avoid one repo starving the other
- Initial sync estimate: 10-20 minutes depending on rate limits
---
## Checkpoint Structure
Each checkpoint is a **testable milestone** where a human can validate the system works before proceeding.
### Checkpoint 0: Project Setup
**Deliverable:** Scaffolded project with GitLab API connection verified and project resolution working
**Automated Tests (Vitest):**
```
tests/unit/config.test.ts
✓ loads config from gi.config.json
✓ throws if config file missing
✓ throws if required fields missing (baseUrl, projects)
✓ validates project paths are non-empty strings
tests/unit/db.test.ts
✓ creates database file if not exists
✓ applies migrations in order
✓ sets WAL journal mode
✓ enables foreign keys
tests/integration/gitlab-client.test.ts
✓ authenticates with valid PAT
✓ returns 401 for invalid PAT
✓ fetches project by path
✓ handles rate limiting (429) with retry
tests/integration/app-lock.test.ts
✓ acquires lock successfully
✓ updates heartbeat during operation
✓ detects stale lock and recovers
✓ refuses concurrent acquisition
tests/integration/init.test.ts
✓ creates config file with valid structure
✓ validates GitLab URL format
✓ validates GitLab connection before writing config
✓ validates each project path exists in GitLab
✓ fails if token not set
✓ fails if GitLab auth fails
✓ fails if any project path not found
✓ prompts before overwriting existing config
✓ respects --force to skip confirmation
```
**Manual CLI Smoke Tests:**
| Command | Expected Output | Pass Criteria |
|---------|-----------------|---------------|
| `gi auth-test` | `Authenticated as @username (User Name)` | Shows GitLab username and display name |
| `gi doctor` | Status table with ✓/✗ for each check | All checks pass (or Ollama shows warning if not running) |
| `gi doctor --json` | JSON object with check results | Valid JSON, `success: true` for required checks |
| `GITLAB_TOKEN=invalid gi auth-test` | Error message | Non-zero exit code, clear error about auth failure |
| `gi init` | Interactive prompts | Creates valid gi.config.json |
| `gi init` (config exists) | Confirmation prompt | Warns before overwriting |
| `gi --help` | Command list | Shows all available commands |
| `gi version` | Version number | Shows installed version |
**Data Integrity Checks:**
- [ ] `projects` table contains rows for each configured project path
- [ ] `gitlab_project_id` matches actual GitLab project IDs
- [ ] `raw_payloads` contains project JSON for each synced project
**Scope:**
- Project structure (TypeScript, ESLint, Vitest)
- GitLab API client with PAT authentication
- Environment and project configuration
- Basic CLI scaffold with `auth-test` command
- `doctor` command for environment verification
- Projects table and initial project resolution (no issue/MR ingestion yet)
- DB migrations + WAL + FK enforcement
- Sync tracking with crash-safe single-flight lock (heartbeat-based)
- Rate limit handling with exponential backoff + jitter
- `gi init` command for guided setup:
- Prompts for GitLab base URL
- Prompts for project paths (comma-separated or multiple prompts)
- Prompts for token environment variable name (default: GITLAB_TOKEN)
- **Validates before writing config:**
- Token must be set in environment
- Tests auth with `GET /user` endpoint
- Validates each project path with `GET /projects/:path`
- Only writes config after all validations pass
- Generates `gi.config.json` with sensible defaults
- `gi --help` shows all available commands
- `gi <command> --help` shows command-specific help
- `gi version` shows installed version
- First-run detection: if no config exists, suggest `gi init`
**Configuration (MVP):**
```json
// gi.config.json
{
"gitlab": {
"baseUrl": "https://gitlab.example.com",
"tokenEnvVar": "GITLAB_TOKEN"
},
"projects": [
{ "path": "group/project-one" },
{ "path": "group/project-two" }
],
"sync": {
"backfillDays": 14,
"staleLockMinutes": 10,
"heartbeatIntervalSeconds": 30,
"cursorRewindSeconds": 2,
"primaryConcurrency": 4,
"dependentConcurrency": 2
},
"storage": {
"compressRawPayloads": true
},
"embedding": {
"provider": "ollama",
"model": "nomic-embed-text",
"baseUrl": "http://localhost:11434",
"concurrency": 4
}
}
```
**DB Runtime Defaults (Checkpoint 0):**
- On every connection:
- `PRAGMA journal_mode=WAL;`
- `PRAGMA foreign_keys=ON;`
**Schema (Checkpoint 0):**
```sql
-- Projects table (configured targets)
CREATE TABLE projects (
id INTEGER PRIMARY KEY,
gitlab_project_id INTEGER UNIQUE NOT NULL,
path_with_namespace TEXT NOT NULL,
default_branch TEXT,
web_url TEXT,
created_at INTEGER,
updated_at INTEGER,
raw_payload_id INTEGER REFERENCES raw_payloads(id)
);
CREATE INDEX idx_projects_path ON projects(path_with_namespace);
-- Sync tracking for reliability
CREATE TABLE sync_runs (
id INTEGER PRIMARY KEY,
started_at INTEGER NOT NULL,
heartbeat_at INTEGER NOT NULL,
finished_at INTEGER,
status TEXT NOT NULL, -- 'running' | 'succeeded' | 'failed'
command TEXT NOT NULL, -- 'ingest issues' | 'sync' | etc.
error TEXT
);
-- Crash-safe single-flight lock (DB-enforced)
CREATE TABLE app_locks (
name TEXT PRIMARY KEY, -- 'sync'
owner TEXT NOT NULL, -- random run token (UUIDv4)
acquired_at INTEGER NOT NULL,
heartbeat_at INTEGER NOT NULL
);
-- Sync cursors for primary resources only
-- Notes and MR changes are dependent resources (fetched via parent updates)
CREATE TABLE sync_cursors (
project_id INTEGER NOT NULL REFERENCES projects(id),
resource_type TEXT NOT NULL, -- 'issues' | 'merge_requests'
updated_at_cursor INTEGER, -- last fully processed updated_at (ms epoch)
tie_breaker_id INTEGER, -- last fully processed gitlab_id (for stable ordering)
PRIMARY KEY(project_id, resource_type)
);
-- Raw payload storage (decoupled from entity tables)
CREATE TABLE raw_payloads (
id INTEGER PRIMARY KEY,
source TEXT NOT NULL, -- 'gitlab'
project_id INTEGER REFERENCES projects(id), -- nullable for instance-level resources
resource_type TEXT NOT NULL, -- 'project' | 'issue' | 'mr' | 'note' | 'discussion'
gitlab_id INTEGER NOT NULL,
fetched_at INTEGER NOT NULL,
content_encoding TEXT NOT NULL DEFAULT 'identity', -- 'identity' | 'gzip'
payload BLOB NOT NULL -- raw JSON or gzip-compressed JSON
);
CREATE INDEX idx_raw_payloads_lookup ON raw_payloads(project_id, resource_type, gitlab_id);
CREATE INDEX idx_raw_payloads_history ON raw_payloads(project_id, resource_type, gitlab_id, fetched_at);
```
---
### Checkpoint 1: Issue Ingestion
**Deliverable:** All issues + labels + issue discussions from target repos stored locally with resumable cursor-based sync
**Automated Tests (Vitest):**
```
tests/unit/issue-transformer.test.ts
✓ transforms GitLab issue payload to normalized schema
✓ extracts labels from issue payload
✓ handles missing optional fields gracefully
tests/unit/pagination.test.ts
✓ fetches all pages when multiple exist
✓ respects per_page parameter
✓ stops when empty page returned
tests/unit/discussion-transformer.test.ts
✓ transforms discussion payload to normalized schema
✓ extracts notes array from discussion
✓ sets individual_note flag correctly
✓ flags system notes with is_system=1
✓ preserves note order via position field
tests/integration/issue-ingestion.test.ts
✓ inserts issues into database
✓ creates labels from issue payloads
✓ links issues to labels via junction table
✓ stores raw payload for each issue
✓ updates cursor after successful page commit
✓ resumes from cursor on subsequent runs
tests/integration/issue-discussion-ingestion.test.ts
✓ fetches discussions for each issue
✓ creates discussion rows with correct issue FK
✓ creates note rows linked to discussions
✓ stores system notes with is_system=1 flag
✓ handles individual_note=true discussions
tests/integration/sync-runs.test.ts
✓ creates sync_run record on start
✓ marks run as succeeded on completion
✓ marks run as failed with error message on failure
✓ refuses concurrent run (single-flight)
✓ allows --force to override stale running status
```
**Manual CLI Smoke Tests:**
| Command | Expected Output | Pass Criteria |
|---------|-----------------|---------------|
| `gi ingest --type=issues` | Progress bar, final count | Completes without error |
| `gi list issues --limit=10` | Table of 10 issues | Shows iid, title, state, author |
| `gi list issues --project=group/project-one` | Filtered list | Only shows issues from that project |
| `gi count issues` | `Issues: 1,234` (example) | Count matches GitLab UI |
| `gi show issue 123` | Issue detail view | Shows title, description, labels, discussions, URL |
| `gi count discussions --type=issue` | `Issue Discussions: 5,678` | Non-zero count |
| `gi count notes --type=issue` | `Issue Notes: 12,345 (excluding 2,345 system)` | Non-zero count |
| `gi sync-status` | Last sync time, cursor positions | Shows successful last run |
**Data Integrity Checks:**
- [ ] `SELECT COUNT(*) FROM issues` matches GitLab issue count for configured projects
- [ ] Every issue has a corresponding `raw_payloads` row
- [ ] Labels in `issue_labels` junction all exist in `labels` table
- [ ] `sync_cursors` has entry for each (project_id, 'issues') pair
- [ ] Re-running `gi ingest --type=issues` fetches 0 new items (cursor is current)
- [ ] `SELECT COUNT(*) FROM discussions WHERE noteable_type='Issue'` is non-zero
- [ ] Every discussion has at least one note
- [ ] `individual_note = true` discussions have exactly one note
**Scope:**
- Issue fetcher with pagination handling
- Raw JSON storage in raw_payloads table
- Normalized issue schema in SQLite
- Labels ingestion derived from issue payload:
- Always persist label names from `labels: string[]`
- Optionally request `with_labels_details=true` to capture color/description when available
- Issue discussions fetcher:
- Uses `GET /projects/:id/issues/:iid/discussions`
- Fetches all discussions for each issue during ingest
- Preserve system notes but flag them with `is_system=1`
- Incremental sync support (run tracking + per-project cursor)
- Basic list/count CLI commands
**Reliability/Idempotency Rules:**
- Every ingest/sync creates a `sync_runs` row
- Single-flight via DB-enforced app lock:
- On start: acquire lock via transactional compare-and-swap:
- `BEGIN IMMEDIATE` (acquires write lock immediately)
- If no row exists → INSERT new lock
- Else if `heartbeat_at` is stale (> staleLockMinutes) → UPDATE owner + timestamps
- Else if `owner` matches current run → UPDATE heartbeat (re-entrant)
- Else → ROLLBACK and fail fast (another run is active)
- `COMMIT`
- During run: update `heartbeat_at` every 30 seconds
- If existing lock's `heartbeat_at` is stale (> 10 minutes), treat as abandoned and acquire
- `--force` remains as operator override for edge cases, but should rarely be needed
- Cursor advances only after successful transaction commit per page/batch
- Ordering: `updated_at ASC`, tie-breaker `gitlab_id ASC`
- Use explicit transactions for batch inserts
**Schema Preview:**
```sql
CREATE TABLE issues (
id INTEGER PRIMARY KEY,
gitlab_id INTEGER UNIQUE NOT NULL,
project_id INTEGER NOT NULL REFERENCES projects(id),
iid INTEGER NOT NULL,
title TEXT,
description TEXT,
state TEXT,
author_username TEXT,
created_at INTEGER,
updated_at INTEGER,
last_seen_at INTEGER NOT NULL, -- updated on every upsert during sync
web_url TEXT,
raw_payload_id INTEGER REFERENCES raw_payloads(id)
);
CREATE INDEX idx_issues_project_updated ON issues(project_id, updated_at);
CREATE INDEX idx_issues_author ON issues(author_username);
CREATE UNIQUE INDEX uq_issues_project_iid ON issues(project_id, iid);
-- Labels are derived from issue payloads (string array)
-- Uniqueness is (project_id, name) since gitlab_id isn't always available
CREATE TABLE labels (
id INTEGER PRIMARY KEY,
gitlab_id INTEGER, -- optional (only if available)
project_id INTEGER NOT NULL REFERENCES projects(id),
name TEXT NOT NULL,
color TEXT,
description TEXT
);
CREATE UNIQUE INDEX uq_labels_project_name ON labels(project_id, name);
CREATE INDEX idx_labels_name ON labels(name);
CREATE TABLE issue_labels (
issue_id INTEGER REFERENCES issues(id),
label_id INTEGER REFERENCES labels(id),
PRIMARY KEY(issue_id, label_id)
);
CREATE INDEX idx_issue_labels_label ON issue_labels(label_id);
-- Discussion threads for issues (MR discussions added in CP2)
CREATE TABLE discussions (
id INTEGER PRIMARY KEY,
gitlab_discussion_id TEXT NOT NULL, -- GitLab's string ID (e.g. "6a9c1750b37d...")
project_id INTEGER NOT NULL REFERENCES projects(id),
issue_id INTEGER REFERENCES issues(id),
merge_request_id INTEGER REFERENCES merge_requests(id),
noteable_type TEXT NOT NULL, -- 'Issue' | 'MergeRequest'
individual_note BOOLEAN NOT NULL, -- standalone comment vs threaded discussion
first_note_at INTEGER, -- for ordering discussions
last_note_at INTEGER, -- for "recently active" queries
last_seen_at INTEGER NOT NULL, -- updated on every upsert during sync
resolvable BOOLEAN, -- MR discussions can be resolved
resolved BOOLEAN,
CHECK (
(noteable_type='Issue' AND issue_id IS NOT NULL AND merge_request_id IS NULL) OR
(noteable_type='MergeRequest' AND merge_request_id IS NOT NULL AND issue_id IS NULL)
)
);
CREATE UNIQUE INDEX uq_discussions_project_discussion_id ON discussions(project_id, gitlab_discussion_id);
CREATE INDEX idx_discussions_issue ON discussions(issue_id);
CREATE INDEX idx_discussions_mr ON discussions(merge_request_id);
CREATE INDEX idx_discussions_last_note ON discussions(last_note_at);
-- Notes belong to discussions (preserving thread context)
CREATE TABLE notes (
id INTEGER PRIMARY KEY,
gitlab_id INTEGER UNIQUE NOT NULL,
discussion_id INTEGER NOT NULL REFERENCES discussions(id),
project_id INTEGER NOT NULL REFERENCES projects(id),
type TEXT, -- 'DiscussionNote' | 'DiffNote' | null
is_system BOOLEAN NOT NULL DEFAULT 0, -- system notes (assignments, label changes, etc.)
author_username TEXT,
body TEXT,
created_at INTEGER,
updated_at INTEGER,
last_seen_at INTEGER NOT NULL, -- updated on every upsert during sync
position INTEGER, -- derived from array order in API response (0-indexed)
resolvable BOOLEAN,
resolved BOOLEAN,
resolved_by TEXT,
resolved_at INTEGER,
-- DiffNote position metadata (only populated for MR DiffNotes in CP2)
position_old_path TEXT,
position_new_path TEXT,
position_old_line INTEGER,
position_new_line INTEGER,
raw_payload_id INTEGER REFERENCES raw_payloads(id)
);
CREATE INDEX idx_notes_discussion ON notes(discussion_id);
CREATE INDEX idx_notes_author ON notes(author_username);
CREATE INDEX idx_notes_system ON notes(is_system);
```
---
### Checkpoint 2: MR Ingestion
**Deliverable:** All MRs + MR discussions + notes with DiffNote paths captured
**Automated Tests (Vitest):**
```
tests/unit/mr-transformer.test.ts
✓ transforms GitLab MR payload to normalized schema
✓ extracts labels from MR payload
✓ handles missing optional fields gracefully
tests/unit/diffnote-transformer.test.ts
✓ extracts DiffNote position metadata (paths and lines)
✓ handles missing position fields gracefully
tests/integration/mr-ingestion.test.ts
✓ inserts MRs into database
✓ creates labels from MR payloads
✓ links MRs to labels via junction table
✓ stores raw payload for each MR
tests/integration/mr-discussion-ingestion.test.ts
✓ fetches discussions for each MR
✓ creates discussion rows with correct MR FK
✓ creates note rows linked to discussions
✓ extracts position_new_path from DiffNotes
✓ captures note-level resolution status
✓ captures note type (DiscussionNote, DiffNote)
```
**Manual CLI Smoke Tests:**
| Command | Expected Output | Pass Criteria |
|---------|-----------------|---------------|
| `gi ingest --type=merge_requests` | Progress bar, final count | Completes without error |
| `gi list mrs --limit=10` | Table of 10 MRs | Shows iid, title, state, author, branch |
| `gi count mrs` | `Merge Requests: 567` (example) | Count matches GitLab UI |
| `gi show mr 123` | MR detail with discussions | Shows title, description, discussion threads |
| `gi count discussions` | `Discussions: 12,345` | Total count (issue + MR) |
| `gi count discussions --type=mr` | `MR Discussions: 6,789` | MR discussions only |
| `gi count notes` | `Notes: 45,678 (excluding 8,901 system)` | Total with system note count |
**Data Integrity Checks:**
- [ ] `SELECT COUNT(*) FROM merge_requests` matches GitLab MR count
- [ ] `SELECT COUNT(*) FROM discussions WHERE noteable_type='MergeRequest'` is non-zero
- [ ] DiffNotes have `position_new_path` populated when available
- [ ] Discussion `first_note_at` <= `last_note_at` for all rows
**Scope:**
- MR fetcher with pagination
- MR discussions fetcher:
- Uses `GET /projects/:id/merge_requests/:iid/discussions`
- Fetches all discussions for each MR during ingest
- Capture DiffNote file path/line metadata from `position` field for filename search
- Relationship linking (discussion → MR, notes → discussion)
- Extended CLI commands for MR display with threads
- Add `idx_notes_type` and `idx_notes_new_path` indexes for DiffNote queries
**Note:** MR file changes (mr_files) are deferred to Checkpoint 6 (File History) since they're only needed for "what MRs touched this file?" queries.
**Schema Additions:**
```sql
CREATE TABLE merge_requests (
id INTEGER PRIMARY KEY,
gitlab_id INTEGER UNIQUE NOT NULL,
project_id INTEGER NOT NULL REFERENCES projects(id),
iid INTEGER NOT NULL,
title TEXT,
description TEXT,
state TEXT,
author_username TEXT,
source_branch TEXT,
target_branch TEXT,
created_at INTEGER,
updated_at INTEGER,
last_seen_at INTEGER NOT NULL, -- updated on every upsert during sync
merged_at INTEGER,
web_url TEXT,
raw_payload_id INTEGER REFERENCES raw_payloads(id)
);
CREATE INDEX idx_mrs_project_updated ON merge_requests(project_id, updated_at);
CREATE INDEX idx_mrs_author ON merge_requests(author_username);
CREATE UNIQUE INDEX uq_mrs_project_iid ON merge_requests(project_id, iid);
-- MR labels (reuse same labels table from CP1)
CREATE TABLE mr_labels (
merge_request_id INTEGER REFERENCES merge_requests(id),
label_id INTEGER REFERENCES labels(id),
PRIMARY KEY(merge_request_id, label_id)
);
CREATE INDEX idx_mr_labels_label ON mr_labels(label_id);
-- Additional indexes for DiffNote queries (tables created in CP1)
CREATE INDEX idx_notes_type ON notes(type);
CREATE INDEX idx_notes_new_path ON notes(position_new_path);
```
**MR Discussion Processing Rules:**
- DiffNote position data is extracted and stored:
- `position.old_path`, `position.new_path` for file-level search
- `position.old_line`, `position.new_line` for line-level context
- MR discussions can be resolvable; resolution status is captured at note level
---
### Checkpoint 3: Document + Embedding Generation with Lexical Search
**Deliverable:** Documents and embeddings generated; `gi search --mode=lexical` works end-to-end
**Automated Tests (Vitest):**
```
tests/unit/document-extractor.test.ts
✓ extracts issue document (title + description)
✓ extracts MR document (title + description)
✓ extracts discussion document with full thread context
✓ includes parent issue/MR title in discussion header
✓ formats notes with author and timestamp
✓ truncates content exceeding 8000 tokens
✓ preserves first and last notes when truncating middle
✓ computes SHA-256 content hash consistently
tests/unit/embedding-client.test.ts
✓ connects to Ollama API
✓ generates embedding for text input
✓ returns 768-dimension vector
✓ handles Ollama connection failure gracefully
✓ batches requests (32 documents per batch)
tests/integration/document-creation.test.ts
✓ creates document for each issue
✓ creates document for each MR
✓ creates document for each discussion
✓ populates document_labels junction table
✓ computes content_hash for each document
tests/integration/embedding-storage.test.ts
✓ stores embedding in sqlite-vss
✓ embedding rowid matches document id
✓ creates embedding_metadata record
✓ skips re-embedding when content_hash unchanged
✓ re-embeds when content_hash changes
```
**Manual CLI Smoke Tests:**
| Command | Expected Output | Pass Criteria |
|---------|-----------------|---------------|
| `gi embed --all` | Progress bar with ETA | Completes without error |
| `gi embed --all` (re-run) | `0 documents to embed` | Skips already-embedded docs |
| `gi stats` | Embedding coverage stats | Shows 100% coverage |
| `gi stats --json` | JSON stats object | Valid JSON with document/embedding counts |
| `gi embed --all` (Ollama stopped) | Clear error message | Non-zero exit, actionable error |
| `gi search "authentication" --mode=lexical` | FTS results | Returns matching documents, no embeddings required |
**Data Integrity Checks:**
- [ ] `SELECT COUNT(*) FROM documents` = issues + MRs + discussions
- [ ] `SELECT COUNT(*) FROM embeddings` = `SELECT COUNT(*) FROM documents`
- [ ] `SELECT COUNT(*) FROM embedding_metadata` = `SELECT COUNT(*) FROM documents`
- [ ] All `embedding_metadata.content_hash` matches corresponding `documents.content_hash`
- [ ] `SELECT COUNT(*) FROM documents WHERE LENGTH(content_text) > 32000` logs truncation warnings
- [ ] Discussion documents include parent title in content_text
**Scope:**
- Ollama integration (nomic-embed-text model)
- Embedding generation pipeline:
- Batch size: 32 documents per batch
- Concurrency: configurable (default 4 workers)
- Retry with exponential backoff for transient failures (max 3 attempts)
- Per-document failure recording to enable targeted re-runs
- Vector storage in SQLite (sqlite-vss extension)
- Progress tracking and resumability
- Document extraction layer:
- Canonical "search documents" derived from issues/MRs/discussions
- Stable content hashing for change detection (SHA-256 of content_text)
- Single embedding per document (chunking deferred to post-MVP)
- Truncation: content_text capped at 8000 tokens (nomic-embed-text limit is 8192)
- **Implementation:** Use character budget, not exact token count
- `maxChars = 32000` (conservative 4 chars/token estimate)
- `approxTokens = ceil(charCount / 4)` for reporting/logging only
- This avoids tokenizer dependency while preventing embedding failures
- Denormalized metadata for fast filtering (author, labels, dates)
- Fast label filtering via `document_labels` join table
- FTS5 index for lexical search (enables `gi search --mode=lexical` without Ollama)
- `gi search --mode=lexical` CLI command (works without Ollama)
**Schema Additions:**
```sql
-- Unified searchable documents (derived from issues/MRs/discussions)
CREATE TABLE documents (
id INTEGER PRIMARY KEY,
source_type TEXT NOT NULL, -- 'issue' | 'merge_request' | 'discussion'
source_id INTEGER NOT NULL, -- local DB id in the source table
project_id INTEGER NOT NULL REFERENCES projects(id),
author_username TEXT, -- for discussions: first note author
label_names TEXT, -- JSON array (display/debug only)
created_at INTEGER,
updated_at INTEGER,
url TEXT,
title TEXT, -- null for discussions
content_text TEXT NOT NULL, -- canonical text for embedding/snippets
content_hash TEXT NOT NULL, -- SHA-256 for change detection
is_truncated BOOLEAN NOT NULL DEFAULT 0,
truncated_reason TEXT, -- 'token_limit_middle_drop' | null
UNIQUE(source_type, source_id)
);
CREATE INDEX idx_documents_project_updated ON documents(project_id, updated_at);
CREATE INDEX idx_documents_author ON documents(author_username);
CREATE INDEX idx_documents_source ON documents(source_type, source_id);
-- Fast label filtering for documents (indexed exact-match)
CREATE TABLE document_labels (
document_id INTEGER NOT NULL REFERENCES documents(id),
label_name TEXT NOT NULL,
PRIMARY KEY(document_id, label_name)
);
CREATE INDEX idx_document_labels_label ON document_labels(label_name);
-- sqlite-vss virtual table
-- Storage rule: embeddings.rowid = documents.id
CREATE VIRTUAL TABLE embeddings USING vss0(
embedding(768)
);
-- Embedding provenance + change detection
-- document_id is PRIMARY KEY and equals embeddings.rowid
CREATE TABLE embedding_metadata (
document_id INTEGER PRIMARY KEY REFERENCES documents(id),
model TEXT NOT NULL, -- 'nomic-embed-text'
dims INTEGER NOT NULL, -- 768
content_hash TEXT NOT NULL, -- copied from documents.content_hash
created_at INTEGER NOT NULL,
-- Error tracking for resumable embedding
last_error TEXT, -- error message from last failed attempt
attempt_count INTEGER NOT NULL DEFAULT 0,
last_attempt_at INTEGER -- when last attempt occurred
);
-- Index for finding failed embeddings to retry
CREATE INDEX idx_embedding_metadata_errors ON embedding_metadata(last_error) WHERE last_error IS NOT NULL;
-- Track sources that require document regeneration (populated during ingestion)
CREATE TABLE dirty_sources (
source_type TEXT NOT NULL, -- 'issue' | 'merge_request' | 'discussion'
source_id INTEGER NOT NULL, -- local DB id
queued_at INTEGER NOT NULL,
PRIMARY KEY(source_type, source_id)
);
-- Fast path filtering for documents (extracted from DiffNote positions)
CREATE TABLE document_paths (
document_id INTEGER NOT NULL REFERENCES documents(id),
path TEXT NOT NULL,
PRIMARY KEY(document_id, path)
);
CREATE INDEX idx_document_paths_path ON document_paths(path);
```
**Storage Rule (MVP):**
- Insert embedding with `rowid = documents.id`
- Upsert `embedding_metadata` by `document_id`
- This alignment simplifies joins and eliminates rowid mapping fragility
**Document Extraction Rules:**
| Source | content_text Construction |
|--------|--------------------------|
| Issue | `title + "\n\n" + description` |
| MR | `title + "\n\n" + description` |
| Discussion | Full thread with context (see below) |
**Discussion Document Format:**
```
[[Discussion]] Issue #234: Authentication redesign
Project: group/project-one
URL: https://gitlab.example.com/group/project-one/-/issues/234#note_12345
Labels: ["bug", "auth"]
Files: ["src/auth/login.ts"] -- present if any DiffNotes exist in thread
--- Thread ---
@johndoe (2024-03-15):
I think we should move to JWT-based auth because the session cookies are causing issues with our mobile clients...
@janedoe (2024-03-15):
Agreed. What about refresh token strategy?
@johndoe (2024-03-16):
Short-lived access tokens (15min), longer refresh (7 days). Here's why...
```
This format preserves:
- Parent context (issue/MR title and number)
- Project path for scoped search
- Direct URL for navigation
- Labels for context
- File paths from DiffNotes (enables immediate file search)
- Author attribution for each note
- Temporal ordering of the conversation
- Full thread semantics for decision traceability
**Truncation:**
If content exceeds 8000 tokens:
**Note:** Token count is approximate (`ceil(charCount / 4)`). Enforce `maxChars = 32000`.
1. Truncate from the middle (preserve first + last notes for context)
2. Set `documents.is_truncated = 1`
3. Set `documents.truncated_reason = 'token_limit_middle_drop'`
4. Log a warning with document ID and original token count
This metadata enables:
- Monitoring truncation frequency in production
- Future investigation of high-value truncated documents
- Debugging when search misses expected content
---
### Checkpoint 4: Hybrid Search (Semantic + Lexical)
**Deliverable:** Working hybrid semantic search (vector + FTS5 + RRF) across all indexed content
**Automated Tests (Vitest):**
```
tests/unit/search-query.test.ts
✓ parses filter flags (--type, --author, --after, --label)
✓ validates date format for --after
✓ handles multiple --label flags
tests/unit/rrf-ranking.test.ts
✓ computes RRF score correctly
✓ merges results from vector and FTS retrievers
✓ handles documents appearing in only one retriever
✓ respects k=60 parameter
tests/integration/vector-search.test.ts
✓ returns results for semantic query
✓ ranks similar content higher
✓ returns empty for nonsense query
tests/integration/fts-search.test.ts
✓ returns exact keyword matches
✓ handles porter stemming (search/searching)
✓ returns empty for non-matching query
tests/integration/hybrid-search.test.ts
✓ combines vector and FTS results
✓ applies type filter correctly
✓ applies author filter correctly
✓ applies date filter correctly
✓ applies label filter correctly
✓ falls back to FTS when Ollama unavailable
tests/e2e/golden-queries.test.ts
✓ "authentication redesign" returns known auth-related items
✓ "database migration" returns known migration items
✓ [8 more domain-specific golden queries]
```
**Manual CLI Smoke Tests:**
| Command | Expected Output | Pass Criteria |
|---------|-----------------|---------------|
| `gi search "authentication"` | Ranked results with snippets | Returns relevant items, shows score |
| `gi search "authentication" --project=group/project-one` | Project-scoped results | Only results from that project |
| `gi search "authentication" --type=mr` | Only MR results | No issues or discussions in output |
| `gi search "authentication" --author=johndoe` | Filtered by author | All results have @johndoe |
| `gi search "authentication" --after=2024-01-01` | Date filtered | All results after date |
| `gi search "authentication" --label=bug` | Label filtered | All results have bug label |
| `gi search "redis" --mode=lexical` | FTS results only | Works without Ollama |
| `gi search "auth" --path=src/auth/` | Path-filtered results | Only results referencing files in src/auth/ |
| `gi search "authentication" --json` | JSON output | Valid JSON matching stable schema |
| `gi search "authentication" --explain` | Rank breakdown | Shows vector/FTS/RRF contributions |
| `gi search "authentication" --limit=5` | 5 results max | Returns at most 5 results |
| `gi search "xyznonexistent123"` | No results message | Graceful empty state |
| `gi search "auth"` (no data synced) | No data message | Shows "Run gi sync first" |
| `gi search "auth"` (Ollama stopped) | FTS results + warning | Shows warning, still returns results |
**Golden Query Test Suite:**
Create `tests/fixtures/golden-queries.json` with 10 queries and expected URLs:
```json
[
{
"query": "authentication redesign",
"expectedUrls": [".../-/issues/234", ".../-/merge_requests/847"],
"minResults": 1,
"maxRank": 10
}
]
```
Each query must have at least one expected URL appear in top 10 results.
**Data Integrity Checks:**
- [ ] `documents_fts` row count matches `documents` row count
- [ ] Search returns results for known content (not empty)
- [ ] JSON output validates against defined schema
- [ ] All result URLs are valid GitLab URLs
**Scope:**
- Hybrid retrieval:
- Vector recall (sqlite-vss) + FTS lexical recall (fts5)
- Merge + rerank results using Reciprocal Rank Fusion (RRF)
- Query embedding generation (same Ollama pipeline as documents)
- Result ranking and scoring (document-level)
- Search filters: `--type=issue|mr|discussion`, `--author=username`, `--after=date`, `--label=name`, `--project=path`, `--path=file`, `--limit=N`
- `--limit=N` controls result count (default: 20, max: 100)
- `--path` filters documents by referenced file paths (from DiffNote positions)
- MVP: substring/exact match; glob patterns deferred
- Label filtering operates on `document_labels` (indexed, exact-match)
- Filters work identically in hybrid and lexical modes
- Debug: `--explain` returns rank contributions from vector + FTS + RRF
- Output formatting: ranked list with title, snippet, score, URL
- JSON output mode for AI/agent consumption (stable documented schema)
- Graceful degradation: if Ollama is unreachable, fall back to FTS5-only search with warning
- Empty state handling:
- No documents indexed: `No data indexed. Run 'gi sync' first.`
- Query returns no results: `No results found for "query".`
- Filters exclude all results: `No results match the specified filters.`
- Helpful hints shown in non-JSON mode (e.g., "Try broadening your search")
**Schema Additions:**
```sql
-- Full-text search for hybrid retrieval
-- Using porter stemmer for better matching of word variants
CREATE VIRTUAL TABLE documents_fts USING fts5(
title,
content_text,
content='documents',
content_rowid='id',
tokenize='porter unicode61'
);
-- Triggers to keep FTS in sync
CREATE TRIGGER documents_ai AFTER INSERT ON documents BEGIN
INSERT INTO documents_fts(rowid, title, content_text)
VALUES (new.id, new.title, new.content_text);
END;
CREATE TRIGGER documents_ad AFTER DELETE ON documents BEGIN
INSERT INTO documents_fts(documents_fts, rowid, title, content_text)
VALUES('delete', old.id, old.title, old.content_text);
END;
CREATE TRIGGER documents_au AFTER UPDATE ON documents BEGIN
INSERT INTO documents_fts(documents_fts, rowid, title, content_text)
VALUES('delete', old.id, old.title, old.content_text);
INSERT INTO documents_fts(rowid, title, content_text)
VALUES (new.id, new.title, new.content_text);
END;
```
**FTS5 Tokenizer Notes:**
- `porter` enables stemming (searching "authentication" matches "authenticating", "authenticated")
- `unicode61` handles Unicode properly
- Code identifiers (snake_case, camelCase, file paths) may not tokenize ideally; post-MVP consideration for custom tokenizer
**Hybrid Search Algorithm (MVP) - Reciprocal Rank Fusion:**
1. Query both vector index (top 50) and FTS5 (top 50)
2. Merge results by document_id
3. Combine with Reciprocal Rank Fusion (RRF):
- For each retriever list, assign ranks (1..N)
- `rrf_score = Σ 1 / (k + rank)` with k=60 (tunable)
- RRF is simpler than weighted sums and doesn't require score normalization
4. Apply filters (type, author, date, label)
5. Return top K
**Why RRF over Weighted Sums:**
- FTS5 BM25 scores and vector distances use different scales
- Weighted sums (`0.7 * vector + 0.3 * fts`) require careful normalization
- RRF operates on ranks, not scores, making it robust to scale differences
- Well-established in information retrieval literature
**Graceful Degradation:**
- If Ollama is unreachable during search, automatically fall back to FTS5-only search
- Display warning: "Embedding service unavailable, using lexical search only"
- `embed` command fails with actionable error if Ollama is down
**CLI Interface:**
```bash
# Basic semantic search
gi search "authentication redesign"
# Search within specific project
gi search "authentication" --project=group/project-one
# Search by file path (finds discussions/MRs touching this file)
gi search "rate limit" --path=src/client.ts
# Pure FTS search (fallback if embeddings unavailable)
gi search "redis" --mode=lexical
# Filtered search
gi search "authentication" --type=mr --after=2024-01-01
# Filter by label
gi search "performance" --label=bug --label=critical
# JSON output for programmatic use
gi search "payment processing" --json
# Explain search (shows RRF contributions)
gi search "auth" --explain
```
**CLI Output Example:**
```
$ gi search "authentication"
Found 23 results (hybrid search, 0.34s)
[1] MR !847 - Refactor auth to use JWT tokens (0.82)
@johndoe · 2024-03-15 · group/project-one
"...moving away from session cookies to JWT for authentication..."
https://gitlab.example.com/group/project-one/-/merge_requests/847
[2] Issue #234 - Authentication redesign discussion (0.79)
@janedoe · 2024-02-28 · group/project-one
"...we need to redesign the authentication flow because..."
https://gitlab.example.com/group/project-one/-/issues/234
[3] Discussion on Issue #234 (0.76)
@johndoe · 2024-03-01 · group/project-one
"I think we should move to JWT-based auth because the session..."
https://gitlab.example.com/group/project-one/-/issues/234#note_12345
```
**JSON Output Schema (Stable):**
For AI/agent consumption, `--json` output follows this stable schema:
```typescript
interface SearchResult {
documentId: number;
sourceType: "issue" | "merge_request" | "discussion";
title: string | null;
url: string;
projectPath: string;
author: string | null;
createdAt: string; // ISO 8601
updatedAt: string; // ISO 8601
score: number; // 0-1 normalized RRF score
snippet: string; // truncated content_text
labels: string[];
// Only present with --explain flag
explain?: {
vectorRank?: number; // null if not in vector results
ftsRank?: number; // null if not in FTS results
rrfScore: number;
};
}
interface SearchResponse {
query: string;
mode: "hybrid" | "lexical" | "semantic";
totalResults: number;
results: SearchResult[];
warnings?: string[]; // e.g., "Embedding service unavailable"
}
```
**Schema versioning:** Breaking changes require major version bump in CLI. Non-breaking additions (new optional fields) are allowed.
---
### Checkpoint 5: Incremental Sync
**Deliverable:** Efficient ongoing synchronization with GitLab
**Automated Tests (Vitest):**
```
tests/unit/cursor-management.test.ts
✓ advances cursor after successful page commit
✓ uses tie-breaker id for identical timestamps
✓ does not advance cursor on failure
✓ resets cursor on --full flag
tests/unit/change-detection.test.ts
✓ detects content_hash mismatch
✓ queues document for re-embedding on change
✓ skips re-embedding when hash unchanged
tests/integration/incremental-sync.test.ts
✓ fetches only items updated after cursor
✓ refetches discussions for updated issues
✓ refetches discussions for updated MRs
✓ updates existing records (not duplicates)
✓ creates new records for new items
✓ re-embeds documents with changed content
tests/integration/sync-recovery.test.ts
✓ resumes from cursor after interrupted sync
✓ marks failed run with error message
✓ handles rate limiting (429) with backoff
✓ respects Retry-After header
```
**Manual CLI Smoke Tests:**
| Command | Expected Output | Pass Criteria |
|---------|-----------------|---------------|
| `gi sync` (no changes) | `0 issues, 0 MRs updated` | Fast completion, no API calls beyond cursor check |
| `gi sync` (after GitLab change) | `1 issue updated, 3 discussions refetched` | Detects and syncs the change |
| `gi sync --full` | Full sync progress | Resets cursors, fetches everything |
| `gi sync-status` | Cursor positions, last sync time | Shows current state |
| `gi sync` (with rate limit) | Backoff messages | Respects rate limits, completes eventually |
| `gi search "new content"` (after sync) | Returns new content | New content is searchable |
**End-to-End Sync Verification:**
1. Note the current `sync_cursors` values
2. Create a new comment on an issue in GitLab
3. Run `gi sync`
4. Verify:
- [ ] Issue's `updated_at` in DB matches GitLab
- [ ] New discussion row exists
- [ ] New note row exists
- [ ] New document row exists for discussion
- [ ] New embedding exists for document
- [ ] `gi search "new comment text"` returns the new discussion
- [ ] Cursor advanced past the updated issue
**Data Integrity Checks:**
- [ ] `sync_cursors` timestamp <= max `updated_at` in corresponding table
- [ ] No orphaned documents (all have valid source_id)
- [ ] `embedding_metadata.content_hash` = `documents.content_hash` for all rows
- [ ] `sync_runs` has complete audit trail
**Scope:**
- Delta sync based on stable tuple cursor `(updated_at, gitlab_id)`
- Rolling backfill window (configurable, default 14 days) to reduce risk of missed updates
- Dependent resources sync strategy (discussions refetched when parent updates)
- Re-embedding based on content_hash change (documents.content_hash != embedding_metadata.content_hash)
- Sync status reporting
- Recommended: run via cron every 10 minutes
**Correctness Rules (MVP):**
1. Fetch pages ordered by `updated_at ASC`, within identical timestamps by `gitlab_id ASC`
2. Cursor is a stable tuple `(updated_at, gitlab_id)`:
- **GitLab API cannot express `(updated_at = X AND id > Y)` server-side.**
- Use **cursor rewind + local filtering**:
- Call GitLab with `updated_after = cursor_updated_at - rewindSeconds` (default 2s, configurable)
- Locally discard items where:
- `updated_at < cursor_updated_at`, OR
- `updated_at = cursor_updated_at AND gitlab_id <= cursor_gitlab_id`
- This makes the tuple cursor rule true in practice while keeping API calls simple.
- Cursor advances only after successful DB commit for that page
- When advancing, set cursor to the last processed item's `(updated_at, gitlab_id)`
3. Dependent resources:
- For each updated issue/MR, refetch ALL its discussions
- Discussion documents are regenerated and re-embedded if content_hash changes
4. Rolling backfill window:
- After cursor-based delta sync, also fetch items where `updated_at > NOW() - backfillDays`
- This catches any items whose timestamps were updated without triggering our cursor
5. A document is queued for embedding iff `documents.content_hash != embedding_metadata.content_hash`
6. Sync run is marked 'failed' with error message if any page fails (can resume from cursor)
**Why Dependent Resource Model:**
- GitLab Discussions API doesn't provide a global `updated_after` stream
- Discussions are listed per-issue or per-MR, not as a top-level resource
- Treating discussions as dependent resources (refetch when parent updates) is simpler and more correct
**CLI Commands:**
```bash
# Full sync orchestration (ingest -> docs -> embed -> ensure FTS synced)
gi sync # orchestrates all steps
gi sync --no-embed # skip embedding step (fast ingest/debug)
gi sync --no-docs # skip document regeneration (debug)
# Force full re-sync (resets cursors)
gi sync --full
# Override stale 'running' run after operator review
gi sync --force
# Show sync status
gi sync-status
```
**Orchestration steps (in order):**
1. Acquire app lock with heartbeat
2. Ingest delta (issues, MRs, discussions) based on cursors
- During ingestion, INSERT into `dirty_sources` for each upserted entity
3. Apply rolling backfill window
4. Regenerate documents for entities in `dirty_sources` (process + delete from queue)
5. Embed documents with changed content_hash
6. FTS triggers auto-sync (no explicit step needed)
7. Release lock, record sync_run as succeeded
Individual commands remain available for checkpoint testing and debugging:
- `gi ingest --type=issues`
- `gi ingest --type=merge_requests`
- `gi embed --all`
- `gi embed --retry-failed`
---
## CLI Command Reference
All commands support `--help` for detailed usage information.
### Setup & Diagnostics
| Command | CP | Description |
|---------|-----|-------------|
| `gi init` | 0 | Interactive setup wizard; creates gi.config.json |
| `gi auth-test` | 0 | Verify GitLab authentication |
| `gi doctor` | 0 | Check environment (GitLab, Ollama, DB) |
| `gi doctor --json` | 0 | JSON output for scripting |
| `gi version` | 0 | Show installed version |
### Data Ingestion
| Command | CP | Description |
|---------|-----|-------------|
| `gi ingest --type=issues` | 1 | Fetch issues from GitLab |
| `gi ingest --type=merge_requests` | 2 | Fetch MRs and discussions |
| `gi embed --all` | 3 | Generate embeddings for all documents |
| `gi embed --retry-failed` | 3 | Retry failed embeddings |
| `gi sync` | 5 | Full sync orchestration (ingest + docs + embed) |
| `gi sync --full` | 5 | Force complete re-sync (reset cursors) |
| `gi sync --force` | 5 | Override stale lock after operator review |
| `gi sync --no-embed` | 5 | Sync without embedding (faster) |
### Data Inspection
| Command | CP | Description |
|---------|-----|-------------|
| `gi list issues [--limit=N] [--project=PATH]` | 1 | List issues |
| `gi list mrs [--limit=N]` | 2 | List merge requests |
| `gi count issues` | 1 | Count issues |
| `gi count mrs` | 2 | Count merge requests |
| `gi count discussions --type=issue` | 1 | Count issue discussions |
| `gi count discussions` | 2 | Count all discussions |
| `gi count discussions --type=mr` | 2 | Count MR discussions |
| `gi count notes --type=issue` | 1 | Count issue notes (excluding system) |
| `gi count notes` | 2 | Count all notes (excluding system) |
| `gi show issue <iid>` | 1 | Show issue details |
| `gi show mr <iid>` | 2 | Show MR details with discussions |
| `gi stats` | 3 | Embedding coverage statistics |
| `gi stats --json` | 3 | JSON stats for scripting |
| `gi sync-status` | 1 | Show cursor positions and last sync |
### Search
| Command | CP | Description |
|---------|-----|-------------|
| `gi search "query"` | 4 | Hybrid semantic + lexical search |
| `gi search "query" --mode=lexical` | 3 | Lexical-only search (no Ollama required) |
| `gi search "query" --type=issue\|mr\|discussion` | 4 | Filter by document type |
| `gi search "query" --author=USERNAME` | 4 | Filter by author |
| `gi search "query" --after=YYYY-MM-DD` | 4 | Filter by date |
| `gi search "query" --label=NAME` | 4 | Filter by label (repeatable) |
| `gi search "query" --project=PATH` | 4 | Filter by project |
| `gi search "query" --path=FILE` | 4 | Filter by file path |
| `gi search "query" --limit=N` | 4 | Limit results (default: 20, max: 100) |
| `gi search "query" --json` | 4 | JSON output for scripting |
| `gi search "query" --explain` | 4 | Show ranking breakdown |
### Database Management
| Command | CP | Description |
|---------|-----|-------------|
| `gi backup` | 0 | Create timestamped database backup |
| `gi reset --confirm` | 0 | Delete database and reset cursors |
---
## Error Handling
Common errors and their resolutions:
### Configuration Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| `Config file not found` | No gi.config.json | Run `gi init` to create configuration |
| `Invalid config: missing baseUrl` | Malformed config | Re-run `gi init` or fix gi.config.json manually |
| `Invalid config: no projects defined` | Empty projects array | Add at least one project path to config |
### Authentication Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| `GITLAB_TOKEN environment variable not set` | Token not exported | `export GITLAB_TOKEN="glpat-xxx"` |
| `401 Unauthorized` | Invalid or expired token | Generate new token with `read_api` scope |
| `403 Forbidden` | Token lacks permissions | Ensure token has `read_api` scope |
### GitLab API Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| `Project not found: group/project` | Invalid project path | Verify path matches GitLab URL (case-sensitive) |
| `429 Too Many Requests` | Rate limited | Wait for Retry-After period; sync will auto-retry |
| `Connection refused` | GitLab unreachable | Check GitLab URL and network connectivity |
### Data Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| `No documents indexed` | Sync not run | Run `gi sync` first |
| `No results found` | Query too specific | Try broader search terms |
| `Database locked` | Concurrent access | Wait for other process; use `gi sync --force` if stale |
### Embedding Errors
| Error | Cause | Resolution |
|-------|-------|------------|
| `Ollama connection refused` | Ollama not running | Start Ollama or use `--mode=lexical` |
| `Model not found: nomic-embed-text` | Model not pulled | Run `ollama pull nomic-embed-text` |
| `Embedding failed for N documents` | Transient failures | Run `gi embed --retry-failed` |
### Operational Behavior
| Scenario | Behavior |
|----------|----------|
| **Ctrl+C during sync** | Graceful shutdown: finishes current page, commits cursor, exits cleanly. Resume with `gi sync`. |
| **Disk full during write** | Fails with clear error. Cursor preserved at last successful commit. Free space and resume. |
| **Stale lock detected** | Lock held > 10 minutes without heartbeat is considered stale. Next sync auto-recovers. |
| **Network interruption** | Retries with exponential backoff. After max retries, sync fails but cursor is preserved. |
---
## Database Management
### Database Location
The SQLite database is stored at an XDG-compliant location:
```
~/.local/share/gi/data.db
```
This can be overridden in `gi.config.json`:
```json
{
"storage": {
"dbPath": "/custom/path/to/data.db"
}
}
```
### Backup
Create a timestamped backup of the database:
```bash
gi backup
# Creates: ~/.local/share/gi/backups/data-2026-01-21T14-30-00.db
```
Backups are SQLite `.backup` command copies (safe even during active writes due to WAL mode).
### Reset
To completely reset the database and all sync cursors:
```bash
gi reset --confirm
```
This deletes:
- The database file
- All sync cursors
- All embeddings
You'll need to run `gi sync` again to repopulate.
### Schema Migrations
Database schema is version-tracked and migrations auto-apply on startup:
1. On first run, schema is created at latest version
2. On subsequent runs, pending migrations are applied automatically
3. Migration version is stored in `schema_version` table
4. Migrations are idempotent and reversible where possible
**Manual migration check:**
```bash
gi doctor --json | jq '.checks.database'
# Shows: { "status": "ok", "schemaVersion": 5, "pendingMigrations": 0 }
```
---
## Future Work (Post-MVP)
The following features are explicitly deferred to keep MVP scope focused:
| Feature | Description | Depends On |
|---------|-------------|------------|
| **File History** | Query "what decisions were made about src/auth/login.ts?" Requires mr_files table (MR→file linkage), commit-level indexing | MVP complete |
| **Personal Dashboard** | Filter by assigned/mentioned, integrate with gitlab-inbox tool | MVP complete |
| **Person Context** | Aggregate contributions by author, expertise inference | MVP complete |
| **Decision Graph** | LLM-assisted decision extraction, relationship visualization | MVP + LLM integration |
| **MCP Server** | Expose search as MCP tool for Claude Code integration | Checkpoint 4 |
| **Custom Tokenizer** | Better handling of code identifiers (snake_case, paths) | Checkpoint 4 |
**Checkpoint 6 (File History) Schema Preview:**
```sql
-- Deferred from MVP; added when file-history feature is built
CREATE TABLE mr_files (
id INTEGER PRIMARY KEY,
merge_request_id INTEGER REFERENCES merge_requests(id),
old_path TEXT,
new_path TEXT,
new_file BOOLEAN,
deleted_file BOOLEAN,
renamed_file BOOLEAN,
UNIQUE(merge_request_id, old_path, new_path)
);
CREATE INDEX idx_mr_files_old_path ON mr_files(old_path);
CREATE INDEX idx_mr_files_new_path ON mr_files(new_path);
-- DiffNote position data (for "show me comments on this file" queries)
-- Populated from notes.type='DiffNote' position object in GitLab API
CREATE TABLE note_positions (
note_id INTEGER PRIMARY KEY REFERENCES notes(id),
old_path TEXT,
new_path TEXT,
old_line INTEGER,
new_line INTEGER,
position_type TEXT -- 'text' | 'image' | etc.
);
CREATE INDEX idx_note_positions_new_path ON note_positions(position_new_path);
```
---
## Verification Strategy
Each checkpoint includes:
1. **Automated tests** - Unit tests for data transformations, integration tests for API calls
2. **CLI smoke tests** - Manual commands with expected outputs documented
3. **Data integrity checks** - Count verification against GitLab, schema validation
4. **Search quality tests** - Known queries with expected results (for Checkpoint 4+)
---
## Risk Mitigation
| Risk | Mitigation |
|------|------------|
| GitLab rate limiting | Exponential backoff, respect Retry-After headers, incremental sync |
| Embedding model quality | Start with nomic-embed-text; architecture allows model swap |
| SQLite scale limits | Monitor performance; Postgres migration path documented |
| Stale data | Incremental sync with change detection |
| Mid-sync failures | Cursor-based resumption, sync_runs audit trail, heartbeat-based lock recovery |
| Missed updates | Rolling backfill window (14 days), tuple cursor semantics |
| Search quality | Hybrid (vector + FTS5) retrieval with RRF, golden query test suite |
| Concurrent sync corruption | DB lock + heartbeat + rolling backfill, automatic stale lock recovery |
| Embedding failures | Per-document error tracking, retry with backoff, targeted re-runs |
**SQLite Performance Defaults (MVP):**
- Enable `PRAGMA journal_mode=WAL;` on every connection
- Enable `PRAGMA foreign_keys=ON;` on every connection
- Use explicit transactions for page/batch inserts
- Targeted indexes on `(project_id, updated_at)` for primary resources
---
## Schema Summary
| Table | Checkpoint | Purpose |
|-------|------------|---------|
| projects | 0 | Configured GitLab projects |
| sync_runs | 0 | Audit trail of sync operations (with heartbeat) |
| app_locks | 0 | Crash-safe single-flight lock |
| sync_cursors | 0 | Resumable sync state per primary resource |
| raw_payloads | 0 | Decoupled raw JSON storage (with project_id) |
| schema_version | 0 | Database migration version tracking |
| issues | 1 | Normalized issues (unique by project+iid) |
| labels | 1 | Label definitions (unique by project + name) |
| issue_labels | 1 | Issue-label junction |
| merge_requests | 2 | Normalized MRs (unique by project+iid) |
| discussions | 1 | Discussion threads (issue discussions in CP1, MR discussions in CP2) |
| notes | 1 | Individual comments with is_system flag (DiffNote paths added in CP2) |
| mr_labels | 2 | MR-label junction |
| documents | 3 | Unified searchable documents with truncation metadata |
| document_labels | 3 | Document-label junction for fast filtering |
| document_paths | 3 | Fast path filtering for documents (DiffNote file paths) |
| dirty_sources | 3 | Queue for incremental document regeneration |
| embeddings | 3 | Vector embeddings (sqlite-vss, rowid=document_id) |
| embedding_metadata | 3 | Embedding provenance + error tracking |
| documents_fts | 4 | Full-text search index (fts5 with porter stemmer) |
| mr_files | 6 | MR file changes (deferred to File History feature) |
---
## Resolved Decisions
| Question | Decision | Rationale |
|----------|----------|-----------|
| Comments structure | **Discussions as first-class entities** | Thread context is essential for decision traceability |
| System notes | **Store flagged, exclude from embeddings** | Preserves audit trail while avoiding semantic noise |
| DiffNote paths | **Capture now** | Enables immediate file/path search without full file-history feature |
| MR file linkage | **Deferred to post-MVP (CP6)** | Only needed for file-history feature |
| Labels | **Index as filters** | `document_labels` table enables fast `--label=X` filtering |
| Labels uniqueness | **By (project_id, name)** | GitLab API returns labels as strings |
| Sync method | **Polling only for MVP** | Webhooks add complexity; polling every 10 min is sufficient |
| Sync safety | **DB lock + heartbeat + rolling backfill** | Prevents race conditions and missed updates |
| Discussions sync | **Dependent resource model** | Discussions API is per-parent; refetch all when parent updates |
| Hybrid ranking | **RRF over weighted sums** | Simpler, no score normalization needed |
| Embedding rowid | **rowid = documents.id** | Eliminates fragile rowid mapping |
| Embedding truncation | **8000 tokens, truncate middle** | Preserve first/last notes for context |
| Embedding batching | **32 docs/batch, 4 concurrent workers** | Balance throughput, memory, and error isolation |
| FTS5 tokenizer | **porter unicode61** | Stemming improves recall |
| Ollama unavailable | **Graceful degradation to FTS5** | Search still works without semantic matching |
| JSON output | **Stable documented schema** | Enables reliable agent/MCP consumption |
| Database location | **XDG compliant: `~/.local/share/gi/`** | Standard location, user-configurable |
| `gi init` validation | **Validate GitLab before writing config** | Fail fast, better UX |
| Ctrl+C handling | **Graceful shutdown** | Finish page, commit cursor, exit cleanly |
| Empty state UX | **Actionable messages** | Guide user to next step |
---
## Next Steps
1. User approves this spec
2. Generate Checkpoint 0 PRD for project setup
3. Implement Checkpoint 0
4. Human validates → proceed to Checkpoint 1
5. Repeat for each checkpoint
Reference in New Issue View Git Blame Copy Permalink