theirongolddev/gitlore
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Update documentation for search pipeline and Phase A spec

Browse Source 
- README.md: Add hybrid search and robot mode to feature list. Update
  quick start to use new noun-first CLI syntax (lore issues, lore mrs,
  lore search). Add embedding configuration section. Update command
  examples throughout.

- AGENTS.md: Update robot mode examples to new CLI syntax. Add search,
  sync, stats, and generate-docs commands to the robot mode reference.
  Update flag conventions (-n for limit, -s for state, -J for JSON).

- docs/prd/checkpoint-3.md: Major expansion with gated milestone
  structure (Gate A: lexical, Gate B: hybrid, Gate C: sync). Add
  prerequisite rename note, code sample conventions, chunking strategy
  details, and sqlite-vec rowid encoding scheme. Clarify that Gate A
  requires only SQLite + FTS5 with no sqlite-vec dependency.

- docs/phase-a-spec.md: New detailed specification for Gate A (lexical
  search MVP) covering document schema, FTS5 configuration, dirty
  queue mechanics, CLI interface, and acceptance criteria.

- docs/api-efficiency-findings.md: Analysis of GitLab API pagination
  behavior and efficiency observations from production sync runs.
  Documents the missing x-next-page header issue and heuristic fix.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Taylor Eernisse
2026-01-30 15:47:33 -05:00
parent d235f2b4dd
commit 9b63671df9
5 changed files with 1902 additions and 377 deletions
Download Patch File Download Diff File Expand all files Collapse all files

441
README.md
View File

@@ -1,6 +1,6 @@
# Gitlore
Local GitLab data management with semantic search. Syncs issues, MRs, discussions, and notes from GitLab to a local SQLite database for fast, offline-capable querying and filtering.
Local GitLab data management with semantic search. Syncs issues, MRs, discussions, and notes from GitLab to a local SQLite database for fast, offline-capable querying, filtering, and hybrid search.
## Features
@@ -9,8 +9,10 @@ Local GitLab data management with semantic search. Syncs issues, MRs, discussion
- **Full re-sync**: Reset cursors and fetch all data from scratch when needed
- **Multi-project**: Track issues and MRs across multiple GitLab projects
- **Rich filtering**: Filter by state, author, assignee, labels, milestone, due date, draft status, reviewer, branches
- **Hybrid search**: Combines FTS5 lexical search with Ollama-powered vector embeddings via Reciprocal Rank Fusion
- **Raw payload storage**: Preserves original GitLab API responses for debugging
- **Discussion threading**: Full support for issue and MR discussions including inline code review comments
- **Robot mode**: Machine-readable JSON output with structured errors and meaningful exit codes
## Installation
@@ -32,25 +34,28 @@ cargo build --release
lore init
# Verify authentication
lore auth-test
lore auth
# Sync issues from GitLab
lore ingest --type issues
# Sync merge requests from GitLab
lore ingest --type mrs
# Sync everything from GitLab (issues + MRs + docs + embeddings)
lore sync
# List recent issues
lore list issues --limit 10
lore issues -n 10
# List open merge requests
lore list mrs --state opened
lore mrs -s opened
# Show issue details
lore show issue 123 --project group/repo
lore issues 123
# Show MR details with discussions
lore show mr 456 --project group/repo
lore mrs 456
# Search across all indexed data
lore search "authentication bug"
# Robot mode (machine-readable JSON)
lore -J issues -n 5 | jq .
```
## Configuration
@@ -79,6 +84,12 @@ Configuration is stored in `~/.config/lore/config.json` (or `$XDG_CONFIG_HOME/lo
},
"storage": {
"compressRawPayloads": true
},
"embedding": {
"provider": "ollama",
"model": "nomic-embed-text",
"baseUrl": "http://localhost:11434",
"concurrency": 4
}
}
```
@@ -87,9 +98,9 @@ Configuration is stored in `~/.config/lore/config.json` (or `$XDG_CONFIG_HOME/lo
| Section | Field | Default | Description |
|---------|-------|---------|-------------|
| `gitlab` | `baseUrl` | — | GitLab instance URL (required) |
| `gitlab` | `baseUrl` | -- | GitLab instance URL (required) |
| `gitlab` | `tokenEnvVar` | `GITLAB_TOKEN` | Environment variable containing API token |
| `projects` | `path` | — | Project path (e.g., `group/project`) |
| `projects` | `path` | -- | Project path (e.g., `group/project`) |
| `sync` | `backfillDays` | `14` | Days to backfill on initial sync |
| `sync` | `staleLockMinutes` | `10` | Minutes before sync lock considered stale |
| `sync` | `heartbeatIntervalSeconds` | `30` | Frequency of lock heartbeat updates |
@@ -107,7 +118,7 @@ Configuration is stored in `~/.config/lore/config.json` (or `$XDG_CONFIG_HOME/lo
### Config File Resolution
The config file is resolved in this order:
1. `--config` CLI flag
1. `--config` / `-c` CLI flag
2. `LORE_CONFIG_PATH` environment variable
3. `~/.config/lore/config.json` (XDG default)
4. `./lore.config.json` (local fallback for development)
@@ -116,7 +127,7 @@ The config file is resolved in this order:
Create a personal access token with `read_api` scope:
1. Go to GitLab → Settings → Access Tokens
1. Go to GitLab > Settings > Access Tokens
2. Create token with `read_api` scope
3. Export it: `export GITLAB_TOKEN=glpat-xxxxxxxxxxxx`
@@ -126,12 +137,185 @@ Create a personal access token with `read_api` scope:
|----------|---------|----------|
| `GITLAB_TOKEN` | GitLab API authentication token (name configurable via `gitlab.tokenEnvVar`) | Yes |
| `LORE_CONFIG_PATH` | Override config file location | No |
| `LORE_ROBOT` | Enable robot mode globally (set to `true` or `1`) | No |
| `XDG_CONFIG_HOME` | XDG Base Directory for config (fallback: `~/.config`) | No |
| `XDG_DATA_HOME` | XDG Base Directory for data (fallback: `~/.local/share`) | No |
| `RUST_LOG` | Logging level filter (e.g., `lore=debug`) | No |
## Commands
### `lore issues`
Query issues from local database, or show a specific issue.
```bash
lore issues # Recent issues (default 50)
lore issues 123 # Show issue #123 with discussions
lore issues 123 -p group/repo # Disambiguate by project
lore issues -n 100 # More results
lore issues -s opened # Only open issues
lore issues -s closed # Only closed issues
lore issues -a username # By author (@ prefix optional)
lore issues -A username # By assignee (@ prefix optional)
lore issues -l bug # By label (AND logic)
lore issues -l bug -l urgent # Multiple labels
lore issues -m "v1.0" # By milestone title
lore issues --since 7d # Updated in last 7 days
lore issues --since 2w # Updated in last 2 weeks
lore issues --since 2024-01-01 # Updated since date
lore issues --due-before 2024-12-31 # Due before date
lore issues --has-due # Only issues with due dates
lore issues -p group/repo # Filter by project
lore issues --sort created --asc # Sort by created date, ascending
lore issues -o # Open first result in browser
```
When listing, output includes: IID, title, state, author, assignee, labels, and update time.
When showing a single issue (e.g., `lore issues 123`), output includes: title, description, state, author, assignees, labels, milestone, due date, web URL, and threaded discussions.
### `lore mrs`
Query merge requests from local database, or show a specific MR.
```bash
lore mrs # Recent MRs (default 50)
lore mrs 456 # Show MR !456 with discussions
lore mrs 456 -p group/repo # Disambiguate by project
lore mrs -n 100 # More results
lore mrs -s opened # Only open MRs
lore mrs -s merged # Only merged MRs
lore mrs -s closed # Only closed MRs
lore mrs -s locked # Only locked MRs
lore mrs -s all # All states
lore mrs -a username # By author (@ prefix optional)
lore mrs -A username # By assignee (@ prefix optional)
lore mrs -r username # By reviewer (@ prefix optional)
lore mrs -d # Only draft/WIP MRs
lore mrs -D # Exclude draft MRs
lore mrs --target main # By target branch
lore mrs --source feature/foo # By source branch
lore mrs -l needs-review # By label (AND logic)
lore mrs --since 7d # Updated in last 7 days
lore mrs -p group/repo # Filter by project
lore mrs --sort created --asc # Sort by created date, ascending
lore mrs -o # Open first result in browser
```
When listing, output includes: IID, title (with [DRAFT] prefix if applicable), state, author, assignee, labels, and update time.
When showing a single MR (e.g., `lore mrs 456`), output includes: title, description, state, draft status, author, assignees, reviewers, labels, source/target branches, merge status, web URL, and threaded discussions. Inline code review comments (DiffNotes) display file context in the format `[src/file.ts:45]`.
### `lore search`
Search across indexed documents using hybrid (lexical + semantic), lexical-only, or semantic-only modes.
```bash
lore search "authentication bug" # Hybrid search (default)
lore search "login flow" --mode lexical # FTS5 lexical only
lore search "login flow" --mode semantic # Vector similarity only
lore search "auth" --type issue # Filter by source type
lore search "auth" --type mr # MR documents only
lore search "auth" --type discussion # Discussion documents only
lore search "deploy" --author username # Filter by author
lore search "deploy" -p group/repo # Filter by project
lore search "deploy" --label backend # Filter by label (AND logic)
lore search "deploy" --path src/ # Filter by file path (trailing / for prefix)
lore search "deploy" --after 7d # Created after (7d, 2w, or YYYY-MM-DD)
lore search "deploy" --updated-after 2w # Updated after
lore search "deploy" -n 50 # Limit results (default 20, max 100)
lore search "deploy" --explain # Show ranking explanation per result
lore search "deploy" --fts-mode raw # Raw FTS5 query syntax (advanced)
```
Requires `lore generate-docs` (or `lore sync`) to have been run at least once. Semantic mode requires Ollama with the configured embedding model.
### `lore sync`
Run the full sync pipeline: ingest from GitLab, generate searchable documents, and compute embeddings.
```bash
lore sync # Full pipeline
lore sync --full # Reset cursors, fetch everything
lore sync --force # Override stale lock
lore sync --no-embed # Skip embedding step
lore sync --no-docs # Skip document regeneration
```
### `lore ingest`
Sync data from GitLab to local database. Runs only the ingestion step (no doc generation or embeddings).
```bash
lore ingest # Ingest everything (issues + MRs)
lore ingest issues # Issues only
lore ingest mrs # MRs only
lore ingest issues -p group/repo # Single project
lore ingest --force # Override stale lock
lore ingest --full # Full re-sync (reset cursors)
```
The `--full` flag resets sync cursors and discussion watermarks, then fetches all data from scratch. Useful when:
- Assignee data or other fields were missing from earlier syncs
- You want to ensure complete data after schema changes
- Troubleshooting sync issues
### `lore generate-docs`
Extract searchable documents from ingested issues, MRs, and discussions for the FTS5 index.
```bash
lore generate-docs # Incremental (dirty items only)
lore generate-docs --full # Full rebuild
lore generate-docs -p group/repo # Single project
```
### `lore embed`
Generate vector embeddings for documents via Ollama. Requires Ollama running with the configured embedding model.
```bash
lore embed # Embed new/changed documents
lore embed --retry-failed # Retry previously failed embeddings
```
### `lore count`
Count entities in local database.
```bash
lore count issues # Total issues
lore count mrs # Total MRs (with state breakdown)
lore count discussions # Total discussions
lore count discussions --for issue # Issue discussions only
lore count discussions --for mr # MR discussions only
lore count notes # Total notes (system vs user breakdown)
lore count notes --for issue # Issue notes only
```
### `lore stats`
Show document and index statistics, with optional integrity checks.
```bash
lore stats # Document and index statistics
lore stats --check # Run integrity checks
lore stats --check --repair # Repair integrity issues
```
### `lore status`
Show current sync state and watermarks.
```bash
lore status
```
Displays:
- Last sync run details (status, timing)
- Cursor positions per project and resource type (issues and MRs)
- Data summary counts
### `lore init`
Initialize configuration and database interactively.
@@ -142,12 +326,12 @@ lore init --force # Overwrite existing config
lore init --non-interactive # Fail if prompts needed
```
### `lore auth-test`
### `lore auth`
Verify GitLab authentication is working.
```bash
lore auth-test
lore auth
# Authenticated as @username (Full Name)
# GitLab: https://gitlab.com
```
@@ -157,8 +341,7 @@ lore auth-test
Check environment health and configuration.
```bash
lore doctor # Human-readable output
lore doctor --json # JSON output for scripting
lore doctor
```
Checks performed:
@@ -168,132 +351,6 @@ Checks performed:
- Project accessibility
- Ollama connectivity (optional)
### `lore ingest`
Sync data from GitLab to local database.
```bash
# Issues
lore ingest --type issues # Sync all projects
lore ingest --type issues --project group/repo # Single project
lore ingest --type issues --force # Override stale lock
lore ingest --type issues --full # Full re-sync (reset cursors)
# Merge Requests
lore ingest --type mrs # Sync all projects
lore ingest --type mrs --project group/repo # Single project
lore ingest --type mrs --full # Full re-sync (reset cursors)
```
The `--full` flag resets sync cursors and discussion watermarks, then fetches all data from scratch. Useful when:
- Assignee data or other fields were missing from earlier syncs
- You want to ensure complete data after schema changes
- Troubleshooting sync issues
### `lore list issues`
Query issues from local database.
```bash
lore list issues # Recent issues (default 50)
lore list issues --limit 100 # More results
lore list issues --state opened # Only open issues
lore list issues --state closed # Only closed issues
lore list issues --author username # By author (@ prefix optional)
lore list issues --assignee username # By assignee (@ prefix optional)
lore list issues --label bug # By label (AND logic)
lore list issues --label bug --label urgent # Multiple labels
lore list issues --milestone "v1.0" # By milestone title
lore list issues --since 7d # Updated in last 7 days
lore list issues --since 2w # Updated in last 2 weeks
lore list issues --since 2024-01-01 # Updated since date
lore list issues --due-before 2024-12-31 # Due before date
lore list issues --has-due-date # Only issues with due dates
lore list issues --project group/repo # Filter by project
lore list issues --sort created --order asc # Sort options
lore list issues --open # Open first result in browser
lore list issues --json # JSON output
```
Output includes: IID, title, state, author, assignee, labels, and update time.
### `lore list mrs`
Query merge requests from local database.
```bash
lore list mrs # Recent MRs (default 50)
lore list mrs --limit 100 # More results
lore list mrs --state opened # Only open MRs
lore list mrs --state merged # Only merged MRs
lore list mrs --state closed # Only closed MRs
lore list mrs --state locked # Only locked MRs
lore list mrs --state all # All states
lore list mrs --author username # By author (@ prefix optional)
lore list mrs --assignee username # By assignee (@ prefix optional)
lore list mrs --reviewer username # By reviewer (@ prefix optional)
lore list mrs --draft # Only draft/WIP MRs
lore list mrs --no-draft # Exclude draft MRs
lore list mrs --target-branch main # By target branch
lore list mrs --source-branch feature/foo # By source branch
lore list mrs --label needs-review # By label (AND logic)
lore list mrs --since 7d # Updated in last 7 days
lore list mrs --project group/repo # Filter by project
lore list mrs --sort created --order asc # Sort options
lore list mrs --open # Open first result in browser
lore list mrs --json # JSON output
```
Output includes: IID, title (with [DRAFT] prefix if applicable), state, author, assignee, labels, and update time.
### `lore show issue`
Display detailed issue information.
```bash
lore show issue 123 # Show issue #123
lore show issue 123 --project group/repo # Disambiguate if needed
```
Shows: title, description, state, author, assignees, labels, milestone, due date, web URL, and threaded discussions.
### `lore show mr`
Display detailed merge request information.
```bash
lore show mr 456 # Show MR !456
lore show mr 456 --project group/repo # Disambiguate if needed
```
Shows: title, description, state, draft status, author, assignees, reviewers, labels, source/target branches, merge status, web URL, and threaded discussions. Inline code review comments (DiffNotes) display file context in the format `[src/file.ts:45]`.
### `lore count`
Count entities in local database.
```bash
lore count issues # Total issues
lore count mrs # Total MRs (with state breakdown)
lore count discussions # Total discussions
lore count discussions --type issue # Issue discussions only
lore count discussions --type mr # MR discussions only
lore count notes # Total notes (shows system vs user breakdown)
```
### `lore sync-status`
Show current sync state and watermarks.
```bash
lore sync-status
```
Displays:
- Last sync run details (status, timing)
- Cursor positions per project and resource type (issues and MRs)
- Data summary counts
### `lore migrate`
Run pending database migrations.
@@ -302,8 +359,6 @@ Run pending database migrations.
lore migrate
```
Shows current schema version and applies any pending migrations.
### `lore version`
Show version information.
@@ -312,26 +367,67 @@ Show version information.
lore version
```
### `lore backup`
## Robot Mode
Create timestamped database backup.
Machine-readable JSON output for scripting and AI agent consumption.
### Activation
```bash
lore backup
# Global flag
lore --robot issues -n 5
# JSON shorthand (-J)
lore -J issues -n 5
# Environment variable
LORE_ROBOT=1 lore issues -n 5
# Auto-detection (when stdout is not a TTY)
lore issues -n 5 | jq .
```
*Note: Not yet implemented.*
### Response Format
### `lore reset`
All commands return consistent JSON:
Delete database and reset all state.
```json
{"ok": true, "data": {...}, "meta": {...}}
```
Errors return structured JSON to stderr:
```json
{"error": {"code": "CONFIG_NOT_FOUND", "message": "...", "suggestion": "Run 'lore init'"}}
```
### Exit Codes
| Code | Meaning |
|------|---------|
| 0 | Success |
| 1 | Internal error |
| 2 | Config not found |
| 3 | Config invalid |
| 4 | Token not set |
| 5 | GitLab auth failed |
| 6 | Resource not found |
| 7 | Rate limited |
| 8 | Network error |
| 9 | Database locked |
| 10 | Database error |
| 11 | Migration failed |
| 12 | I/O error |
| 13 | Transform error |
## Global Options
```bash
lore reset --confirm
lore -c /path/to/config.json <command> # Use alternate config
lore --robot <command> # Machine-readable JSON
lore -J <command> # JSON shorthand
```
*Note: Not yet implemented.*
## Database Schema
Data is stored in SQLite with WAL mode and foreign keys enabled. Main tables:
@@ -350,6 +446,9 @@ Data is stored in SQLite with WAL mode and foreign keys enabled. Main tables:
| `mr_reviewers` | Many-to-many MR-reviewer relationships |
| `discussions` | Issue/MR discussion threads |
| `notes` | Individual notes within discussions (with system note flag and DiffNote position data) |
| `documents` | Extracted searchable text for FTS and embedding |
| `documents_fts` | FTS5 full-text search index |
| `embeddings` | Vector embeddings for semantic search |
| `sync_runs` | Audit trail of sync operations |
| `sync_cursors` | Cursor positions for incremental sync |
| `app_locks` | Crash-safe single-flight lock |
@@ -358,12 +457,6 @@ Data is stored in SQLite with WAL mode and foreign keys enabled. Main tables:
The database is stored at `~/.local/share/lore/lore.db` by default (XDG compliant).
## Global Options
```bash
lore --config /path/to/config.json <command> # Use alternate config
```
## Development
```bash
@@ -371,10 +464,10 @@ lore --config /path/to/config.json <command> # Use alternate config
cargo test
# Run with debug logging
RUST_LOG=lore=debug lore list issues
RUST_LOG=lore=debug lore issues
# Run with trace logging
RUST_LOG=lore=trace lore ingest --type issues
RUST_LOG=lore=trace lore ingest issues
# Check formatting
cargo fmt --check
@@ -386,7 +479,8 @@ cargo clippy
## Tech Stack
- **Rust** (2024 edition)
- **SQLite** via rusqlite (bundled)
- **SQLite** via rusqlite (bundled) with FTS5 and sqlite-vec
- **Ollama** for vector embeddings (nomic-embed-text)
- **clap** for CLI parsing
- **reqwest** for HTTP
- **tokio** for async runtime
@@ -394,23 +488,6 @@ cargo clippy
- **tracing** for logging
- **indicatif** for progress bars
## Current Status
This is Checkpoint 2 (CP2) of the Gitlore project. Currently implemented:
- Issue ingestion with cursor-based incremental sync
- Merge request ingestion with cursor-based incremental sync
- Discussion and note syncing for issues and MRs
- DiffNote support for inline code review comments
- Rich filtering and querying for both issues and MRs
- Full re-sync capability with watermark reset
Not yet implemented:
- Semantic search with embeddings (CP3+)
- Backup and reset commands
See [SPEC.md](SPEC.md) for the full project roadmap and architecture.
## License
MIT