417 lines
13 KiB
Markdown
417 lines
13 KiB
Markdown
# Gitlore
|
|
|
|
Local GitLab data management with semantic search. Syncs issues, MRs, discussions, and notes from GitLab to a local SQLite database for fast, offline-capable querying and filtering.
|
|
|
|
## Features
|
|
|
|
- **Local-first**: All data stored in SQLite for instant queries
|
|
- **Incremental sync**: Cursor-based sync only fetches changes since last sync
|
|
- **Full re-sync**: Reset cursors and fetch all data from scratch when needed
|
|
- **Multi-project**: Track issues and MRs across multiple GitLab projects
|
|
- **Rich filtering**: Filter by state, author, assignee, labels, milestone, due date, draft status, reviewer, branches
|
|
- **Raw payload storage**: Preserves original GitLab API responses for debugging
|
|
- **Discussion threading**: Full support for issue and MR discussions including inline code review comments
|
|
|
|
## Installation
|
|
|
|
```bash
|
|
cargo install --path .
|
|
```
|
|
|
|
Or build from source:
|
|
|
|
```bash
|
|
cargo build --release
|
|
./target/release/lore --help
|
|
```
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Initialize configuration (interactive)
|
|
lore init
|
|
|
|
# Verify authentication
|
|
lore auth-test
|
|
|
|
# Sync issues from GitLab
|
|
lore ingest --type issues
|
|
|
|
# Sync merge requests from GitLab
|
|
lore ingest --type mrs
|
|
|
|
# List recent issues
|
|
lore list issues --limit 10
|
|
|
|
# List open merge requests
|
|
lore list mrs --state opened
|
|
|
|
# Show issue details
|
|
lore show issue 123 --project group/repo
|
|
|
|
# Show MR details with discussions
|
|
lore show mr 456 --project group/repo
|
|
```
|
|
|
|
## Configuration
|
|
|
|
Configuration is stored in `~/.config/lore/config.json` (or `$XDG_CONFIG_HOME/lore/config.json`).
|
|
|
|
### Example Configuration
|
|
|
|
```json
|
|
{
|
|
"gitlab": {
|
|
"baseUrl": "https://gitlab.com",
|
|
"tokenEnvVar": "GITLAB_TOKEN"
|
|
},
|
|
"projects": [
|
|
{ "path": "group/project" },
|
|
{ "path": "other-group/other-project" }
|
|
],
|
|
"sync": {
|
|
"backfillDays": 14,
|
|
"staleLockMinutes": 10,
|
|
"heartbeatIntervalSeconds": 30,
|
|
"cursorRewindSeconds": 2,
|
|
"primaryConcurrency": 4,
|
|
"dependentConcurrency": 2
|
|
},
|
|
"storage": {
|
|
"compressRawPayloads": true
|
|
}
|
|
}
|
|
```
|
|
|
|
### Configuration Options
|
|
|
|
| Section | Field | Default | Description |
|
|
|---------|-------|---------|-------------|
|
|
| `gitlab` | `baseUrl` | — | GitLab instance URL (required) |
|
|
| `gitlab` | `tokenEnvVar` | `GITLAB_TOKEN` | Environment variable containing API token |
|
|
| `projects` | `path` | — | Project path (e.g., `group/project`) |
|
|
| `sync` | `backfillDays` | `14` | Days to backfill on initial sync |
|
|
| `sync` | `staleLockMinutes` | `10` | Minutes before sync lock considered stale |
|
|
| `sync` | `heartbeatIntervalSeconds` | `30` | Frequency of lock heartbeat updates |
|
|
| `sync` | `cursorRewindSeconds` | `2` | Seconds to rewind cursor for overlap safety |
|
|
| `sync` | `primaryConcurrency` | `4` | Concurrent GitLab requests for primary resources |
|
|
| `sync` | `dependentConcurrency` | `2` | Concurrent requests for dependent resources |
|
|
| `storage` | `dbPath` | `~/.local/share/lore/lore.db` | Database file path |
|
|
| `storage` | `backupDir` | `~/.local/share/lore/backups` | Backup directory |
|
|
| `storage` | `compressRawPayloads` | `true` | Compress stored API responses with gzip |
|
|
| `embedding` | `provider` | `ollama` | Embedding provider |
|
|
| `embedding` | `model` | `nomic-embed-text` | Model name for embeddings |
|
|
| `embedding` | `baseUrl` | `http://localhost:11434` | Ollama server URL |
|
|
| `embedding` | `concurrency` | `4` | Concurrent embedding requests |
|
|
|
|
### Config File Resolution
|
|
|
|
The config file is resolved in this order:
|
|
1. `--config` CLI flag
|
|
2. `LORE_CONFIG_PATH` environment variable
|
|
3. `~/.config/lore/config.json` (XDG default)
|
|
4. `./lore.config.json` (local fallback for development)
|
|
|
|
### GitLab Token
|
|
|
|
Create a personal access token with `read_api` scope:
|
|
|
|
1. Go to GitLab → Settings → Access Tokens
|
|
2. Create token with `read_api` scope
|
|
3. Export it: `export GITLAB_TOKEN=glpat-xxxxxxxxxxxx`
|
|
|
|
## Environment Variables
|
|
|
|
| Variable | Purpose | Required |
|
|
|----------|---------|----------|
|
|
| `GITLAB_TOKEN` | GitLab API authentication token (name configurable via `gitlab.tokenEnvVar`) | Yes |
|
|
| `LORE_CONFIG_PATH` | Override config file location | No |
|
|
| `XDG_CONFIG_HOME` | XDG Base Directory for config (fallback: `~/.config`) | No |
|
|
| `XDG_DATA_HOME` | XDG Base Directory for data (fallback: `~/.local/share`) | No |
|
|
| `RUST_LOG` | Logging level filter (e.g., `lore=debug`) | No |
|
|
|
|
## Commands
|
|
|
|
### `lore init`
|
|
|
|
Initialize configuration and database interactively.
|
|
|
|
```bash
|
|
lore init # Interactive setup
|
|
lore init --force # Overwrite existing config
|
|
lore init --non-interactive # Fail if prompts needed
|
|
```
|
|
|
|
### `lore auth-test`
|
|
|
|
Verify GitLab authentication is working.
|
|
|
|
```bash
|
|
lore auth-test
|
|
# Authenticated as @username (Full Name)
|
|
# GitLab: https://gitlab.com
|
|
```
|
|
|
|
### `lore doctor`
|
|
|
|
Check environment health and configuration.
|
|
|
|
```bash
|
|
lore doctor # Human-readable output
|
|
lore doctor --json # JSON output for scripting
|
|
```
|
|
|
|
Checks performed:
|
|
- Config file existence and validity
|
|
- Database existence and pragmas (WAL mode, foreign keys)
|
|
- GitLab authentication
|
|
- Project accessibility
|
|
- Ollama connectivity (optional)
|
|
|
|
### `lore ingest`
|
|
|
|
Sync data from GitLab to local database.
|
|
|
|
```bash
|
|
# Issues
|
|
lore ingest --type issues # Sync all projects
|
|
lore ingest --type issues --project group/repo # Single project
|
|
lore ingest --type issues --force # Override stale lock
|
|
lore ingest --type issues --full # Full re-sync (reset cursors)
|
|
|
|
# Merge Requests
|
|
lore ingest --type mrs # Sync all projects
|
|
lore ingest --type mrs --project group/repo # Single project
|
|
lore ingest --type mrs --full # Full re-sync (reset cursors)
|
|
```
|
|
|
|
The `--full` flag resets sync cursors and discussion watermarks, then fetches all data from scratch. Useful when:
|
|
- Assignee data or other fields were missing from earlier syncs
|
|
- You want to ensure complete data after schema changes
|
|
- Troubleshooting sync issues
|
|
|
|
### `lore list issues`
|
|
|
|
Query issues from local database.
|
|
|
|
```bash
|
|
lore list issues # Recent issues (default 50)
|
|
lore list issues --limit 100 # More results
|
|
lore list issues --state opened # Only open issues
|
|
lore list issues --state closed # Only closed issues
|
|
lore list issues --author username # By author (@ prefix optional)
|
|
lore list issues --assignee username # By assignee (@ prefix optional)
|
|
lore list issues --label bug # By label (AND logic)
|
|
lore list issues --label bug --label urgent # Multiple labels
|
|
lore list issues --milestone "v1.0" # By milestone title
|
|
lore list issues --since 7d # Updated in last 7 days
|
|
lore list issues --since 2w # Updated in last 2 weeks
|
|
lore list issues --since 2024-01-01 # Updated since date
|
|
lore list issues --due-before 2024-12-31 # Due before date
|
|
lore list issues --has-due-date # Only issues with due dates
|
|
lore list issues --project group/repo # Filter by project
|
|
lore list issues --sort created --order asc # Sort options
|
|
lore list issues --open # Open first result in browser
|
|
lore list issues --json # JSON output
|
|
```
|
|
|
|
Output includes: IID, title, state, author, assignee, labels, and update time.
|
|
|
|
### `lore list mrs`
|
|
|
|
Query merge requests from local database.
|
|
|
|
```bash
|
|
lore list mrs # Recent MRs (default 50)
|
|
lore list mrs --limit 100 # More results
|
|
lore list mrs --state opened # Only open MRs
|
|
lore list mrs --state merged # Only merged MRs
|
|
lore list mrs --state closed # Only closed MRs
|
|
lore list mrs --state locked # Only locked MRs
|
|
lore list mrs --state all # All states
|
|
lore list mrs --author username # By author (@ prefix optional)
|
|
lore list mrs --assignee username # By assignee (@ prefix optional)
|
|
lore list mrs --reviewer username # By reviewer (@ prefix optional)
|
|
lore list mrs --draft # Only draft/WIP MRs
|
|
lore list mrs --no-draft # Exclude draft MRs
|
|
lore list mrs --target-branch main # By target branch
|
|
lore list mrs --source-branch feature/foo # By source branch
|
|
lore list mrs --label needs-review # By label (AND logic)
|
|
lore list mrs --since 7d # Updated in last 7 days
|
|
lore list mrs --project group/repo # Filter by project
|
|
lore list mrs --sort created --order asc # Sort options
|
|
lore list mrs --open # Open first result in browser
|
|
lore list mrs --json # JSON output
|
|
```
|
|
|
|
Output includes: IID, title (with [DRAFT] prefix if applicable), state, author, assignee, labels, and update time.
|
|
|
|
### `lore show issue`
|
|
|
|
Display detailed issue information.
|
|
|
|
```bash
|
|
lore show issue 123 # Show issue #123
|
|
lore show issue 123 --project group/repo # Disambiguate if needed
|
|
```
|
|
|
|
Shows: title, description, state, author, assignees, labels, milestone, due date, web URL, and threaded discussions.
|
|
|
|
### `lore show mr`
|
|
|
|
Display detailed merge request information.
|
|
|
|
```bash
|
|
lore show mr 456 # Show MR !456
|
|
lore show mr 456 --project group/repo # Disambiguate if needed
|
|
```
|
|
|
|
Shows: title, description, state, draft status, author, assignees, reviewers, labels, source/target branches, merge status, web URL, and threaded discussions. Inline code review comments (DiffNotes) display file context in the format `[src/file.ts:45]`.
|
|
|
|
### `lore count`
|
|
|
|
Count entities in local database.
|
|
|
|
```bash
|
|
lore count issues # Total issues
|
|
lore count mrs # Total MRs (with state breakdown)
|
|
lore count discussions # Total discussions
|
|
lore count discussions --type issue # Issue discussions only
|
|
lore count discussions --type mr # MR discussions only
|
|
lore count notes # Total notes (shows system vs user breakdown)
|
|
```
|
|
|
|
### `lore sync-status`
|
|
|
|
Show current sync state and watermarks.
|
|
|
|
```bash
|
|
lore sync-status
|
|
```
|
|
|
|
Displays:
|
|
- Last sync run details (status, timing)
|
|
- Cursor positions per project and resource type (issues and MRs)
|
|
- Data summary counts
|
|
|
|
### `lore migrate`
|
|
|
|
Run pending database migrations.
|
|
|
|
```bash
|
|
lore migrate
|
|
```
|
|
|
|
Shows current schema version and applies any pending migrations.
|
|
|
|
### `lore version`
|
|
|
|
Show version information.
|
|
|
|
```bash
|
|
lore version
|
|
```
|
|
|
|
### `lore backup`
|
|
|
|
Create timestamped database backup.
|
|
|
|
```bash
|
|
lore backup
|
|
```
|
|
|
|
*Note: Not yet implemented.*
|
|
|
|
### `lore reset`
|
|
|
|
Delete database and reset all state.
|
|
|
|
```bash
|
|
lore reset --confirm
|
|
```
|
|
|
|
*Note: Not yet implemented.*
|
|
|
|
## Database Schema
|
|
|
|
Data is stored in SQLite with WAL mode and foreign keys enabled. Main tables:
|
|
|
|
| Table | Purpose |
|
|
|-------|---------|
|
|
| `projects` | Tracked GitLab projects with metadata |
|
|
| `issues` | Issue metadata (title, state, author, due date, milestone) |
|
|
| `merge_requests` | MR metadata (title, state, draft, branches, merge status) |
|
|
| `milestones` | Project milestones with state and due dates |
|
|
| `labels` | Project labels with colors |
|
|
| `issue_labels` | Many-to-many issue-label relationships |
|
|
| `issue_assignees` | Many-to-many issue-assignee relationships |
|
|
| `mr_labels` | Many-to-many MR-label relationships |
|
|
| `mr_assignees` | Many-to-many MR-assignee relationships |
|
|
| `mr_reviewers` | Many-to-many MR-reviewer relationships |
|
|
| `discussions` | Issue/MR discussion threads |
|
|
| `notes` | Individual notes within discussions (with system note flag and DiffNote position data) |
|
|
| `sync_runs` | Audit trail of sync operations |
|
|
| `sync_cursors` | Cursor positions for incremental sync |
|
|
| `app_locks` | Crash-safe single-flight lock |
|
|
| `raw_payloads` | Compressed original API responses |
|
|
| `schema_version` | Migration version tracking |
|
|
|
|
The database is stored at `~/.local/share/lore/lore.db` by default (XDG compliant).
|
|
|
|
## Global Options
|
|
|
|
```bash
|
|
lore --config /path/to/config.json <command> # Use alternate config
|
|
```
|
|
|
|
## Development
|
|
|
|
```bash
|
|
# Run tests
|
|
cargo test
|
|
|
|
# Run with debug logging
|
|
RUST_LOG=lore=debug lore list issues
|
|
|
|
# Run with trace logging
|
|
RUST_LOG=lore=trace lore ingest --type issues
|
|
|
|
# Check formatting
|
|
cargo fmt --check
|
|
|
|
# Lint
|
|
cargo clippy
|
|
```
|
|
|
|
## Tech Stack
|
|
|
|
- **Rust** (2024 edition)
|
|
- **SQLite** via rusqlite (bundled)
|
|
- **clap** for CLI parsing
|
|
- **reqwest** for HTTP
|
|
- **tokio** for async runtime
|
|
- **serde** for serialization
|
|
- **tracing** for logging
|
|
- **indicatif** for progress bars
|
|
|
|
## Current Status
|
|
|
|
This is Checkpoint 2 (CP2) of the Gitlore project. Currently implemented:
|
|
|
|
- Issue ingestion with cursor-based incremental sync
|
|
- Merge request ingestion with cursor-based incremental sync
|
|
- Discussion and note syncing for issues and MRs
|
|
- DiffNote support for inline code review comments
|
|
- Rich filtering and querying for both issues and MRs
|
|
- Full re-sync capability with watermark reset
|
|
|
|
Not yet implemented:
|
|
- Semantic search with embeddings (CP3+)
|
|
- Backup and reset commands
|
|
|
|
See [SPEC.md](SPEC.md) for the full project roadmap and architecture.
|
|
|
|
## License
|
|
|
|
MIT
|