# Gitlore

Local GitLab data management with semantic search. Syncs issues, MRs, discussions, and notes from GitLab to a local SQLite database for fast, offline-capable querying and filtering.

## Features

- **Local-first**: All data stored in SQLite for instant queries
- **Incremental sync**: Cursor-based sync only fetches changes since last sync
- **Full re-sync**: Reset cursors and fetch all data from scratch when needed
- **Multi-project**: Track issues and MRs across multiple GitLab projects
- **Rich filtering**: Filter by state, author, assignee, labels, milestone, due date, draft status, reviewer, branches
- **Raw payload storage**: Preserves original GitLab API responses for debugging
- **Discussion threading**: Full support for issue and MR discussions including inline code review comments

## Installation

```bash
cargo install --path .
```

Or build from source:

```bash
cargo build --release
./target/release/lore --help
```

## Quick Start

```bash
# Initialize configuration (interactive)
lore init

# Verify authentication
lore auth-test

# Sync issues from GitLab
lore ingest --type issues

# Sync merge requests from GitLab
lore ingest --type mrs

# List recent issues
lore list issues --limit 10

# List open merge requests
lore list mrs --state opened

# Show issue details
lore show issue 123 --project group/repo

# Show MR details with discussions
lore show mr 456 --project group/repo
```

## Configuration

Configuration is stored in `~/.config/lore/config.json` (or `$XDG_CONFIG_HOME/lore/config.json`).

### Example Configuration

```json
{
  "gitlab": {
    "baseUrl": "https://gitlab.com",
    "tokenEnvVar": "GITLAB_TOKEN"
  },
  "projects": [
    { "path": "group/project" },
    { "path": "other-group/other-project" }
  ],
  "sync": {
    "backfillDays": 14,
    "staleLockMinutes": 10,
    "heartbeatIntervalSeconds": 30,
    "cursorRewindSeconds": 2,
    "primaryConcurrency": 4,
    "dependentConcurrency": 2
  },
  "storage": {
    "compressRawPayloads": true
  }
}
```

### Configuration Options

| Section | Field | Default | Description |
|---------|-------|---------|-------------|
| `gitlab` | `baseUrl` | — | GitLab instance URL (required) |
| `gitlab` | `tokenEnvVar` | `GITLAB_TOKEN` | Environment variable containing API token |
| `projects` | `path` | — | Project path (e.g., `group/project`) |
| `sync` | `backfillDays` | `14` | Days to backfill on initial sync |
| `sync` | `staleLockMinutes` | `10` | Minutes before sync lock considered stale |
| `sync` | `heartbeatIntervalSeconds` | `30` | Frequency of lock heartbeat updates |
| `sync` | `cursorRewindSeconds` | `2` | Seconds to rewind cursor for overlap safety |
| `sync` | `primaryConcurrency` | `4` | Concurrent GitLab requests for primary resources |
| `sync` | `dependentConcurrency` | `2` | Concurrent requests for dependent resources |
| `storage` | `dbPath` | `~/.local/share/lore/lore.db` | Database file path |
| `storage` | `backupDir` | `~/.local/share/lore/backups` | Backup directory |
| `storage` | `compressRawPayloads` | `true` | Compress stored API responses with gzip |
| `embedding` | `provider` | `ollama` | Embedding provider |
| `embedding` | `model` | `nomic-embed-text` | Model name for embeddings |
| `embedding` | `baseUrl` | `http://localhost:11434` | Ollama server URL |
| `embedding` | `concurrency` | `4` | Concurrent embedding requests |

### Config File Resolution

The config file is resolved in this order:
1. `--config` CLI flag
2. `LORE_CONFIG_PATH` environment variable
3. `~/.config/lore/config.json` (XDG default)
4. `./lore.config.json` (local fallback for development)

### GitLab Token

Create a personal access token with `read_api` scope:

1. Go to GitLab → Settings → Access Tokens
2. Create token with `read_api` scope
3. Export it: `export GITLAB_TOKEN=glpat-xxxxxxxxxxxx`

## Environment Variables

| Variable | Purpose | Required |
|----------|---------|----------|
| `GITLAB_TOKEN` | GitLab API authentication token (name configurable via `gitlab.tokenEnvVar`) | Yes |
| `LORE_CONFIG_PATH` | Override config file location | No |
| `XDG_CONFIG_HOME` | XDG Base Directory for config (fallback: `~/.config`) | No |
| `XDG_DATA_HOME` | XDG Base Directory for data (fallback: `~/.local/share`) | No |
| `RUST_LOG` | Logging level filter (e.g., `lore=debug`) | No |

## Commands

### `lore init`

Initialize configuration and database interactively.

```bash
lore init                    # Interactive setup
lore init --force            # Overwrite existing config
lore init --non-interactive  # Fail if prompts needed
```

### `lore auth-test`

Verify GitLab authentication is working.

```bash
lore auth-test
# Authenticated as @username (Full Name)
# GitLab: https://gitlab.com
```

### `lore doctor`

Check environment health and configuration.

```bash
lore doctor          # Human-readable output
lore doctor --json   # JSON output for scripting
```

Checks performed:
- Config file existence and validity
- Database existence and pragmas (WAL mode, foreign keys)
- GitLab authentication
- Project accessibility
- Ollama connectivity (optional)

### `lore ingest`

Sync data from GitLab to local database.

```bash
# Issues
lore ingest --type issues                       # Sync all projects
lore ingest --type issues --project group/repo  # Single project
lore ingest --type issues --force               # Override stale lock
lore ingest --type issues --full                # Full re-sync (reset cursors)

# Merge Requests
lore ingest --type mrs                          # Sync all projects
lore ingest --type mrs --project group/repo     # Single project
lore ingest --type mrs --full                   # Full re-sync (reset cursors)
```

The `--full` flag resets sync cursors and discussion watermarks, then fetches all data from scratch. Useful when:
- Assignee data or other fields were missing from earlier syncs
- You want to ensure complete data after schema changes
- Troubleshooting sync issues

### `lore list issues`

Query issues from local database.

```bash
lore list issues                              # Recent issues (default 50)
lore list issues --limit 100                  # More results
lore list issues --state opened               # Only open issues
lore list issues --state closed               # Only closed issues
lore list issues --author username            # By author (@ prefix optional)
lore list issues --assignee username          # By assignee (@ prefix optional)
lore list issues --label bug                  # By label (AND logic)
lore list issues --label bug --label urgent   # Multiple labels
lore list issues --milestone "v1.0"           # By milestone title
lore list issues --since 7d                   # Updated in last 7 days
lore list issues --since 2w                   # Updated in last 2 weeks
lore list issues --since 2024-01-01           # Updated since date
lore list issues --due-before 2024-12-31      # Due before date
lore list issues --has-due-date               # Only issues with due dates
lore list issues --project group/repo         # Filter by project
lore list issues --sort created --order asc   # Sort options
lore list issues --open                       # Open first result in browser
lore list issues --json                       # JSON output
```

Output includes: IID, title, state, author, assignee, labels, and update time.

### `lore list mrs`

Query merge requests from local database.

```bash
lore list mrs                                 # Recent MRs (default 50)
lore list mrs --limit 100                     # More results
lore list mrs --state opened                  # Only open MRs
lore list mrs --state merged                  # Only merged MRs
lore list mrs --state closed                  # Only closed MRs
lore list mrs --state locked                  # Only locked MRs
lore list mrs --state all                     # All states
lore list mrs --author username               # By author (@ prefix optional)
lore list mrs --assignee username             # By assignee (@ prefix optional)
lore list mrs --reviewer username             # By reviewer (@ prefix optional)
lore list mrs --draft                         # Only draft/WIP MRs
lore list mrs --no-draft                      # Exclude draft MRs
lore list mrs --target-branch main            # By target branch
lore list mrs --source-branch feature/foo     # By source branch
lore list mrs --label needs-review            # By label (AND logic)
lore list mrs --since 7d                      # Updated in last 7 days
lore list mrs --project group/repo            # Filter by project
lore list mrs --sort created --order asc      # Sort options
lore list mrs --open                          # Open first result in browser
lore list mrs --json                          # JSON output
```

Output includes: IID, title (with [DRAFT] prefix if applicable), state, author, assignee, labels, and update time.

### `lore show issue`

Display detailed issue information.

```bash
lore show issue 123                      # Show issue #123
lore show issue 123 --project group/repo # Disambiguate if needed
```

Shows: title, description, state, author, assignees, labels, milestone, due date, web URL, and threaded discussions.

### `lore show mr`

Display detailed merge request information.

```bash
lore show mr 456                         # Show MR !456
lore show mr 456 --project group/repo    # Disambiguate if needed
```

Shows: title, description, state, draft status, author, assignees, reviewers, labels, source/target branches, merge status, web URL, and threaded discussions. Inline code review comments (DiffNotes) display file context in the format `[src/file.ts:45]`.

### `lore count`

Count entities in local database.

```bash
lore count issues                    # Total issues
lore count mrs                       # Total MRs (with state breakdown)
lore count discussions               # Total discussions
lore count discussions --type issue  # Issue discussions only
lore count discussions --type mr     # MR discussions only
lore count notes                     # Total notes (shows system vs user breakdown)
```

### `lore sync-status`

Show current sync state and watermarks.

```bash
lore sync-status
```

Displays:
- Last sync run details (status, timing)
- Cursor positions per project and resource type (issues and MRs)
- Data summary counts

### `lore migrate`

Run pending database migrations.

```bash
lore migrate
```

Shows current schema version and applies any pending migrations.

### `lore version`

Show version information.

```bash
lore version
```

### `lore backup`

Create timestamped database backup.

```bash
lore backup
```

*Note: Not yet implemented.*

### `lore reset`

Delete database and reset all state.

```bash
lore reset --confirm
```

*Note: Not yet implemented.*

## Database Schema

Data is stored in SQLite with WAL mode and foreign keys enabled. Main tables:

| Table | Purpose |
|-------|---------|
| `projects` | Tracked GitLab projects with metadata |
| `issues` | Issue metadata (title, state, author, due date, milestone) |
| `merge_requests` | MR metadata (title, state, draft, branches, merge status) |
| `milestones` | Project milestones with state and due dates |
| `labels` | Project labels with colors |
| `issue_labels` | Many-to-many issue-label relationships |
| `issue_assignees` | Many-to-many issue-assignee relationships |
| `mr_labels` | Many-to-many MR-label relationships |
| `mr_assignees` | Many-to-many MR-assignee relationships |
| `mr_reviewers` | Many-to-many MR-reviewer relationships |
| `discussions` | Issue/MR discussion threads |
| `notes` | Individual notes within discussions (with system note flag and DiffNote position data) |
| `sync_runs` | Audit trail of sync operations |
| `sync_cursors` | Cursor positions for incremental sync |
| `app_locks` | Crash-safe single-flight lock |
| `raw_payloads` | Compressed original API responses |
| `schema_version` | Migration version tracking |

The database is stored at `~/.local/share/lore/lore.db` by default (XDG compliant).

## Global Options

```bash
lore --config /path/to/config.json <command>  # Use alternate config
```

## Development

```bash
# Run tests
cargo test

# Run with debug logging
RUST_LOG=lore=debug lore list issues

# Run with trace logging
RUST_LOG=lore=trace lore ingest --type issues

# Check formatting
cargo fmt --check

# Lint
cargo clippy
```

## Tech Stack

- **Rust** (2024 edition)
- **SQLite** via rusqlite (bundled)
- **clap** for CLI parsing
- **reqwest** for HTTP
- **tokio** for async runtime
- **serde** for serialization
- **tracing** for logging
- **indicatif** for progress bars

## Current Status

This is Checkpoint 2 (CP2) of the Gitlore project. Currently implemented:

- Issue ingestion with cursor-based incremental sync
- Merge request ingestion with cursor-based incremental sync
- Discussion and note syncing for issues and MRs
- DiffNote support for inline code review comments
- Rich filtering and querying for both issues and MRs
- Full re-sync capability with watermark reset

Not yet implemented:
- Semantic search with embeddings (CP3+)
- Backup and reset commands

See [SPEC.md](SPEC.md) for the full project roadmap and architecture.

## License

MIT