Gitlore CLI ↔ GitLab API Review

lore <noun> [verb/arg]    # Primary pattern
lore issues              # List all issues
lore issues 42           # Show issue #42
lore mrs                 # List all merge requests
lore mrs 17              # Show MR #17
lore ingest issues       # Fetch issues from GitLab
lore ingest mrs          # Fetch MRs from GitLab
lore count issues        # Count local issues
lore count discussions   # Count local discussions
lore status              # Show sync state
lore auth                # Verify GitLab auth
lore doctor              # Health check
lore init                # Initialize config + DB
lore migrate             # Run DB migrations
lore version             # Show version

Three ways to activate:

lore --robot list issues          # Explicit flag
lore list issues | jq .           # Auto: stdout not a TTY
LORE_ROBOT=1 lore list issues     # Environment variable

Robot mode returns JSON: {"ok":true,"data":{...},"meta":{...}}

Errors go to stderr: {"error":{"code":"...","message":"...","suggestion":"..."}}

Cursor State: (updated_at_cursor: i64, tie_breaker_id: i64)

First sync:
  updated_after = (now - backfillDays)

Subsequent syncs:
  updated_after = cursor.updated_at - cursorRewindSeconds

  For each fetched resource:
    if (gitlab_id, updated_at) <= cursor:
      SKIP (already processed in overlap zone)
    else:
      UPSERT into database

  After each page boundary:
    UPDATE sync_cursors  (crash recovery safe)

For each issue/MR where updated_at > discussions_synced_for_updated_at:

  1. PREFETCH (parallel, configurable concurrency):
     GET /projects/:id/issues/:iid/discussions  (all pages)

  2. WRITE (serial, inside transaction):
     DELETE FROM discussions WHERE issue_id = ?
     DELETE FROM notes WHERE discussion_id IN (...)
     INSERT discussions + notes (fresh data)
     UPDATE issues SET discussions_synced_for_updated_at = updated_at

Full-refresh avoids complexity of detecting deleted/edited notes. Trade-off: more API calls for heavily-discussed items.

RateLimiter {
  min_interval: 100ms  (= 1s / 10 req/s)
  jitter: 0-50ms random

  acquire():
    elapsed = now - last_request
    if elapsed < min_interval:
      sleep(min_interval - elapsed + random_jitter)
    last_request = now
}

Async stream-based. Fallback chain for next-page detection:

1. Link header (RFC 8288) — parse rel="next"
2. x-next-page header — direct page number
3. Full-page heuristic — if response has 100 items, assume more pages

UNIQUE constraint on (project_id, resource_type, gitlab_id, payload_hash) prevents storing identical payloads.

AppLock (database-enforced mutex):
  name: 'sync' (PK)
  owner: UUIDv4 (unique per process)
  heartbeat_at: updated every 30s

  Acquire:
    INSERT OR fail if row exists
    Check stale: if heartbeat > staleLockMinutes, force-acquire

  Release:
    DELETE WHERE owner = my_uuid

Although many fields are dropped during transformation, the raw_payloads table stores the complete original JSON response (with SHA-256 dedup and optional gzip). This means all "dropped" data is still recoverable from the blob storage without re-fetching from GitLab. The normalized tables are optimized for query patterns, not completeness.

Every time an issue/MR is updated, ALL its discussions are re-fetched and replaced (DELETE + INSERT). For heavily-discussed items (50+ comments), this is expensive.

Trade-off: Full-refresh is simpler and guarantees consistency (catches edits, deletes). Incremental would miss deleted notes.

Gitlore uses page=N&per_page=100 offset pagination. GitLab supports keyset pagination for some endpoints (Issues, MRs), which is more efficient for large datasets and recommended by GitLab.

Current:  GET /projects/:id/issues?page=5&per_page=100
Keyset:   GET /projects/:id/issues?pagination=keyset&per_page=100
          (uses Link header rel="next" with cursor)

Benefit: Keyset pagination is O(1) per page (vs O(N) for offset). GitLab recommends it for >10,000 records. Gitlore already parses Link headers, so the client-side support partially exists.

GitLab returns ETag headers on API responses. Sending If-None-Match on subsequent requests would return 304 Not Modified without consuming rate limit quota on some endpoints. Currently all requests are unconditional.

Impact: Moderate. The cursor-based sync already avoids re-fetching unchanged data, so ETag would mainly help with the discussions full-refresh scenario where nothing changed.

Gitlore extracts labels from the labels[] string array embedded in issue/MR responses. The dedicated GET /projects/:id/labels endpoint returns richer data:

Impact: The labels table has color and description columns but they may not be populated from the embedded string array. A single Labels API call (one request, non-paginated for most projects) would enrich the local label catalog.

Currently stores only username for authors, assignees, reviewers. The API returns name, avatar_url, web_url, and state for every user reference. A users table could deduplicate this data and provide richer displays.

-- Potential schema
CREATE TABLE users (
  username TEXT PRIMARY KEY,
  name TEXT,
  gitlab_id INTEGER,
  avatar_url TEXT,
  state TEXT,           -- "active", "blocked", etc.
  last_seen_at INTEGER  -- auto-updated on encounter
);

Cost: No additional API calls. Data is already in every issue/MR/note response. Just needs extraction during transform.

Milestones are stored for issues but the MR transformer does not extract the milestone object from MR responses, even though GitLab returns it. The merge_requests table has no milestone_id column.

Impact: Cannot query "which MRs are in milestone X?" locally. The data is in the raw payload but not indexed.

MRs store references.short and references.full, but the issue transformer drops the references object entirely. This means issues lack the cross-project reference format (e.g., group/project#42).

Instead of polling, GitLab can push events via POST /projects/:id/hooks. Would enable near-real-time sync without rate-limit cost. Requires a listener endpoint.

GET /projects/:id/events returns a stream of all project activity. Could be used as a fast "has anything changed?" check before running expensive issue/MR sync. Much lighter than fetching full issue lists.

GitLab's GraphQL API allows requesting exactly the fields needed. Would eliminate bandwidth waste from ~50% of response fields being silently dropped. Trade-off: different pagination model, potentially less stable API surface.

Gitlore is well-optimized for its core use case (read-only local sync). The cursor-based incremental sync and raw payload archival are sophisticated. The main opportunities are:

1. Capture more "free" data — Fields like user_notes_count, upvotes, has_conflicts are already in API responses. Storing them costs zero API calls and enables richer queries.
2. Discussion sync efficiency — The full-refresh strategy is the biggest source of redundant API calls. Even a simple user_notes_count comparison could skip unchanged discussions.
3. Keyset pagination — A meaningful improvement for large projects (>10K issues), and Gitlore already has partial infrastructure for it.
4. MR milestone parity — Low-effort gap to close with issue milestone support.