Phase A: Complete API Field Capture

Migration 007 — Mirror all GitLab API response data into the local DB
No new API calls No new tables 1 migration 6 files touched Independent of CP3
Overview
Issues
Merge Requests
Migration SQL
Serde Structs
Transformers
Files Touched
Decisions
Guiding Principle: Mirror everything GitLab gives us. The local DB should be a complete representation of all data returned by the API. This ensures maximum context for processing and analysis in later steps.
0
New columns total
0
New issue columns
0
New MR columns
6
New serde helper types
2
Fields excluded
6
Files touched

Issues table

18 existing 0 new

Merge Requests table

29 existing 0 new
Scope

One migration. Three categories of work:

  1. New columns on issues and merge_requests for fields currently dropped by serde or during transform
  2. New serde fields on GitLabIssue and GitLabMergeRequest to deserialize currently-silently-dropped JSON fields
  3. Transformer + insert updates to pass the new fields through to the DB
What this does NOT include
  • No new API endpoints called
  • No new tables (except reusing existing milestones for MRs)
  • No CLI changes (new fields stored but not surfaced in lore issues / lore mrs)
  • No changes to discussion/note ingestion (Phase A is issues + MRs only)
  • No observability instrumentation (that's Phase B)

Issues: Field Gap Inventory

Currently Stored
id, iid, project_id, title, description, state, author_username, created_at, updated_at, web_url, due_date, milestone_id, milestone_title, raw_payload_id, last_seen_at, discussions_synced_for_updated_at, labels (junction), assignees (junction)
API Field Type DB Column Category Status Notes

Merge Requests: Field Gap Inventory

Currently Stored
id, iid, project_id, title, description, state, draft, author_username, source_branch, target_branch, head_sha, references_short, references_full, detailed_merge_status, merge_user_username, created_at, updated_at, merged_at, closed_at, last_seen_at, web_url, raw_payload_id, discussions_synced_for_updated_at, labels (junction), assignees (junction), reviewers (junction)
API Field Type DB Column Category Status Notes

Migration 007: complete_field_capture.sql

Issues columns
Merge Requests columns

Serde Struct Changes

New Helper Types
GitLabIssue: new fields
GitLabMergeRequest: new fields

Transformer Changes

IssueRow: transform rules

All new fields map 1:1 from the serde struct except these special cases:

NormalizedMergeRequest: transform rules

Same patterns as issues, plus:

Insert statement changes
Both process_issue_in_transaction and process_mr_in_transaction need their INSERT and ON CONFLICT DO UPDATE statements extended with all new columns. The ON CONFLICT clause should update all new fields on re-sync.

Files Touched

Resolved Decisions

Exclusions (2 fields)
subscribed — Excluded
User-relative field. Reflects the token holder's subscription state, not a property of the entity itself. Changes meaning if the token is rotated to a different user. Not entity data.
_links — Excluded
HATEOAS API navigation metadata, not entity data. Every URL is deterministically constructable from project_id + iid + GitLab base URL.
Note: closed_as_duplicate_of inside _links contains a real entity reference. Extracting that is deferred to a future phase.
Included with special handling
epic / iteration — Flatten to columns
Same denormalization pattern as milestones. Epic gets 5 columns (epic_id, epic_iid, epic_title, epic_url, epic_group_id). Iteration gets 6 columns (iteration_id, iteration_iid, iteration_title, iteration_state, iteration_start_date, iteration_due_date). Both nullable (null on Free tier).