1. fetch_open_threads: replace N+1 loop (2 queries per thread) with a
single query using correlated subqueries for note_count and started_by.
2. extract_key_decisions: track consumed notes so the same note is not
matched to multiple events, preventing duplicate decision entries.
3. build_timeline_excerpt_from_pipeline: log tracing::warn on seed/collect
failures instead of silently returning empty timeline.
The Database(rusqlite::Error) catch-all variant was suggesting
'lore reset --yes' for ALL database errors, including transient
SQLITE_BUSY lock contention. This was wrong on two counts:
1. `lore reset` is not implemented (prints "not yet implemented")
2. Nuking the database is not the fix for a transient lock
Changes:
- Detect SQLITE_BUSY specifically via sqlite_error_code() and provide
targeted advice: "Another process has the database locked" with
common causes (cron sync, concurrent lore command)
- Map SQLITE_BUSY to ErrorCode::DatabaseLocked (exit code 9) instead
of DatabaseError (exit code 10) — semantically correct
- Set BUSY actions to ["lore cron status"] (diagnostic) instead of
the useless "lore sync --force" (--force overrides the app-level
lock table, but SQLITE_BUSY fires before that table is even reached)
- Fix MigrationFailed suggestion: also referenced non-existent
'lore reset', now says "try again" with lore migrate / lore doctor
- Non-BUSY database errors get a simpler suggestion pointing to
lore doctor (no more phantom reset command)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add the ability to sync specific issues or merge requests by IID without
running a full incremental sync. This enables fast, targeted data refresh
for individual entities — useful for agent workflows, debugging, and
real-time investigation of specific issues or MRs.
Architecture:
- New CLI flags: --issue <IID> and --mr <IID> (repeatable, up to 100 total)
scoped to a single project via -p/--project
- Preflight phase validates all IIDs exist on GitLab before any DB writes,
with TOCTOU-aware soft verification at ingest time
- 6-stage pipeline: preflight -> fetch -> ingest -> dependents -> docs -> embed
- Each stage is cancellation-aware via ShutdownSignal
- Dedicated SyncRunRecorder extensions track surgical-specific counters
(issues_fetched, mrs_ingested, docs_regenerated, etc.)
New modules:
- src/ingestion/surgical.rs: Core surgical fetch/ingest/dependent logic
with preflight_fetch(), ingest_issue_by_iid(), ingest_mr_by_iid(),
and fetch_dependents_for_{issue,mr}()
- src/cli/commands/sync_surgical.rs: Full CLI orchestrator with progress
spinners, human/robot output, and cancellation handling
- src/embedding/pipeline.rs: embed_documents_by_ids() for scoped embedding
- src/documents/regenerator.rs: regenerate_dirty_documents_for_sources()
for scoped document regeneration
Database changes:
- Migration 027: Extends sync_runs with mode, phase, surgical_iids_json,
per-entity counters, and cancelled_at column
- New indexes: idx_sync_runs_mode_started, idx_sync_runs_status_phase_started
GitLab client:
- get_issue_by_iid() and get_mr_by_iid() single-entity fetch methods
Error handling:
- New SurgicalPreflightFailed error variant with entity_type, iid, project,
and reason fields. Shares exit code 6 with GitLabNotFound.
Includes comprehensive test coverage:
- 645 lines of surgical ingestion tests (wiremock-based)
- 184 lines of scoped embedding tests
- 85 lines of scoped regeneration tests
- 113 lines of GitLab client single-entity tests
- 236 lines of sync_run surgical column/counter tests
- Unit tests for SyncOptions, error codes, and CLI validation
Add a "Phase 1.5" status enrichment step to the issue ingestion pipeline
that fetches work item statuses via the GitLab GraphQL API after the
standard REST API ingestion completes.
Schema changes (migration 021):
- Add status_name, status_category, status_color, status_icon_name, and
status_synced_at columns to the issues table (all nullable)
Ingestion pipeline changes:
- New `enrich_issue_statuses_txn()` function that applies fetched
statuses in a single transaction with two phases: clear stale statuses
for issues that no longer have a status widget, then apply new/updated
statuses from the GraphQL response
- ProgressEvent variants for status enrichment (complete/skipped)
- IngestProjectResult tracks enrichment metrics (seen, enriched, cleared,
without_widget, partial_error_count, enrichment_mode, errors)
- Robot mode JSON output includes per-project status enrichment details
Configuration:
- New `sync.fetchWorkItemStatus` config option (defaults true) to disable
GraphQL status enrichment on instances without Premium/Ultimate
- `LoreError::GitLabAuthFailed` now treated as permanent API error so
status enrichment auth failures don't trigger retries
Also removes the unnecessary nested SAVEPOINT in store_closes_issues_refs
(already runs within the orchestrator's transaction context).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Robot mode consistency improvements across all command output:
Timing:
- Every robot JSON response now includes meta.elapsed_ms measuring
wall-clock time from command start to serialization. Agents can use
this to detect slow queries and tune --limit or --project filters.
Field selection (--fields):
- print_list_issues_json and print_list_mrs_json accept an optional
fields slice that prunes each item in the response array to only
the requested keys. A "minimal" preset expands to
[iid, title, state, updated_at_iso] for token-efficient agent scans.
- filter_fields and expand_fields_preset live in the new
src/cli/robot.rs module alongside RobotMeta.
Actionable error recovery:
- LoreError gains an actions() method returning concrete shell commands
an agent can execute to recover (e.g. "ollama serve" for
OllamaUnavailable, "lore init" for ConfigNotFound).
- RobotError now serializes an "actions" array (empty array omitted)
so agents can parse and offer one-click fixes.
Envelope consistency:
- show issue/MR JSON responses now use the standard
{"ok":true,"data":...,"meta":...} envelope instead of bare data,
matching all other commands.
Files: src/cli/robot.rs (new), src/core/error.rs,
src/cli/commands/{count,embed,generate_docs,ingest,list,show,stats,sync_status}.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Shutdown signal improvements:
- Upgrade ShutdownSignal from Relaxed to Release/Acquire ordering.
Relaxed was technically sufficient for a single flag but
Release/Acquire is the textbook correct pattern and ensures
visibility guarantees across threads without relying on x86 TSO.
- Add double Ctrl+C support to all three signal handlers (ingest,
embed, sync). First Ctrl+C sets cooperative flag with user message;
second Ctrl+C force-exits with code 130 (standard SIGINT convention).
CLI hardening:
- LORE_ROBOT env var now checks for truthy values (!empty, !="0",
!="false") instead of mere existence. Setting LORE_ROBOT=0 or
LORE_ROBOT=false no longer activates robot mode.
- Replace unreachable!() in color mode match with defensive warning
and fallback to auto. Clap validates the values but defense in depth
prevents panics if the value_parser is ever changed.
- Replace unreachable!() in completions shell match with proper error
return for unsupported shells.
Exit code collision fix:
- ConfigNotFound was mapped to exit code 2 (error.rs:56) which
collided with handle_clap_error() also using exit code 2 for parse
errors. Agents calling lore --robot could not distinguish "bad
arguments" from "missing config file."
- Restore ConfigNotFound to exit code 20 (its original dedicated code).
- Update robot-docs exit code table: code 2 = "Usage error", code 20 =
"Config not found".
Build script:
- Track .git/refs/heads directory for Cargo rebuild triggers. Ensures
GIT_HASH env var updates when branch refs change, not just HEAD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Removes module-level doc comments (//! lines) and excessive inline doc
comments that were duplicating information already evident from:
- Function/struct names (self-documenting code)
- Type signatures (the what is clear from types)
- Implementation context (the how is clear from code)
Affected modules:
- cli/* - Removed command descriptions duplicating clap help text
- core/* - Removed module headers and obvious function docs
- documents/* - Removed extractor/regenerator/truncation docs
- embedding/* - Removed pipeline and chunking docs
- gitlab/* - Removed client and transformer docs (kept type definitions)
- ingestion/* - Removed orchestrator and ingestion docs
- search/* - Removed FTS and vector search docs
Philosophy: Code should be self-documenting. Comments should explain
"why" (business decisions, non-obvious constraints) not "what" (which
the code itself shows). This change reduces noise and maintenance burden
while keeping the codebase just as understandable.
Retains comments for:
- Non-obvious business logic
- Important safety invariants
- Complex algorithm explanations
- Public API boundaries where generated docs matter
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Introduce the foundational observability layer for the sync pipeline:
- MetricsLayer: Custom tracing subscriber layer that captures span timing
and structured fields, materializing them into a hierarchical
Vec<StageTiming> tree for robot-mode performance data output
- logging: Dual-layer subscriber infrastructure with configurable stderr
verbosity (-v/-vv/-vvv) and always-on JSON file logging with daily
rotation and configurable retention (default 30 days)
- SyncRunRecorder: Compile-time enforced lifecycle recorder for sync_runs
table (start -> succeed|fail), with correlation IDs and aggregate counts
- LoggingConfig: New config section for log_dir, retention_days, and
file_logging toggle
- get_log_dir(): Path helper for log directory resolution
- is_permanent_api_error(): Distinguish retryable vs permanent API failures
(only 404 is truly permanent; 403/auth errors may be environmental)
Database changes:
- Migration 013: Add resource_events_synced_for_updated_at watermark columns
to issues and merge_requests tables for incremental resource event sync
- Migration 014: Enrich sync_runs with run_id correlation ID, aggregate
counts (total_items_processed, total_errors), and run_id index
- Wrap file-based migrations in savepoints for rollback safety
Dependencies: Add uuid (run_id generation), tracing-appender (file logging)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Automated formatting and lint corrections from parallel agent work:
- cargo fmt: import reordering (alphabetical), line wrapping to respect
max width, trailing comma normalization, destructuring alignment,
function signature reformatting, match arm formatting
- clippy (pedantic): Range::contains() instead of manual comparisons,
i64::from() instead of `as i64` casts, .clamp() instead of
.max().min() chains, let-chain refactors (if-let with &&),
#[allow(clippy::too_many_arguments)] and
#[allow(clippy::field_reassign_with_default)] where warranted
- Removed trailing blank lines and extra whitespace
No behavioral changes. All existing tests pass unmodified.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ConfigNotFound previously used exit code 2 which collides with clap's
usage error code. Remap it to exit 20 to avoid ambiguity. Also add
dedicated NotFound (exit 17) and Ambiguous (exit 18) error codes with
proper ErrorCode variants and Display implementations, replacing the
previous incorrect mapping of these errors to GitLabNotFound.
Co-Authored-By: Claude (us.anthropic.claude-opus-4-5-20251101-v1:0) <noreply@anthropic.com>
Mechanical rename of GiError -> LoreError across the core module to
match the project's rebranding from gitlab-inbox to gitlore/lore.
Updates the error enum name, all From impls, and the Result type alias.
Additionally introduces:
- New error variants for embedding pipeline: OllamaUnavailable,
OllamaModelNotFound, EmbeddingFailed, EmbeddingsNotBuilt. Each
includes actionable suggestions (e.g., "ollama serve", "ollama pull
nomic-embed-text") to guide users through recovery.
- New error codes 14-16 for programmatic handling of Ollama failures.
- Savepoint-based migration execution in db.rs: each migration now
runs inside a SQLite SAVEPOINT so a failed migration rolls back
cleanly without corrupting the schema_version tracking. Previously
a partial migration could leave the database in an inconsistent
state.
- core::backoff module: exponential backoff with jitter utility for
retry loops in the embedding pipeline and discussion queues.
- core::project module: helper for resolving project IDs and paths
from the local database, used by the document regenerator and
search filters.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Error suggestions now include concrete CLI examples so users
(and robot-mode consumers) can act immediately without consulting
docs. For instance, ConfigNotFound now shows the expected path
and the exact command to run, TokenNotSet shows the export syntax,
and Ambiguous shows the -p flag with example project paths.
Also fixes the error code for Ambiguous errors: it now maps to
GitLabNotFound instead of InternalError, since the entity exists
but the user needs to disambiguate -- not an internal failure.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Improves core infrastructure with robot-friendly error output and
faster lock release for better sync behavior.
Error handling improvements (error.rs):
- ErrorCode::exit_code(): Unique exit codes per error type (1-13)
for programmatic error handling in scripts/agents
- GiError::suggestion(): Helpful hints for common error recovery
- GiError::to_robot_error(): Structured JSON error conversion
- RobotError/RobotErrorOutput: Serializable error types with code,
message, and optional suggestion fields
Lock improvements (lock.rs):
- Heartbeat thread now polls every 100ms for release flag, only
updating database heartbeat at full interval (5s default)
- Eliminates 5-10s delay after sync completion when waiting for
heartbeat thread to notice release
- Reduces lock hold time after operation completes
Database (db.rs):
- Bump expected schema version to 6 for MR migration
The exit code mapping enables shell scripts and CI/CD pipelines to
distinguish between configuration errors (2-4), GitLab API errors
(5-8), and database errors (9-11) for appropriate retry/alert logic.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
