Embedding pipeline improvements building on the concurrent batching
foundation:
- Track docs_embedded vs chunks_embedded separately. A document counts
as embedded only when ALL its chunks succeed, giving accurate
progress reporting. The sync command reads docs_embedded for its
document count.
- Reuse a single Vec<u8> buffer (embed_buf) across all store_embedding
calls instead of allocating per chunk. Eliminates ~3KB allocation per
768-dim embedding.
- Detect and record errors when Ollama silently returns fewer
embeddings than inputs (batch mismatch). Previously these dropped
chunks were invisible.
- Improve retry error messages: distinguish "retry returned unexpected
result" (wrong dims/count) from "retry request failed" (network
error) instead of generic "chunk too large" message.
- Convert all hot-path SQL from conn.execute() to prepare_cached() for
statement cache reuse (clear_document_embeddings, store_embedding,
record_embedding_error).
- Record embedding_metadata errors for empty documents so they don't
appear as perpetually pending on subsequent runs.
- Accept concurrency parameter (configurable via config.embedding.concurrency)
instead of hardcoded EMBED_CONCURRENCY=2.
- Add schema version pre-flight check in embed command to fail fast
with actionable error instead of cryptic SQL errors.
- Fix --retry-failed to use DELETE instead of UPDATE. UPDATE clears
last_error but the row still matches config params in the LEFT JOIN,
making the doc permanently invisible to find_pending_documents.
DELETE removes the row entirely so the LEFT JOIN returns NULL.
Regression test added (old_update_approach_leaves_doc_invisible).
- Add chunking forward-progress guard: after floor_char_boundary()
rounds backward, ensure start advances by at least one full
character to prevent infinite loops on multi-byte sequences
(box-drawing chars, smart quotes). Test cases cover the exact
patterns that caused production hangs on document 18526.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes to the embedding pipeline:
1. Concurrent HTTP batching: fire EMBED_CONCURRENCY (2) Ollama requests
in parallel via join_all, then write results serially to SQLite.
~2x throughput improvement on GPU-bound workloads.
2. UTF-8 boundary safety: all computed byte offsets in split_into_chunks
(paragraph/sentence/word break finders + overlap advance) now use
floor_char_boundary() to prevent panics on multi-byte characters
like smart quotes and non-breaking spaces.
3. CHUNK_MAX_BYTES reduced from 6000 to 1500 to fit nomic-embed-text's
actual 2048-token context window, eliminating context-length retry
storms that were causing 10x slowdowns.
Also threads ShutdownSignal through embed pipeline for graceful Ctrl+C.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sync pipeline was hardcoding `false` for the `full` parameter when
calling run_generate_docs, so `lore sync --full` would re-ingest all
entities but then only regenerate documents for newly-dirtied ones.
Entities loaded before migration 007 (which introduced the dirty_sources
system) were never marked dirty and thus never got documents generated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes to the sync pipeline:
1. Atomic watermarks: wrap complete_job + update_watermark in a single
SQLite transaction so crash between them can't leave partial state.
2. Concurrent drain loops: prefetch HTTP requests via join_all (batch
size = dependent_concurrency), then write serially to DB. Reduces
~9K sequential requests from ~19 min to ~2.4 min.
3. Graceful shutdown: install Ctrl+C handler via ShutdownSignal
(Arc<AtomicBool>), thread through orchestrator/CLI, release locked
jobs on interrupt, record sync_run as "failed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The robot JSON envelope's meta.total_events field was incorrectly
reporting events.len() (the post-limit count), making it identical
to meta.showing. This defeated the purpose of having both fields.
Changes across the pipeline to fix this:
- collect_events now returns (Vec<TimelineEvent>, usize) where the
second element is the total event count before truncation
- TimelineResult gains a total_events_before_limit field (serde-skipped)
so the value flows cleanly from collect through to the renderer
- main.rs passes the real total instead of the events.len() workaround
Additional cleanup in this pass:
- Derive PartialEq/Eq/PartialOrd/Ord on TimelineEventType, replacing
the hand-rolled event_type_discriminant() function. Variant declaration
order now defines sort tiebreak, documented in a doc comment.
- Validate --since input with a proper LoreError::Other instead of
silently treating invalid values as None
- Fix ANSI-aware tag column padding with console::pad_str (colored tags
like "[merged]" were misaligned because ANSI escapes consumed width)
- Remove dead print_timeline_json and infer_max_depth functions that
were superseded by print_timeline_json_with_meta
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete Gate 3 by implementing the final three beads:
- bd-2f2: Human output renderer with colored event tags, entity refs,
evidence snippets, and expansion summary footer
- bd-dty: Robot JSON output with {ok,data,meta} envelope, ISO timestamps,
nested via provenance, and per-event-type details objects
- bd-1nf: CLI wiring with TimelineArgs (9 flags), Commands::Timeline
variant, handle_timeline handler, VALID_COMMANDS entry, and robot-docs
manifest with temporal_intelligence workflow
All 7 Gate 3 children now closed. Pipeline: SEED -> HYDRATE -> EXPAND ->
COLLECT -> RENDER fully operational.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents the timeline pipeline feature in the README:
- New feature bullets: timeline pipeline, git history linking, file
change tracking
- Updated schema table: merge_requests now includes commit SHAs,
added mr_file_changes table
- New "Timeline Pipeline" section explaining the 5-stage architecture
(SEED -> HYDRATE -> EXPAND -> COLLECT -> RENDER) with a table of all
event types and a note on unresolved cross-project references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a SAFETY comment explaining why the transmute of sqlite3_vec_init
to the sqlite3_auto_extension callback type is sound. The three
invariants (stable C-ABI signature, single-call-per-connection contract,
idempotency) were previously undocumented, which left the lone unsafe
block without justification for future readers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds tests/timeline_pipeline_tests.rs with end-to-end integration tests
that exercise the complete timeline pipeline against an in-memory SQLite
database with realistic data:
- pipeline_seed_expand_collect_end_to_end: Full scenario with an issue
closed by an MR, state changes, and label events. Verifies that seed
finds entities via FTS, expand discovers the closing MR through the
entity_references graph, and collect assembles a chronologically sorted
event stream containing Created, StateChanged, LabelAdded, and Merged
events.
- pipeline_empty_query_produces_empty_result: Validates graceful
degradation when FTS returns zero matches -- all three stages should
produce empty results without errors.
- pipeline_since_filter_excludes_old_events: Verifies that the since
timestamp filter propagates correctly through collect, excluding events
before the cutoff while retaining newer ones.
- pipeline_unresolved_refs_have_optional_iid: Tests the Option<i64>
target_iid on UnresolvedRef by creating cross-project references both
with and without known IIDs.
- shared_resolve_entity_ref_scoping: Unit tests for the new shared
resolve_entity_ref helper covering project-scoped lookup, unscoped
lookup, wrong-project rejection, unknown entity types, and nonexistent
entity IDs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Follows up on the resolve_entity_ref extraction by updating all three
pipeline stages to consume the shared helper and removing their local
duplicates (~75 lines of dead code eliminated).
timeline_seed.rs:
- Switch from local resolve_entity to shared resolve_entity_ref with
explicit Some(proj_id) scoping
- Add tracing::debug for orphaned discussion parents instead of silently
skipping them, aiding debugging when evidence notes go missing
- Use saturating_mul for the over-fetch multiplier to prevent overflow on
pathological max_seeds values
timeline_expand.rs:
- Switch from local resolve_entity_ref to shared version with None
project scoping (cross-project traversal)
- Pass Option<i64> for target_iid in UnresolvedRef construction instead
of unwrap_or(0) sentinel
- Update test assertion to compare against Some(42)
timeline_collect.rs:
- Make entity_id_column return Result instead of silently defaulting to
issue_id for unknown entity types. The previous fallback could produce
incorrect SQL queries that return wrong results rather than failing
- Replace if-let chains in collect_merged_event with exhaustive match
blocks that propagate real DB errors while gracefully handling expected
missing-data cases (QueryReturnedNoRows, NULL merged_at)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The seed, expand, and collect stages each had their own near-identical
resolve_entity_ref helper that converted internal DB IDs to full EntityRef
structs. This duplication made it easy for bug fixes to land in one copy
but not the others.
Extract a single public resolve_entity_ref into timeline.rs with an
optional project_id parameter:
- Some(project_id): scopes the lookup (used by seed, which knows the
project from the FTS result)
- None: unscoped lookup (used by expand, which traverses cross-project
references)
Also changes UnresolvedRef.target_iid from i64 to Option<i64>. Cross-
project references parsed from descriptions may not always carry an IID
(e.g. when the reference is malformed or the target was deleted). The
previous sentinel value of 0 was semantically incorrect since GitLab IIDs
start at 1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tests/perf_benchmark.rs with three side-by-side benchmarks that
compare old vs new approaches for the optimizations introduced in the
preceding commits:
- bench_label_insert_individual_vs_batch: measures N individual INSERTs
vs single multi-row INSERT (5k iterations, ~1.6x speedup)
- bench_string_building_old_vs_new: measures format!+push_str vs
writeln! (50k iterations, ~1.9x speedup)
- bench_prepare_vs_prepare_cached: measures prepare vs prepare_cached
(10k iterations, ~1.6x speedup)
Each benchmark verifies correctness (both approaches produce identical
output) and uses std::hint::black_box to prevent dead-code
elimination. Run with: cargo test --test perf_benchmark -- --nocapture
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change OllamaClient::embed_batch to accept &[&str] instead of
Vec<String>. The EmbedRequest struct now borrows both model name and
input texts, eliminating per-batch cloning of chunk text (up to 32KB
per chunk x 32 chunks per batch). Serialization output is identical
since serde serializes &str and String to the same JSON.
In hybrid search, defer the RrfResult->HybridResult mapping until
after filter+take, so only `limit` items (typically 20) are
constructed instead of up to 1,500 at RECALL_CAP. Also switch
filtered_ids to into_iter() to avoid an extra .copied() pass.
Switch FTS search_fts from prepare() to prepare_cached() for statement
reuse across repeated searches. Benchmarked at ~1.6x faster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace individual INSERT-per-label and INSERT-per-path loops in
upsert_document_inner with single multi-row INSERT statements. For a
document with 5 labels, this reduces 5 SQL round-trips to 1.
Replace format!()+push_str() with writeln!() in all three document
extractors (issue, MR, discussion). writeln! writes directly into the
String buffer, avoiding the intermediate allocation that format!
creates. Benchmarked at ~1.9x faster for string building and ~1.6x
faster for batch inserts (measured over 5k iterations in-memory).
Also switch get_existing_hash from prepare() to prepare_cached() since
it is called once per document during regeneration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical fix: Migration 015 exists on disk but was not registered in db.rs.
All beads referencing "migration 015 for mr_file_changes" corrected to migration
016. bd-1oo retitled to reflect dual responsibility (register 015 + create 016).
bd-2y79 renumbered from 016 to 017.
Revised beads: bd-1oo, bd-2yo, bd-1yx, bd-2y79, bd-1nf, bd-2f2, bd-ike,
bd-14q, bd-1ht, bd-z94, bd-2n4.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration 015 adds merge_commit_sha/squash_commit_sha to merge_requests
(Gate 4/5 prerequisites), closes_issues_synced_for_updated_at watermark
for incremental sync, and the missing idx_label_events_label index.
The MR transformer and ingestion pipeline now populate commit SHAs during
sync. The orchestrator uses watermark-based filtering for closes_issues
jobs instead of re-enqueuing all MRs every sync.
The Phase B PRD is updated to match the actual codebase: corrected
migration numbering (011-015), documented nullable label/milestone
fields (migration 012), watermark patterns (013), observability
infrastructure (014), simplified source_method values, and updated
entity_references schema to match implementation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix 6 beads (bd-1ht, bd-2n4, bd-9dd, bd-z94, bd-1yx, bd-3as) that
incorrectly claimed merge_requests has NO merged_at column. Migration
006 defines it and it's used throughout the codebase. Updated SQL
ordering to use COALESCE(merged_at, updated_at).
- Fix bd-32q: build_safe_fts_query() -> to_fts_query(query, FtsQueryMode::Safe)
(actual function in src/search/fts.rs)
- Add Rust JSON struct examples to bd-dty (robot mode output)
- Add edge cases section to bd-jec (config flag)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Audited all 24 open beads against the actual codebase state and
Phase B spec. Key corrections:
- Added Codebase Context sections documenting Gates 1-2 as COMPLETE
(migrations 011-014, all resource event + reference infrastructure)
- Fixed entity_type from &'static str to String for Serialize compat
- Documented actual source_method values (api/note_parse/description_parse)
vs spec's original values (api_closes_issues etc.)
- Noted merge_requests has NO merged_at column (use updated_at)
- Confirmed migration 015 as next sequential number
- Added NULL label_name/milestone_title handling (migration 012)
- Fixed --since filter threading through collect_events phase
- Added Merged event deduplication from StateChanged{merged}
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Score-2 beads (11 beads, previously stubs) now include:
- Background with rationale and system fit
- Approach with exact code snippets, SQL queries, and type signatures
- Binary acceptance criteria with specific file paths
- TDD loops with test names and verify commands
- Edge cases and gotchas
Score-3 beads (10 beads, previously adequate) enriched with:
- Concrete TDD loops and test names
- Specific SQL queries for database operations
- Edge case documentation
All beads now target score 4+ for autonomous agent execution.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove artificial length limits from `lore show` output to display
complete descriptions and discussion threads.
Previously, descriptions were truncated to 500 characters and discussion
notes to 300 characters, which cut off important context when reviewing
issues and MRs. Users often need the full content to understand the
complete discussion history.
Changes:
- Remove truncate() helper function and its 2 unit tests
- Pass description and note bodies directly to wrap_text()
- Affects both print_show_issue() and print_show_mr()
The wrap_text() function continues to handle line wrapping for
readability at the configured widths (76/72/68 chars depending on
nesting level).
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
CLI help improvements (cli/mod.rs):
- Add descriptive help text to all global flags (-c, --robot, -J, etc.)
- Add descriptions to all subcommands (Issues, Mrs, Sync, etc.)
- Add --no-quiet flag for explicit quiet override
- Shell completions now shows installation instructions for each shell
- Optional subcommand: running bare 'lore' shows help in terminal mode,
robot-docs in robot mode
Structured clap error handling (main.rs):
- Early robot mode detection before parsing (env + args)
- JSON error output for parse failures in robot mode
- Semantic error codes: UNKNOWN_COMMAND, UNKNOWN_FLAG, MISSING_REQUIRED,
INVALID_VALUE, ARGUMENT_CONFLICT, etc.
- Fuzzy command suggestion using Jaro-Winkler similarity (>0.7 threshold)
- Help/version requests handled normally (exit 0, not error)
Robot-docs enhancements (main.rs):
- Document deprecated command aliases (list issues -> issues, etc.)
- Document clap error codes for programmatic error handling
- Include completions command in manifest
- Update flag documentation to show short forms (-n, -s, -p, etc.)
Dependencies:
- Add strsim 0.11 for Jaro-Winkler fuzzy matching
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Enables preview of operations without making changes, useful for
understanding what would happen before committing to a full sync.
Ingest dry-run (--dry-run flag):
- Shows resource type, sync mode (full vs incremental), project list
- Per-project info: existing count, has_cursor, last_synced timestamp
- No GitLab API calls, no database writes
Sync dry-run (--dry-run flag):
- Preview all four stages: issues ingest, MRs ingest, docs, embed
- Shows which stages would run vs be skipped (--no-docs, --no-embed)
- Per-project breakdown for both entity types
Stats repair dry-run (--dry-run flag):
- Shows what would be repaired without executing repairs
- "would fix" vs "fixed" indicator in terminal output
- dry_run: true field in JSON response
Implementation details:
- DryRunPreview struct captures project-level sync state
- SyncDryRunResult aggregates previews for all sync stages
- Terminal output uses yellow styling for "would" actions
- JSON output includes dry_run: true at top level
Flag handling:
- --dry-run and --no-dry-run pair for explicit control
- Defaults to false (normal operation)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Issue detail now includes:
- assignees: List of assigned usernames from issue_assignees table
- due_date: Issue due date when set
- milestone: Milestone title when assigned
- closing_merge_requests: MRs that will close this issue when merged
Closing MR detection:
- Queries entity_references table for 'closes' reference type
- Shows MR iid, title, state (with color coding) in terminal output
- Full MR metadata included in JSON output
Human-readable output:
- "Assignees:" line shows comma-separated @usernames
- "Development:" section lists closing MRs with state indicator
- Green for merged, cyan for opened, red for closed
JSON output:
- New fields: assignees, due_date, milestone, closing_merge_requests
- closing_merge_requests array contains iid, title, state, web_url
Test coverage:
- get_issue_assignees: empty, single, multiple (alphabetical order)
- get_closing_mrs: empty, single, ignores 'mentioned' references
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
HTTP client initialization (embedding/ollama.rs, gitlab/client.rs):
- Replace expect/panic with unwrap_or_else fallback to default Client
- Log warning when configured client fails to build
- Prevents crash on TLS/system configuration issues
Doctor command (cli/commands/doctor.rs):
- Handle reqwest Client::builder() failure in Ollama health check
- Return Warning status with descriptive message instead of panicking
- Ensures doctor command remains operational even with HTTP issues
These changes improve resilience when running in unusual environments
(containers with limited TLS, restrictive network policies, etc.)
without affecting normal operation.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Change detection queries (embedding/change_detector.rs):
- Replace triple-EXISTS subquery pattern with LEFT JOIN + NULL check
- SQLite now scans embedding_metadata once instead of three times
- Semantically identical: returns docs needing embedding when no
embedding exists, hash changed, or config mismatch
Count queries (cli/commands/count.rs):
- Consolidate 3 separate COUNT queries for issues into single query
using conditional aggregation (CASE WHEN state = 'x' THEN 1)
- Same optimization for MRs: 5 queries reduced to 1
Search filter queries (search/filters.rs):
- Replace N separate EXISTS clauses for label filtering with single
IN() clause with COUNT/GROUP BY HAVING pattern
- For multi-label AND queries, this reduces N subqueries to 1
FTS tokenization (search/fts.rs):
- Replace collect-into-Vec-then-join pattern with direct String building
- Pre-allocate capacity hint for result string
Discussion truncation (documents/truncation.rs):
- Calculate total length without allocating concatenated string first
- Only allocate full string when we know it fits within limit
Embedding pipeline (embedding/pipeline.rs):
- Add Vec::with_capacity hints for chunk work and cleared_docs hashset
- Reduces reallocations during embedding batch processing
Backoff calculation (core/backoff.rs):
- Replace unchecked addition with saturating_add to prevent overflow
- Add test case verifying overflow protection
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Applies the same doc comment cleanup to test files:
- Removes test module headers (//! lines)
- Removes obvious test function comments
- Retains comments explaining non-obvious test scenarios
Test names should be descriptive enough to convey intent without
additional comments. Complex test setup or assertions that need
explanation retain their comments.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Removes module-level doc comments (//! lines) and excessive inline doc
comments that were duplicating information already evident from:
- Function/struct names (self-documenting code)
- Type signatures (the what is clear from types)
- Implementation context (the how is clear from code)
Affected modules:
- cli/* - Removed command descriptions duplicating clap help text
- core/* - Removed module headers and obvious function docs
- documents/* - Removed extractor/regenerator/truncation docs
- embedding/* - Removed pipeline and chunking docs
- gitlab/* - Removed client and transformer docs (kept type definitions)
- ingestion/* - Removed orchestrator and ingestion docs
- search/* - Removed FTS and vector search docs
Philosophy: Code should be self-documenting. Comments should explain
"why" (business decisions, non-obvious constraints) not "what" (which
the code itself shows). This change reduces noise and maintenance burden
while keeping the codebase just as understandable.
Retains comments for:
- Non-obvious business logic
- Important safety invariants
- Complex algorithm explanations
- Public API boundaries where generated docs matter
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds test coverage for the new GitLabIssueRef type used by the
MR closes_issues API endpoint:
- deserializes_gitlab_issue_ref: Single object with all fields
- deserializes_gitlab_issue_ref_array: Array of refs (typical API response)
Validates that cross-project references (different project_id values)
deserialize correctly, which is important for cross-project close links.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extends the MR ingestion pipeline to populate the entity_references table
from multiple sources:
1. Resource state events (extract_refs_from_state_events):
Called after draining the resource_events queue for both issues and MRs.
Extracts "closes" relationships from the structured API data.
2. System notes (extract_refs_from_system_notes):
Called during MR ingestion to parse "mentioned in" and "closed by"
patterns from discussion note bodies.
3. MR closes_issues API (new):
- enqueue_mr_closes_issues_jobs(): Queues jobs for all MRs
- drain_mr_closes_issues(): Fetches closes_issues for each MR
- Records cross-references with source_method='closes_issues_api'
New progress events:
- ClosesIssuesFetchStarted { total }
- ClosesIssueFetched { current, total }
- ClosesIssuesFetchComplete { fetched, failed }
New result fields on IngestMrProjectResult:
- closes_issues_fetched: Count of successful fetches
- closes_issues_failed: Count of failed fetches
The pipeline now comprehensively builds the relationship graph between
issues and MRs, enabling queries like "what will close this issue?"
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extends the GitLab client to fetch the list of issues that an MR will close
when merged, using the /projects/:id/merge_requests/:iid/closes_issues endpoint.
New type:
- GitLabIssueRef: Lightweight issue reference with id, iid, project_id, title,
state, and web_url. Used for the closes_issues response which returns a list
of issue summaries rather than full GitLabIssue objects.
New client method:
- fetch_mr_closes_issues(gitlab_project_id, iid): Returns Vec<GitLabIssueRef>
for all issues that the MR's description/commits indicate will be closed.
This enables building the entity_references table from API data in addition to
parsing system notes, providing more reliable cross-reference discovery.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The note_parser module requires regex for extracting "mentioned in" and
"closed by" patterns from GitLab system notes. The regex crate provides:
- LazyLock-compatible lazy compilation (Regex::new at first use)
- Named capture groups for clean field extraction
- Efficient iteration over all matches via captures_iter()
Version 1.x is the current stable release with good compile times.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Introduces two new modules for extracting and storing entity cross-references
from GitLab data:
note_parser.rs:
- Parses system notes for "mentioned in" and "closed by" patterns
- Extracts cross-project references (group/project#42, group/project!123)
- Uses lazy-compiled regexes for performance
- Handles both issue (#) and MR (!) sigils
- Provides extract_refs_from_system_notes() for batch processing
references.rs:
- Extracts refs from resource_state_events table (API-sourced closes links)
- Provides insert_entity_reference() for storing discovered references
- Includes resolution helpers: resolve_issue_local_id, resolve_mr_local_id,
resolve_project_path for converting iids to internal IDs
- Enables cross-project reference resolution
These modules power the entity_references table, enabling features like
"find all MRs that close this issue" and "find all issues mentioned in this MR".
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Stages 3 (generate-docs) and 4 (embed) reported progress by appending
"(N/M)" text to the stage spinner message, while stages 1-2 (ingest)
used dedicated indicatif progress bars with animated [====> ] rendering
registered with the global MultiProgress. This visual inconsistency
was introduced when progress callbacks were wired through in 266ed78.
Replace the spinner.set_message() callbacks with proper ProgressBar
instances that match the ingest stage pattern:
- Create a bar-style ProgressBar registered via multi().add()
- Use the same template/progress_chars as the ingest discussion bars
- Lazy-init the tick via AtomicBool to avoid showing the bar before
the first callback fires (matching how ingest enables ticks only
at DiscussionSyncStarted)
- Update set_length on every callback for the docs stage, since the
regenerator's estimated_total can grow if new dirty items are
queued during processing (using .max() internally)
- Clean up both the sub-bar and stage spinner on completion/error
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Three defensive improvements from peer code review:
Replace unreachable!() in GitLab client retry loops:
Both request() and request_with_headers() had unreachable!() after
their for loops. While the logic was sound (the final iteration always
reaches the return/break), any refactor to the loop condition would
turn this into a runtime panic. Restructured both to store
last_response with explicit break, making the control flow
self-documenting and the .expect() message useful if ever violated.
Doctor model name comparison asymmetry:
Ollama model names were stripped of their tag (:latest, :v1.5) for
comparison, but the configured model name was compared as-is. A config
value like "nomic-embed-text:v1.5" would never match. Now strips the
tag from both sides before comparing.
Regenerator savepoint cleanup and progress accuracy:
- upsert_document's error path did ROLLBACK TO but never RELEASE,
leaving a dangling savepoint that could nest on the next call. Added
RELEASE after rollback so the connection is clean.
- estimated_total for progress reporting was computed once at start but
the dirty queue can grow during processing. Now recounts each loop
iteration with max() so the progress fraction never goes backwards.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Three correctness fixes found during peer code review:
Embedding pipeline savepoint leak (HIGH severity):
The SAVEPOINT embed_page / RELEASE embed_page pattern had ~10 `?`
propagation points between them. Any error from record_embedding_error,
clear_document_embeddings, or store_embedding would exit the function
without rolling back, leaving the SQLite connection in a broken
transactional state and causing cascading failures for the rest of the
session. Fixed by extracting page processing into `embed_page()` and
wrapping with explicit rollback-on-error handling.
Dependent queue fail_job race (MEDIUM severity):
fail_job performed a SELECT followed by a separate UPDATE on the
attempts counter without a transaction. Under concurrent lock
reclamation, the attempts value could be read stale. Replaced with a
single atomic UPDATE that increments attempts and computes exponential
backoff entirely in SQL, also halving DB round-trips. Added explicit
error when the job no longer exists.
RRF duplicate document score inflation (MEDIUM severity):
If a retriever returned the same document_id multiple times, the RRF
score accumulated multiple rank contributions while the rank only
recorded the first occurrence. Moved the score accumulation inside the
`if is_none` guard so only the first occurrence per list contributes.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The sync command's stage spinners now show real-time aggregate progress
for each pipeline phase instead of static "syncing..." messages.
- Add `progress_callback` parameter to `run_embed` and
`run_generate_docs` so callers can receive `(processed, total)` updates
- Add `stage_bar` parameter to `run_ingest` for aggregate progress
across concurrently-ingested projects using shared AtomicUsize counters
- Update `stage_spinner` to use `{prefix}` for the `[N/M]` label,
allowing `{msg}` to be updated independently with progress details
- Thread `ProgressBar` clones into each concurrent project task so
per-entity progress (fetch, discussions, events) is reflected on the
aggregate spinner
- Pass `None` for progress callbacks at standalone CLI entry points
(handle_ingest, handle_generate_docs, handle_embed) to preserve
existing behavior when commands are run outside of sync
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add new beads for MR orchestrator integration, sync run observability,
metrics collection, logging infrastructure, and CLI verbosity controls.
Update last-touched timestamp.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- prd-observability.md: Product requirements document for the sync pipeline
observability system, covering structured logging, metrics collection,
sync run tracking, and robot-mode performance output
- gitlore-sync-explorer.html: Self-contained interactive HTML visualization
for exploring sync pipeline stage timings and data flow
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Targeted fixes across multiple subsystems:
dependent_queue:
- Add project_id parameter to claim_jobs() for project-scoped job claiming,
preventing cross-project job theft during concurrent multi-project ingestion
- Add project_id parameter to count_pending_jobs() with optional scoping
(None returns global counts, Some(pid) returns per-project counts)
gitlab/client:
- Downgrade rate-limit log from warn to info (429s are expected operational
behavior, not warnings) and add structured fields (path, status_code)
for better log filtering and aggregation
gitlab/transformers/discussion:
- Add tracing::warn on invalid timestamp parse instead of silent fallback
to epoch 0, making data quality issues visible in logs
ingestion/merge_requests:
- Remove duplicate doc comment on upsert_label_tx
search/rrf:
- Replace partial_cmp().unwrap_or() with total_cmp() for f64 sorting,
eliminating the NaN edge case entirely (total_cmp treats NaN consistently)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add end-to-end observability to the sync and ingest pipelines:
Sync command:
- Generate UUID-based run_id for each sync invocation, propagated through
all child spans for log correlation across stages
- Accept MetricsLayer reference to extract hierarchical StageTiming data
after pipeline completion for robot-mode performance output
- Record sync runs in DB via SyncRunRecorder (start/succeed/fail lifecycle)
- Wrap entire sync execution in a root tracing span with run_id field
Ingest command:
- Wrap run_ingest in an instrumented root span with run_id and resource_type
- Add project path prefix to discussion progress bars for multi-project clarity
- Reset resource_events_synced_for_updated_at on --full re-sync
Sync status:
- Expand from single last_run to configurable recent runs list (default 10)
- Parse and expose StageTiming metrics from stored metrics_json
- Add run_id, total_items_processed, total_errors to SyncRunInfo
- Add mr_count to DataSummary for complete entity coverage
Orchestrator:
- Add #[instrument] with structured fields to issue and MR ingestion functions
- Record items_processed, items_skipped, errors on span close for MetricsLayer
- Emit granular progress events (IssuesFetchStarted, IssuesFetchComplete)
- Pass project_id through to drain_resource_events for scoped job claiming
Document regenerator and embedding pipeline:
- Add #[instrument] spans with items_processed, items_skipped, errors fields
- Record final counts on span close for metrics extraction
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Overhaul the CLI logging infrastructure for production observability:
CLI flags:
- Add -v/-vv/-vvv (--verbose) for progressive stderr verbosity control:
0=INFO, 1=DEBUG app, 2=DEBUG all, 3+=TRACE
- Add --log-format text|json for structured stderr output in automation
- Existing -q/--quiet overrides verbosity for silent operation
Subscriber architecture (main.rs):
- Replace single-layer subscriber with triple-layer setup:
1. stderr layer: human-readable or JSON, filtered by -v flags
2. file layer: always-on JSON to daily-rotated logs (lore.YYYY-MM-DD.log)
3. MetricsLayer: captures span timing for robot-mode performance payloads
- Parse CLI before subscriber init so verbosity is known at setup time
- Load LoggingConfig early (with graceful fallback for pre-init commands)
- Clean up old log files before subscriber init to avoid holding deleted handles
- Hold WorkerGuard at function scope to ensure flush on exit
Doctor command:
- Add logging health check: validates log directory exists, reports file
count and total size, warns on missing or inaccessible log directory
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
