The vector search multiplier could grow unbounded on documents with
many chunks, producing enormous k values that cause SQLite to scan
far more rows than necessary. Clamp the multiplier to [8, 200] and
cap k at 10,000 to prevent degenerate performance on large corpora.
Also adds a debug_assert in decode_rowid to catch negative rowids
early — these indicate a bug in the encoding pipeline and should
fail fast rather than silently produce garbage document IDs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds mr_diffs_fetched and mr_diffs_failed fields to IngestResult and
SyncResult, threads them through the orchestrator aggregation, includes
them in the structured tracing span and human-readable sync summary.
Previously MR diff failures were silently swallowed — now they appear
alongside resource event counts for full pipeline observability.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Previously, drain_resource_events, drain_mr_closes_issues, and
drain_mr_diffs each opened a transaction only for the job-complete +
watermark update, but the store operation ran outside that transaction.
If the process crashed between the store and the watermark update, data
would be persisted without the watermark advancing, causing silent
duplicates on the next sync.
Now each drain function opens the transaction before the store call and
commits it only after both the store and the watermark update succeed.
On error, the transaction is explicitly dropped so the connection is
not left in a half-committed state.
Also:
- store_resource_events no longer manages its own transaction; the caller
passes in a connection (which is actually the transaction)
- upsert_mr_file_changes wraps DELETE + INSERT in a transaction internally
- reset_discussion_watermarks now also clears diffs_synced_for_updated_at
- Orchestrator error span now includes closes_issues_failed + mr_diffs_failed
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The --since naming is more intuitive (matches git log --since) and
consistent with the list commands which already use --since. Renames
the CLI flags, SearchCliFilters fields, SearchFilters fields,
autocorrect registry, and robot-docs manifest. No behavioral change.
Affected paths:
- cli/mod.rs: SearchArgs field + clap attribute rename
- cli/commands/search.rs: SearchCliFilters + run_search plumbing
- search/filters.rs: SearchFilters struct + apply_filters logic
- main.rs: handle_search + robot-docs JSON
- cli/autocorrect.rs: COMMAND_FLAGS entry for search
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration 019 created performance indexes but never recorded itself
in the schema_version table. Without this row the migration runner
considers the schema outdated and would attempt to re-apply. Adds
the standard INSERT INTO schema_version for version 19.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bump version from 0.1.0 to 0.5.0 to reflect the maturity of the CLI
after months of development — robot mode, search pipeline, ingestion
orchestrator, who commands, timeline pipeline, and embedding support
are all implemented and stable.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a Claude Code skill that automates the release workflow:
parse bump type (major/minor/patch), update Cargo.toml + Cargo.lock,
commit, and tag. Intentionally does not auto-push so the user
retains control over when releases go to the remote.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signal 4 (mr_reviewers + mr_file_changes) was missing the self-review
exclusion that signal 1 (DiffNote reviewer) already had. An MR author
listed as their own reviewer would be double-counted as both author
and reviewer, inflating their score.
Also removes redundant SELECT DISTINCT from signal 2 (GROUP BY
already ensures uniqueness).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Chain: bd-jec (config flag) -> bd-2yo (fetch MR diffs) -> bd-3qn6 (rewrite who queries)
- Add fetch_mr_file_changes config option and --no-file-changes CLI flag
- Add GitLab MR diffs API fetch pipeline with watermark-based sync
- Create migration 020 for diffs_synced_for_updated_at watermark column
- Rewrite query_expert() and query_overlap() to use 4-signal UNION ALL:
DiffNote reviewers, DiffNote MR authors, file-change authors, file-change reviewers
- Deduplicate across signal types via COUNT(DISTINCT CASE WHEN ... THEN mr_id END)
- Add insert_file_change test helper, 8 new who tests, all 397 tests pass
- Also includes: list performance migration 019, autocorrect module, README updates
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three micro-optimizations with zero behavioral change:
1. timeline_collect.rs: Reorder format!() before enum construction so
the owned String moves into the variant directly, eliminating
.clone() on state, label, and milestone strings in StateChanged,
LabelAdded/Removed, and MilestoneSet/Removed event paths.
2. pipeline.rs: Use Arc<str> for doc_hash shared across a document's
chunks instead of cloning the full String per chunk. Also remove
redundant embed_buf.reserve() since extend_from_slice already
handles growth and the buffer is reused across iterations.
3. rrf.rs: Pre-allocate HashMap with combined vector+fts result count
via with_capacity() to avoid rehashing during RRF score accumulation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive peer code review identified and fixed the following:
1. who.rs: @-prefixed path routing used `target` (with @) instead of
`clean` (stripped) when checking for '/' and passing to Expert mode,
causing `lore who @src/auth/` to silently return zero results because
the SQL LIKE matched against `@src/auth/%` which never exists.
2. db.rs: After ROLLBACK TO savepoint on migration failure, the savepoint
was never RELEASEd, leaving it active on the connection. Fixed in both
run_migrations() and run_migrations_from_dir().
3. lock.rs: Multiple acquire() calls (e.g. re-acquiring a stale lock)
replaced the heartbeat_handle without stopping the old thread, causing
two concurrent heartbeat writers competing on the same lock row. Now
signals the old thread to stop and joins it before spawning a new one.
4. chunk_ids.rs: encode_rowid() had no guard for chunk_index >= 1000
(CHUNK_ROWID_MULTIPLIER), which would cause rowid collisions between
adjacent documents. Added range assertion [0, 1000).
5. main.rs: Fallback JSON error formatting in handle_auth_test
interpolated LoreError Display output without escaping quotes or
backslashes, potentially producing malformed JSON for robot-mode
consumers. Now escapes both characters before interpolation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three defensive improvements found via peer code review:
1. lock.rs: Lock release errors were silently discarded with `let _ =`.
If the DELETE failed (disk full, corruption), the lock stayed in the
database with no diagnostic. Next sync would require --force with no
clue why. Now logs with error!() including the underlying error message.
2. filters.rs: Dynamic SQL label filter construction had no upper bound
on bind parameters. With many combined filters, param_idx + labels.len()
could exceed SQLite's 999-parameter limit, producing an opaque error.
Added a guard that caps labels at 900 - param_idx.
3. vector.rs: max_chunks_per_document returned i64 which was cast to
usize. A negative value from a corrupt database would wrap to a huge
number, causing overflow in the multiplier calculation. Now clamped
to .max(1) and cast via unsigned_abs().
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The orchestrator already accepted a ShutdownSignal but only checked it
between phases (after all issues fetched, before discussions). The inner
loops in ingest_issues() and ingest_merge_requests() consumed entire
paginated streams without checking for cancellation.
On a large initial sync (thousands of issues/MRs), Ctrl+C could be
unresponsive for minutes while the current entity type finished draining.
Now both functions accept &ShutdownSignal and check is_cancelled() at
the top of each iteration, breaking out promptly and committing the
cursor for whatever was already processed.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Peer code review found multiple panic-reachable paths:
1. serde_json::to_string().unwrap() in 4 robot-mode output functions
(who.rs, main.rs x3). If serialization ever failed (e.g., NaN from
edge-case division), the CLI would panic with an unhelpful stack trace.
Replaced with unwrap_or_else that emits a structured JSON error fallback.
2. encode_rowid() in chunk_ids.rs used unchecked multiplication
(document_id * 1000). On extreme document IDs this could silently wrap
in release mode, causing embedding rowid collisions. Now uses
checked_mul + checked_add with a diagnostic panic message.
3. HTTP response body truncation at byte index 500 in client.rs could
split a multi-byte UTF-8 character, causing a panic. Now uses
floor_char_boundary(500) for safe truncation.
4. who.rs reviews mode: SQL used `m.author_username != ?1` which silently
dropped MRs with NULL author_username (SQL NULL != anything = NULL).
Changed to `(m.author_username IS NULL OR m.author_username != ?1)`
to match the pattern already used in expert mode.
5. handle_auth_test hardcoded exit code 5 for all errors regardless of
type. Config not found (20), token not set (4), and network errors (8)
all incorrectly returned 5. Now uses e.exit_code() from the actual
LoreError, with proper suggestion hints in human mode.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three correctness bugs found via peer code review:
1. TimelineEvent PartialEq/Ord omitted entity_type — issue #42 and MR #42
with the same timestamp and event_type were treated as equal. In a
BTreeSet or dedup, one would silently be dropped. Added entity_type to
both PartialEq and Ord comparisons.
2. discussions.rs: store_payload() was called outside the transaction
(on bare conn) while upsert_discussion/notes were inside. A crash
between them left orphaned payload rows. Moved store_payload inside
the unchecked_transaction block, matching mr_discussions.rs pattern.
3. Migration 017 created idx_issue_assignees_username(username, issue_id)
but migration 005 already created the same index name with just
(username). SQLite's IF NOT EXISTS silently skipped the composite
version on every existing database. New migration 018 drops and
recreates the index with correct composite columns.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add `lore who` command with 5 query modes answering collaboration questions
using existing DB data (280K notes, 210K discussions, 33K DiffNotes):
- Expert: who knows about a file/directory (DiffNote path analysis + MR breadth scoring)
- Workload: what is a person working on (assigned issues, authored/reviewing MRs, discussions)
- Active: what discussions need attention (unresolved resolvable, global/project-scoped)
- Overlap: who else is touching these files (dual author+reviewer role tracking)
- Reviews: what review patterns does a person have (prefix-based category extraction)
Includes migration 017 (5 composite indexes), CLI skeleton with clap conflicts_with
validation, robot JSON output with input+resolved_input reproducibility, human terminal
output, and 20 unit tests. All quality gates pass.
Closes: bd-1q8z, bd-34rr, bd-2rk9, bd-2ldg, bd-zqpf, bd-s3rc, bd-m7k1, bd-b51e,
bd-2711, bd-1rdi, bd-3mj2, bd-tfh3, bd-zibc, bd-g0d5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sync the agent instructions with the current robot mode implementation:
- Add RUST_CLI_TOOLS_BEST_PRACTICES.md reference for Rust coding guidance
- Expand robot mode description to cover all new capabilities
- Add --fields examples (minimal preset, custom field lists)
- Document error actions array for automated recovery workflows
- Update response format to show elapsed_ms and actions in error envelope
- Add field selection section with usage examples
- Separate health check to exit code 19 (was overloaded on exit code 1)
- Add robot-docs recommendation for response schema discovery
- Update best practices with --fields minimal for token efficiency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add YAML frontmatter metadata (plan: true, status: drafting, iteration: 0)
to integrate with the iterative plan review workflow.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Comprehensive update to the robot mode design document bringing it in sync
with the actual implementation after the elapsed_ms, --fields, and error
actions features landed.
Major additions:
- Response envelope section documenting compact JSON with elapsed_ms timing
- Error actions table mapping each error code to executable recovery commands
- Field selection section with presets (minimal) and per-entity available fields
- Expanded exit codes table (14-20) covering Ollama, embedding, ambiguity errors
- Updated command examples to use current CLI syntax (lore issues vs lore list issues)
- Added -J shorthand and --fields to global flags table
- Best practices section with --fields minimal for token efficiency (~60% reduction)
Removed outdated sections that no longer match the implementation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Design document for `lore who` — a people intelligence query layer over
existing GitLab data (280K notes, 210K discussions, 33K DiffNotes, 53
participants). Answers five collaboration questions: expert lookup by
file/path, workload summary, review pattern analysis, active discussion
tracking, and file overlap detection.
Key design decisions refined across 8 feedback iterations:
- All SQL is fully static (no format!()) with prepare_cached() throughout
- Exact vs prefix path matching via PathQuery struct (two static SQL variants)
- Self-review exclusion (author != reviewer) on all DiffNote branches
- Deterministic output: sorted GROUP_CONCAT results, stable tie-breakers
- Bounded payloads with *_total/*_truncated metadata for robot consumers
- Truncation transparency via LIMIT+1 overflow detection pattern
- Robot JSON includes resolved_input for reproducibility (since_mode tri-state)
- Multi-project correctness with project-qualified entity references
- Composite migration indexes designed for query selectivity on hot paths
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents the robot mode enhancements from the previous commits:
- Field selection (--fields flag and minimal preset) with examples
and complete field lists for issues and MRs
- Updated response format section to show meta.elapsed_ms and compact
single-line JSON
- Error actions array with recovery shell commands
- Agent self-discovery section explaining robot-docs response_schema
- Exit code 19 for health check failure added to the table
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Addresses findings from a comprehensive CLI readiness audit:
Flag design (I2):
- Add hidden --no-verbose flag with overrides_with semantics, matching
the --no-quiet pattern already established for all other boolean flags.
Help text (I3):
- Add after_help examples to issues, mrs, search, sync, and timeline
subcommands. Each shows 3-4 concrete, runnable commands with comments.
Help headings (I4/P5):
- Move --mode and --fts-mode from "Output" heading to "Mode" heading
in the search subcommand. These control search strategy, not output
format — "Output" is reserved for --limit, --explain, --fields.
Exit codes (I5):
- Health check failure now exits 19 (was 1). Exit code 1 is reserved
for internal errors only. robot-docs updated to document code 19.
Deprecation visibility (P4):
- Deprecated commands (list, show, auth-test, sync-status) now emit
structured JSON warnings to stderr in robot mode:
{"warning":{"type":"DEPRECATED","message":"...","successor":"..."}}
Previously these were silently swallowed in robot mode.
Version string (P1):
- Cli struct uses env!("LORE_VERSION") from build.rs so --version shows
git hash (see previous commit).
Fields flag (P3):
- --fields help text updated to document the "minimal" preset.
Robot-docs (parallel work):
- response_schema added for every command, documenting the JSON shape
agents will receive. Agents can now introspect expected fields before
calling a command.
- error_format documents the new "actions" array.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Robot mode consistency improvements across all command output:
Timing:
- Every robot JSON response now includes meta.elapsed_ms measuring
wall-clock time from command start to serialization. Agents can use
this to detect slow queries and tune --limit or --project filters.
Field selection (--fields):
- print_list_issues_json and print_list_mrs_json accept an optional
fields slice that prunes each item in the response array to only
the requested keys. A "minimal" preset expands to
[iid, title, state, updated_at_iso] for token-efficient agent scans.
- filter_fields and expand_fields_preset live in the new
src/cli/robot.rs module alongside RobotMeta.
Actionable error recovery:
- LoreError gains an actions() method returning concrete shell commands
an agent can execute to recover (e.g. "ollama serve" for
OllamaUnavailable, "lore init" for ConfigNotFound).
- RobotError now serializes an "actions" array (empty array omitted)
so agents can parse and offer one-click fixes.
Envelope consistency:
- show issue/MR JSON responses now use the standard
{"ok":true,"data":...,"meta":...} envelope instead of bare data,
matching all other commands.
Files: src/cli/robot.rs (new), src/core/error.rs,
src/cli/commands/{count,embed,generate_docs,ingest,list,show,stats,sync_status}.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The clap --version flag now shows the git hash alongside the semver
version (e.g. "lore 0.1.0 (a573d69)") instead of bare "lore 0.1.0".
LORE_VERSION is constructed at compile time in build.rs from
CARGO_PKG_VERSION + the short git hash, and consumed via
env!("LORE_VERSION") in the Cli struct's #[command(version)] attribute.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Two new microbenchmarks measuring optimizations applied in this session:
bench_redundant_hash_query_elimination:
Compares the old 2-query pattern (get_existing_hash + full SELECT)
against the new single-query pattern where upsert_document_inner
returns change detection info directly. Uses 100 seeded documents
with 10K iterations, prepare_cached, and black_box to prevent
elision.
bench_embedding_bytes_alloc_vs_reuse:
Compares per-call Vec<u8> allocation against the reusable embed_buf
pattern now used in store_embedding. Simulates 768-dim embeddings
(nomic-embed-text) with 50K iterations. Includes correctness
assertion that both approaches produce identical byte output.
Both benchmarks use informational-only timing (no pass/fail on speed)
with correctness assertions as the actual test criteria, ensuring they
never flake on CI.
Notes recorded in benchmark file:
- SHA256 hex formatting optimization measured at 1.01x (reverted)
- compute_list_hash sort strategy measured at 1.02x (reverted)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Shutdown signal improvements:
- Upgrade ShutdownSignal from Relaxed to Release/Acquire ordering.
Relaxed was technically sufficient for a single flag but
Release/Acquire is the textbook correct pattern and ensures
visibility guarantees across threads without relying on x86 TSO.
- Add double Ctrl+C support to all three signal handlers (ingest,
embed, sync). First Ctrl+C sets cooperative flag with user message;
second Ctrl+C force-exits with code 130 (standard SIGINT convention).
CLI hardening:
- LORE_ROBOT env var now checks for truthy values (!empty, !="0",
!="false") instead of mere existence. Setting LORE_ROBOT=0 or
LORE_ROBOT=false no longer activates robot mode.
- Replace unreachable!() in color mode match with defensive warning
and fallback to auto. Clap validates the values but defense in depth
prevents panics if the value_parser is ever changed.
- Replace unreachable!() in completions shell match with proper error
return for unsupported shells.
Exit code collision fix:
- ConfigNotFound was mapped to exit code 2 (error.rs:56) which
collided with handle_clap_error() also using exit code 2 for parse
errors. Agents calling lore --robot could not distinguish "bad
arguments" from "missing config file."
- Restore ConfigNotFound to exit code 20 (its original dedicated code).
- Update robot-docs exit code table: code 2 = "Usage error", code 20 =
"Config not found".
Build script:
- Track .git/refs/heads directory for Cargo rebuild triggers. Ensures
GIT_HASH env var updates when branch refs change, not just HEAD.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three hardening improvements to the ingestion orchestrator:
- Replace .unwrap_or(0) with ? on COUNT(*) queries for total_issues
and total_mrs. These are simple aggregate queries that should never
fail, but if they do (e.g. table missing after failed migration),
propagating the error gives an actionable message instead of silently
reporting 0 items.
- Wrap store_closes_issues_refs in a SAVEPOINT with proper
ROLLBACK/RELEASE. Previously, a failure mid-loop (e.g. on the 5th of
10 close-issue references) would leave partial refs committed. Now
the entire batch is atomic.
- Replace silent catch-all (_ => {}) arms in enqueue_resource_events
and update_resource_event_watermark with explicit warnings for
unknown entity_type values. Makes debugging easier when new entity
types are added but the match arms aren't updated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The document regenerator was making two queries per document:
1. get_existing_hash() — SELECT content_hash
2. upsert_document_inner() — SELECT id, content_hash, labels_hash, paths_hash
Query 2 already returns the content_hash needed for change detection.
Remove get_existing_hash() entirely and compute content_changed inside
upsert_document_inner() from the existing row data.
upsert_document_inner now returns Result<bool> (true = content changed)
which propagates up through upsert_document and regenerate_one,
replacing the separate pre-check. The triple-hash fast-path (all three
hashes match → return Ok(false) with no writes) is preserved.
This halves the query count for unchanged documents, which dominate
incremental syncs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add input validation for Raw FTS query mode to prevent expensive or
malformed queries from reaching SQLite FTS5:
- Reject unbalanced double quotes (would cause FTS5 syntax error)
- Reject leading wildcard-only queries ("*", "* OR ...") that trigger
expensive full-table scans
- Reject empty/whitespace-only queries
- Invalid raw input falls back to Safe mode automatically instead of
erroring, so callers never see FTS5 parse failures
The Safe mode already escapes all tokens with double-quote wrapping
and handles embedded quotes via doubling. Raw mode now has a
validation layer on top.
All queries remain parameterized (?1, ?2) — user input never enters
SQL strings directly.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Embedding pipeline improvements building on the concurrent batching
foundation:
- Track docs_embedded vs chunks_embedded separately. A document counts
as embedded only when ALL its chunks succeed, giving accurate
progress reporting. The sync command reads docs_embedded for its
document count.
- Reuse a single Vec<u8> buffer (embed_buf) across all store_embedding
calls instead of allocating per chunk. Eliminates ~3KB allocation per
768-dim embedding.
- Detect and record errors when Ollama silently returns fewer
embeddings than inputs (batch mismatch). Previously these dropped
chunks were invisible.
- Improve retry error messages: distinguish "retry returned unexpected
result" (wrong dims/count) from "retry request failed" (network
error) instead of generic "chunk too large" message.
- Convert all hot-path SQL from conn.execute() to prepare_cached() for
statement cache reuse (clear_document_embeddings, store_embedding,
record_embedding_error).
- Record embedding_metadata errors for empty documents so they don't
appear as perpetually pending on subsequent runs.
- Accept concurrency parameter (configurable via config.embedding.concurrency)
instead of hardcoded EMBED_CONCURRENCY=2.
- Add schema version pre-flight check in embed command to fail fast
with actionable error instead of cryptic SQL errors.
- Fix --retry-failed to use DELETE instead of UPDATE. UPDATE clears
last_error but the row still matches config params in the LEFT JOIN,
making the doc permanently invisible to find_pending_documents.
DELETE removes the row entirely so the LEFT JOIN returns NULL.
Regression test added (old_update_approach_leaves_doc_invisible).
- Add chunking forward-progress guard: after floor_char_boundary()
rounds backward, ensure start advances by at least one full
character to prevent infinite loops on multi-byte sequences
(box-drawing chars, smart quotes). Test cases cover the exact
patterns that caused production hangs on document 18526.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes to the embedding pipeline:
1. Concurrent HTTP batching: fire EMBED_CONCURRENCY (2) Ollama requests
in parallel via join_all, then write results serially to SQLite.
~2x throughput improvement on GPU-bound workloads.
2. UTF-8 boundary safety: all computed byte offsets in split_into_chunks
(paragraph/sentence/word break finders + overlap advance) now use
floor_char_boundary() to prevent panics on multi-byte characters
like smart quotes and non-breaking spaces.
3. CHUNK_MAX_BYTES reduced from 6000 to 1500 to fit nomic-embed-text's
actual 2048-token context window, eliminating context-length retry
storms that were causing 10x slowdowns.
Also threads ShutdownSignal through embed pipeline for graceful Ctrl+C.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The sync pipeline was hardcoding `false` for the `full` parameter when
calling run_generate_docs, so `lore sync --full` would re-ingest all
entities but then only regenerate documents for newly-dirtied ones.
Entities loaded before migration 007 (which introduced the dirty_sources
system) were never marked dirty and thus never got documents generated.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes to the sync pipeline:
1. Atomic watermarks: wrap complete_job + update_watermark in a single
SQLite transaction so crash between them can't leave partial state.
2. Concurrent drain loops: prefetch HTTP requests via join_all (batch
size = dependent_concurrency), then write serially to DB. Reduces
~9K sequential requests from ~19 min to ~2.4 min.
3. Graceful shutdown: install Ctrl+C handler via ShutdownSignal
(Arc<AtomicBool>), thread through orchestrator/CLI, release locked
jobs on interrupt, record sync_run as "failed".
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The robot JSON envelope's meta.total_events field was incorrectly
reporting events.len() (the post-limit count), making it identical
to meta.showing. This defeated the purpose of having both fields.
Changes across the pipeline to fix this:
- collect_events now returns (Vec<TimelineEvent>, usize) where the
second element is the total event count before truncation
- TimelineResult gains a total_events_before_limit field (serde-skipped)
so the value flows cleanly from collect through to the renderer
- main.rs passes the real total instead of the events.len() workaround
Additional cleanup in this pass:
- Derive PartialEq/Eq/PartialOrd/Ord on TimelineEventType, replacing
the hand-rolled event_type_discriminant() function. Variant declaration
order now defines sort tiebreak, documented in a doc comment.
- Validate --since input with a proper LoreError::Other instead of
silently treating invalid values as None
- Fix ANSI-aware tag column padding with console::pad_str (colored tags
like "[merged]" were misaligned because ANSI escapes consumed width)
- Remove dead print_timeline_json and infer_max_depth functions that
were superseded by print_timeline_json_with_meta
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Complete Gate 3 by implementing the final three beads:
- bd-2f2: Human output renderer with colored event tags, entity refs,
evidence snippets, and expansion summary footer
- bd-dty: Robot JSON output with {ok,data,meta} envelope, ISO timestamps,
nested via provenance, and per-event-type details objects
- bd-1nf: CLI wiring with TimelineArgs (9 flags), Commands::Timeline
variant, handle_timeline handler, VALID_COMMANDS entry, and robot-docs
manifest with temporal_intelligence workflow
All 7 Gate 3 children now closed. Pipeline: SEED -> HYDRATE -> EXPAND ->
COLLECT -> RENDER fully operational.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Documents the timeline pipeline feature in the README:
- New feature bullets: timeline pipeline, git history linking, file
change tracking
- Updated schema table: merge_requests now includes commit SHAs,
added mr_file_changes table
- New "Timeline Pipeline" section explaining the 5-stage architecture
(SEED -> HYDRATE -> EXPAND -> COLLECT -> RENDER) with a table of all
event types and a note on unresolved cross-project references
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds a SAFETY comment explaining why the transmute of sqlite3_vec_init
to the sqlite3_auto_extension callback type is sound. The three
invariants (stable C-ABI signature, single-call-per-connection contract,
idempotency) were previously undocumented, which left the lone unsafe
block without justification for future readers.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Adds tests/timeline_pipeline_tests.rs with end-to-end integration tests
that exercise the complete timeline pipeline against an in-memory SQLite
database with realistic data:
- pipeline_seed_expand_collect_end_to_end: Full scenario with an issue
closed by an MR, state changes, and label events. Verifies that seed
finds entities via FTS, expand discovers the closing MR through the
entity_references graph, and collect assembles a chronologically sorted
event stream containing Created, StateChanged, LabelAdded, and Merged
events.
- pipeline_empty_query_produces_empty_result: Validates graceful
degradation when FTS returns zero matches -- all three stages should
produce empty results without errors.
- pipeline_since_filter_excludes_old_events: Verifies that the since
timestamp filter propagates correctly through collect, excluding events
before the cutoff while retaining newer ones.
- pipeline_unresolved_refs_have_optional_iid: Tests the Option<i64>
target_iid on UnresolvedRef by creating cross-project references both
with and without known IIDs.
- shared_resolve_entity_ref_scoping: Unit tests for the new shared
resolve_entity_ref helper covering project-scoped lookup, unscoped
lookup, wrong-project rejection, unknown entity types, and nonexistent
entity IDs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Follows up on the resolve_entity_ref extraction by updating all three
pipeline stages to consume the shared helper and removing their local
duplicates (~75 lines of dead code eliminated).
timeline_seed.rs:
- Switch from local resolve_entity to shared resolve_entity_ref with
explicit Some(proj_id) scoping
- Add tracing::debug for orphaned discussion parents instead of silently
skipping them, aiding debugging when evidence notes go missing
- Use saturating_mul for the over-fetch multiplier to prevent overflow on
pathological max_seeds values
timeline_expand.rs:
- Switch from local resolve_entity_ref to shared version with None
project scoping (cross-project traversal)
- Pass Option<i64> for target_iid in UnresolvedRef construction instead
of unwrap_or(0) sentinel
- Update test assertion to compare against Some(42)
timeline_collect.rs:
- Make entity_id_column return Result instead of silently defaulting to
issue_id for unknown entity types. The previous fallback could produce
incorrect SQL queries that return wrong results rather than failing
- Replace if-let chains in collect_merged_event with exhaustive match
blocks that propagate real DB errors while gracefully handling expected
missing-data cases (QueryReturnedNoRows, NULL merged_at)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The seed, expand, and collect stages each had their own near-identical
resolve_entity_ref helper that converted internal DB IDs to full EntityRef
structs. This duplication made it easy for bug fixes to land in one copy
but not the others.
Extract a single public resolve_entity_ref into timeline.rs with an
optional project_id parameter:
- Some(project_id): scopes the lookup (used by seed, which knows the
project from the FTS result)
- None: unscoped lookup (used by expand, which traverses cross-project
references)
Also changes UnresolvedRef.target_iid from i64 to Option<i64>. Cross-
project references parsed from descriptions may not always carry an IID
(e.g. when the reference is malformed or the target was deleted). The
previous sentinel value of 0 was semantically incorrect since GitLab IIDs
start at 1.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add tests/perf_benchmark.rs with three side-by-side benchmarks that
compare old vs new approaches for the optimizations introduced in the
preceding commits:
- bench_label_insert_individual_vs_batch: measures N individual INSERTs
vs single multi-row INSERT (5k iterations, ~1.6x speedup)
- bench_string_building_old_vs_new: measures format!+push_str vs
writeln! (50k iterations, ~1.9x speedup)
- bench_prepare_vs_prepare_cached: measures prepare vs prepare_cached
(10k iterations, ~1.6x speedup)
Each benchmark verifies correctness (both approaches produce identical
output) and uses std::hint::black_box to prevent dead-code
elimination. Run with: cargo test --test perf_benchmark -- --nocapture
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Change OllamaClient::embed_batch to accept &[&str] instead of
Vec<String>. The EmbedRequest struct now borrows both model name and
input texts, eliminating per-batch cloning of chunk text (up to 32KB
per chunk x 32 chunks per batch). Serialization output is identical
since serde serializes &str and String to the same JSON.
In hybrid search, defer the RrfResult->HybridResult mapping until
after filter+take, so only `limit` items (typically 20) are
constructed instead of up to 1,500 at RECALL_CAP. Also switch
filtered_ids to into_iter() to avoid an extra .copied() pass.
Switch FTS search_fts from prepare() to prepare_cached() for statement
reuse across repeated searches. Benchmarked at ~1.6x faster.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace individual INSERT-per-label and INSERT-per-path loops in
upsert_document_inner with single multi-row INSERT statements. For a
document with 5 labels, this reduces 5 SQL round-trips to 1.
Replace format!()+push_str() with writeln!() in all three document
extractors (issue, MR, discussion). writeln! writes directly into the
String buffer, avoiding the intermediate allocation that format!
creates. Benchmarked at ~1.9x faster for string building and ~1.6x
faster for batch inserts (measured over 5k iterations in-memory).
Also switch get_existing_hash from prepare() to prepare_cached() since
it is called once per document during regeneration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Critical fix: Migration 015 exists on disk but was not registered in db.rs.
All beads referencing "migration 015 for mr_file_changes" corrected to migration
016. bd-1oo retitled to reflect dual responsibility (register 015 + create 016).
bd-2y79 renumbered from 016 to 017.
Revised beads: bd-1oo, bd-2yo, bd-1yx, bd-2y79, bd-1nf, bd-2f2, bd-ike,
bd-14q, bd-1ht, bd-z94, bd-2n4.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration 015 adds merge_commit_sha/squash_commit_sha to merge_requests
(Gate 4/5 prerequisites), closes_issues_synced_for_updated_at watermark
for incremental sync, and the missing idx_label_events_label index.
The MR transformer and ingestion pipeline now populate commit SHAs during
sync. The orchestrator uses watermark-based filtering for closes_issues
jobs instead of re-enqueuing all MRs every sync.
The Phase B PRD is updated to match the actual codebase: corrected
migration numbering (011-015), documented nullable label/milestone
fields (migration 012), watermark patterns (013), observability
infrastructure (014), simplified source_method values, and updated
entity_references schema to match implementation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix 6 beads (bd-1ht, bd-2n4, bd-9dd, bd-z94, bd-1yx, bd-3as) that
incorrectly claimed merge_requests has NO merged_at column. Migration
006 defines it and it's used throughout the codebase. Updated SQL
ordering to use COALESCE(merged_at, updated_at).
- Fix bd-32q: build_safe_fts_query() -> to_fts_query(query, FtsQueryMode::Safe)
(actual function in src/search/fts.rs)
- Add Rust JSON struct examples to bd-dty (robot mode output)
- Add edge cases section to bd-jec (config flag)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
