Replace serde_json::to_string(&output).unwrap() with match-based error
handling across all robot-mode JSON printers. On serialization failure,
the error is now written to stderr instead of panicking. This hardens
the CLI against unexpected Serialize failures in production.
Affected commands: count (2), embed, generate-docs, ingest (2), search,
stats, sync (2), sync-status, timeline.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all console::style() calls in command modules with the centralized
Theme API and render:: utility functions. This ensures consistent color
behavior across the entire CLI, proper NO_COLOR/--color never support via
the LoreRenderer singleton, and eliminates duplicated formatting code.
Changes per module:
- count.rs: Theme for table headers, render::format_number replacing local
duplicate. Removed local format_number implementation.
- doctor.rs: Theme::success/warning/error for check status symbols and
messages. Unicode escapes for check/warning/cross symbols.
- drift.rs: Theme::bold/error/success for drift detection headers and
status messages.
- embed.rs: Compact output format — headline with count, zero-suppressed
detail lines, 'nothing to embed' short-circuit for no-op runs.
- generate_docs.rs: Same compact pattern — headline + detail + hint for
next step. No-op short-circuit when regenerated==0.
- ingest.rs: Theme for project summaries, sync status, dry-run preview.
All console::style -> Theme replacements.
- list.rs: Replace comfy-table with render::LoreTable for issue/MR listing.
Remove local colored_cell, colored_cell_hex, format_relative_time,
truncate_with_ellipsis, and format_labels (all moved to render.rs).
- list_tests.rs: Update test assertions to use render:: functions.
- search.rs: Add render_snippet() for FTS5 <mark> tag highlighting via
Theme::bold().underline(). Compact result layout with type badges.
- show.rs: Theme for entity detail views, delegate format_date and
wrap_text to render module.
- stats.rs: Section-based layout using render::section_divider. Compact
middle-dot format for document counts. Color-coded embedding coverage
percentage (green >=95%, yellow >=50%, red <50%).
- sync.rs: Compact sync summary — headline with counts and elapsed time,
zero-suppressed detail lines, visually prominent error-only section.
- sync_status.rs: Theme for run history headers, removed local
format_number duplicate.
- timeline.rs: Theme for headers/footers, render:: for date/truncate,
standard format! padding replacing console::pad_str.
- who.rs: Theme for all expert/workload/active/overlap/review output
modes, render:: for relative time and truncation.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Robot mode consistency improvements across all command output:
Timing:
- Every robot JSON response now includes meta.elapsed_ms measuring
wall-clock time from command start to serialization. Agents can use
this to detect slow queries and tune --limit or --project filters.
Field selection (--fields):
- print_list_issues_json and print_list_mrs_json accept an optional
fields slice that prunes each item in the response array to only
the requested keys. A "minimal" preset expands to
[iid, title, state, updated_at_iso] for token-efficient agent scans.
- filter_fields and expand_fields_preset live in the new
src/cli/robot.rs module alongside RobotMeta.
Actionable error recovery:
- LoreError gains an actions() method returning concrete shell commands
an agent can execute to recover (e.g. "ollama serve" for
OllamaUnavailable, "lore init" for ConfigNotFound).
- RobotError now serializes an "actions" array (empty array omitted)
so agents can parse and offer one-click fixes.
Envelope consistency:
- show issue/MR JSON responses now use the standard
{"ok":true,"data":...,"meta":...} envelope instead of bare data,
matching all other commands.
Files: src/cli/robot.rs (new), src/core/error.rs,
src/cli/commands/{count,embed,generate_docs,ingest,list,show,stats,sync_status}.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Embedding pipeline improvements building on the concurrent batching
foundation:
- Track docs_embedded vs chunks_embedded separately. A document counts
as embedded only when ALL its chunks succeed, giving accurate
progress reporting. The sync command reads docs_embedded for its
document count.
- Reuse a single Vec<u8> buffer (embed_buf) across all store_embedding
calls instead of allocating per chunk. Eliminates ~3KB allocation per
768-dim embedding.
- Detect and record errors when Ollama silently returns fewer
embeddings than inputs (batch mismatch). Previously these dropped
chunks were invisible.
- Improve retry error messages: distinguish "retry returned unexpected
result" (wrong dims/count) from "retry request failed" (network
error) instead of generic "chunk too large" message.
- Convert all hot-path SQL from conn.execute() to prepare_cached() for
statement cache reuse (clear_document_embeddings, store_embedding,
record_embedding_error).
- Record embedding_metadata errors for empty documents so they don't
appear as perpetually pending on subsequent runs.
- Accept concurrency parameter (configurable via config.embedding.concurrency)
instead of hardcoded EMBED_CONCURRENCY=2.
- Add schema version pre-flight check in embed command to fail fast
with actionable error instead of cryptic SQL errors.
- Fix --retry-failed to use DELETE instead of UPDATE. UPDATE clears
last_error but the row still matches config params in the LEFT JOIN,
making the doc permanently invisible to find_pending_documents.
DELETE removes the row entirely so the LEFT JOIN returns NULL.
Regression test added (old_update_approach_leaves_doc_invisible).
- Add chunking forward-progress guard: after floor_char_boundary()
rounds backward, ensure start advances by at least one full
character to prevent infinite loops on multi-byte sequences
(box-drawing chars, smart quotes). Test cases cover the exact
patterns that caused production hangs on document 18526.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Three fixes to the embedding pipeline:
1. Concurrent HTTP batching: fire EMBED_CONCURRENCY (2) Ollama requests
in parallel via join_all, then write results serially to SQLite.
~2x throughput improvement on GPU-bound workloads.
2. UTF-8 boundary safety: all computed byte offsets in split_into_chunks
(paragraph/sentence/word break finders + overlap advance) now use
floor_char_boundary() to prevent panics on multi-byte characters
like smart quotes and non-breaking spaces.
3. CHUNK_MAX_BYTES reduced from 6000 to 1500 to fit nomic-embed-text's
actual 2048-token context window, eliminating context-length retry
storms that were causing 10x slowdowns.
Also threads ShutdownSignal through embed pipeline for graceful Ctrl+C.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Removes module-level doc comments (//! lines) and excessive inline doc
comments that were duplicating information already evident from:
- Function/struct names (self-documenting code)
- Type signatures (the what is clear from types)
- Implementation context (the how is clear from code)
Affected modules:
- cli/* - Removed command descriptions duplicating clap help text
- core/* - Removed module headers and obvious function docs
- documents/* - Removed extractor/regenerator/truncation docs
- embedding/* - Removed pipeline and chunking docs
- gitlab/* - Removed client and transformer docs (kept type definitions)
- ingestion/* - Removed orchestrator and ingestion docs
- search/* - Removed FTS and vector search docs
Philosophy: Code should be self-documenting. Comments should explain
"why" (business decisions, non-obvious constraints) not "what" (which
the code itself shows). This change reduces noise and maintenance burden
while keeping the codebase just as understandable.
Retains comments for:
- Non-obvious business logic
- Important safety invariants
- Complex algorithm explanations
- Public API boundaries where generated docs matter
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The sync command's stage spinners now show real-time aggregate progress
for each pipeline phase instead of static "syncing..." messages.
- Add `progress_callback` parameter to `run_embed` and
`run_generate_docs` so callers can receive `(processed, total)` updates
- Add `stage_bar` parameter to `run_ingest` for aggregate progress
across concurrently-ingested projects using shared AtomicUsize counters
- Update `stage_spinner` to use `{prefix}` for the `[N/M]` label,
allowing `{msg}` to be updated independently with progress details
- Thread `ProgressBar` clones into each concurrent project task so
per-entity progress (fetch, discussions, events) is reflected on the
aggregate spinner
- Pass `None` for progress callbacks at standalone CLI entry points
(handle_ingest, handle_generate_docs, handle_embed) to preserve
existing behavior when commands are run outside of sync
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Automated formatting and lint corrections from parallel agent work:
- cargo fmt: import reordering (alphabetical), line wrapping to respect
max width, trailing comma normalization, destructuring alignment,
function signature reformatting, match arm formatting
- clippy (pedantic): Range::contains() instead of manual comparisons,
i64::from() instead of `as i64` casts, .clamp() instead of
.max().min() chains, let-chain refactors (if-let with &&),
#[allow(clippy::too_many_arguments)] and
#[allow(clippy::field_reassign_with_default)] where warranted
- Removed trailing blank lines and extra whitespace
No behavioral changes. All existing tests pass unmodified.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add --full / --no-full flag pair to EmbedArgs with overrides_with
semantics matching the existing flag pattern. When active, atomically
DELETEs all embedding_metadata and embeddings before re-embedding.
- Thread the full flag through run_embed -> run_sync so that
'lore sync --full' triggers a complete re-embed alongside the full
re-ingest it already performed.
- Add indicatif spinners to sync stages with dynamic stage numbering
that adjusts when --no-docs or --no-embed skip stages. Spinners are
hidden in robot mode.
- Update robot-docs manifest to advertise the new --full flag on the
embed command.
- Replace hardcoded schema version 9 in health check with the
LATEST_SCHEMA_VERSION constant from db.rs.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Extends the CLI with six new commands that complete the search pipeline:
- lore search <QUERY>: Hybrid search with mode selection (lexical,
hybrid, semantic), rich filtering (--type, --author, --project,
--label, --path, --after, --updated-after), result limits, and
optional explain mode showing RRF score breakdowns. Safe FTS mode
sanitizes user input; raw mode passes through for power users.
- lore stats: Document and index statistics with optional --check
for integrity verification and --repair to fix inconsistencies
(orphaned documents, missing FTS entries, stale dirty queue items).
- lore embed: Generate vector embeddings via Ollama. Supports
--retry-failed to re-attempt previously failed embeddings.
- lore generate-docs: Drain the dirty queue to regenerate documents.
--full seeds all entities for complete rebuild. --project scopes
to a single project.
- lore sync: Full pipeline orchestration (ingest issues + MRs,
generate-docs, embed) with --no-embed and --no-docs flags for
partial runs. Reports per-stage results and total elapsed time.
- lore health: Quick pre-flight check (config exists, DB exists,
schema current). Returns exit code 1 if unhealthy. Designed for
agent pre-flight scripts.
- lore robot-docs: Machine-readable command manifest for agent
self-discovery. Returns all commands, flags, examples, exit codes,
and recommended workflows as structured JSON.
Also enhances lore init with --gitlab-url, --token-env-var, and
--projects flags for fully non-interactive robot-mode initialization.
Fixes init's force/non-interactive precedence logic and adds JSON
output for robot mode.
Updates all command files for the GiError -> LoreError rename.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
