feat(observability): Add metrics, logging, and sync-run core modules
Introduce the foundational observability layer for the sync pipeline: - MetricsLayer: Custom tracing subscriber layer that captures span timing and structured fields, materializing them into a hierarchical Vec<StageTiming> tree for robot-mode performance data output - logging: Dual-layer subscriber infrastructure with configurable stderr verbosity (-v/-vv/-vvv) and always-on JSON file logging with daily rotation and configurable retention (default 30 days) - SyncRunRecorder: Compile-time enforced lifecycle recorder for sync_runs table (start -> succeed|fail), with correlation IDs and aggregate counts - LoggingConfig: New config section for log_dir, retention_days, and file_logging toggle - get_log_dir(): Path helper for log directory resolution - is_permanent_api_error(): Distinguish retryable vs permanent API failures (only 404 is truly permanent; 403/auth errors may be environmental) Database changes: - Migration 013: Add resource_events_synced_for_updated_at watermark columns to issues and merge_requests tables for incremental resource event sync - Migration 014: Enrich sync_runs with run_id correlation ID, aggregate counts (total_items_processed, total_errors), and run_id index - Wrap file-based migrations in savepoints for rollback safety Dependencies: Add uuid (run_id generation), tracing-appender (file logging) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
10
migrations/013_resource_event_watermarks.sql
Normal file
10
migrations/013_resource_event_watermarks.sql
Normal file
@@ -0,0 +1,10 @@
|
||||
-- Migration 013: Add resource event sync watermarks
|
||||
-- Mirrors the discussions_synced_for_updated_at pattern so that only entities
|
||||
-- whose updated_at exceeds the last resource event sync get re-enqueued.
|
||||
|
||||
ALTER TABLE issues ADD COLUMN resource_events_synced_for_updated_at INTEGER;
|
||||
ALTER TABLE merge_requests ADD COLUMN resource_events_synced_for_updated_at INTEGER;
|
||||
|
||||
-- Update schema version
|
||||
INSERT INTO schema_version (version, applied_at, description)
|
||||
VALUES (13, strftime('%s', 'now') * 1000, 'Add resource event sync watermarks to issues and merge_requests');
|
||||
12
migrations/014_sync_runs_enrichment.sql
Normal file
12
migrations/014_sync_runs_enrichment.sql
Normal file
@@ -0,0 +1,12 @@
|
||||
-- Migration 014: sync_runs enrichment for observability
|
||||
-- Adds correlation ID and aggregate counts for queryable sync history
|
||||
|
||||
ALTER TABLE sync_runs ADD COLUMN run_id TEXT;
|
||||
ALTER TABLE sync_runs ADD COLUMN total_items_processed INTEGER DEFAULT 0;
|
||||
ALTER TABLE sync_runs ADD COLUMN total_errors INTEGER DEFAULT 0;
|
||||
|
||||
-- Index for correlation queries (find run by run_id from logs)
|
||||
CREATE INDEX IF NOT EXISTS idx_sync_runs_run_id ON sync_runs(run_id);
|
||||
|
||||
INSERT INTO schema_version (version, applied_at, description)
|
||||
VALUES (14, strftime('%s', 'now') * 1000, 'Sync runs enrichment for observability');
|
||||
Reference in New Issue
Block a user