feat(queue): Add generic dependent fetch queue with exponential backoff
New module src/core/dependent_queue.rs provides job queue operations against the pending_dependent_fetches table. Designed for second-pass fetches that depend on primary entity ingestion (resource events, MR close references, MR file diffs). Queue operations: - enqueue_job: Idempotent INSERT OR IGNORE keyed on the UNIQUE (project_id, entity_type, entity_iid, job_type) constraint. Returns bool indicating whether the row was actually inserted. - claim_jobs: Two-phase claim — SELECT available jobs (unlocked, past retry window) then UPDATE locked_at in batch. Orders by enqueued_at ASC for FIFO processing within a job type. - complete_job: DELETE the row on successful processing. - fail_job: Increments attempts, calculates exponential backoff (30s * 2^(attempts-1), capped at 480s), sets next_retry_at, clears locked_at, and records the error message. Reads current attempts via query with unwrap_or(0) fallback for robustness. - reclaim_stale_locks: Clears locked_at on jobs locked longer than a configurable threshold, recovering from worker crashes. - count_pending_jobs: GROUP BY job_type aggregation for progress reporting and stats display. Registers both events_db and dependent_queue in src/core/mod.rs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -3,7 +3,9 @@
|
||||
pub mod backoff;
|
||||
pub mod config;
|
||||
pub mod db;
|
||||
pub mod dependent_queue;
|
||||
pub mod error;
|
||||
pub mod events_db;
|
||||
pub mod lock;
|
||||
pub mod paths;
|
||||
pub mod payloads;
|
||||
|
||||
Reference in New Issue
Block a user