feat(sync): concurrent drains, atomic watermarks, graceful Ctrl+C shutdown

Three fixes to the sync pipeline:

1. Atomic watermarks: wrap complete_job + update_watermark in a single
   SQLite transaction so crash between them can't leave partial state.

2. Concurrent drain loops: prefetch HTTP requests via join_all (batch
   size = dependent_concurrency), then write serially to DB. Reduces
   ~9K sequential requests from ~19 min to ~2.4 min.

3. Graceful shutdown: install Ctrl+C handler via ShutdownSignal
   (Arc<AtomicBool>), thread through orchestrator/CLI, release locked
   jobs on interrupt, record sync_run as "failed".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Taylor Eernisse
2026-02-06 11:22:04 -05:00
parent 32783080f1
commit 405e5370dc
9 changed files with 536 additions and 92 deletions

63
src/core/shutdown.rs Normal file
View File

@@ -0,0 +1,63 @@
use std::sync::Arc;
use std::sync::atomic::{AtomicBool, Ordering};
/// A cooperative cancellation token for graceful shutdown.
///
/// Clone-able and cheaply checkable from any thread or async task.
/// When `cancel()` is called (typically from a Ctrl+C signal handler),
/// all clones observe the cancellation via `is_cancelled()`.
#[derive(Clone)]
pub struct ShutdownSignal {
cancelled: Arc<AtomicBool>,
}
impl ShutdownSignal {
pub fn new() -> Self {
Self {
cancelled: Arc::new(AtomicBool::new(false)),
}
}
pub fn cancel(&self) {
self.cancelled.store(true, Ordering::Relaxed);
}
pub fn is_cancelled(&self) -> bool {
self.cancelled.load(Ordering::Relaxed)
}
}
impl Default for ShutdownSignal {
fn default() -> Self {
Self::new()
}
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn signal_starts_uncancelled() {
let signal = ShutdownSignal::new();
assert!(!signal.is_cancelled());
}
#[test]
fn cancel_sets_flag() {
let signal = ShutdownSignal::new();
signal.cancel();
assert!(signal.is_cancelled());
}
#[test]
fn clone_propagates_cancellation() {
let signal = ShutdownSignal::new();
let clone = signal.clone();
signal.cancel();
assert!(
clone.is_cancelled(),
"clone should see cancellation from original"
);
}
}