perf: Configurable rate limit, 429 auto-retry, concurrent project ingestion
The sync pipeline was bottlenecked at 10 req/s (hardcoded) with sequential project processing and no retry on rate limiting. These changes target 3-5x throughput improvement. Rate limit configuration: - Add requestsPerSecond to SyncConfig (default 30.0, was hardcoded 10) - Pass configured rate through to GitLabClient::new from ingest - Floor rate at 0.1 rps in RateLimiter::new to prevent panic on Duration::from_secs_f64(1.0 / 0.0) — now reachable via user config 429 auto-retry: - Both request() and request_with_headers() retry up to 3 times on HTTP 429, respecting the retry-after header (default 60s) - Extract parse_retry_after helper, reused by handle_response fallback - After exhausting retries, the 429 error propagates as before - Improved JSON decode errors now include a response body preview Concurrent project ingestion: - Derive Clone on GitLabClient (cheap: shares Arc<Mutex<RateLimiter>> and reqwest::Client which is already Arc-backed) - Restructure project loop to use futures::stream::buffer_unordered with primary_concurrency (default 4) as the parallelism bound - Each project gets its own SQLite connection (WAL mode + busy_timeout handles concurrent writes) - Add show_spinner field to IngestDisplay to separate the per-project spinner from the sync-level stage spinner - Error aggregation defers failures: all successful projects get their summaries printed and results counted before returning the first error - Bump dependentConcurrency default from 2 to 8 for discussion prefetch Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -51,6 +51,9 @@ pub struct SyncConfig {
|
||||
#[serde(rename = "dependentConcurrency")]
|
||||
pub dependent_concurrency: u32,
|
||||
|
||||
#[serde(rename = "requestsPerSecond")]
|
||||
pub requests_per_second: f64,
|
||||
|
||||
#[serde(rename = "fetchResourceEvents", default = "default_true")]
|
||||
pub fetch_resource_events: bool,
|
||||
}
|
||||
@@ -67,7 +70,8 @@ impl Default for SyncConfig {
|
||||
heartbeat_interval_seconds: 30,
|
||||
cursor_rewind_seconds: 2,
|
||||
primary_concurrency: 4,
|
||||
dependent_concurrency: 2,
|
||||
dependent_concurrency: 8,
|
||||
requests_per_second: 30.0,
|
||||
fetch_resource_events: true,
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user