5fd1ce6905
Issue discussion sync was ~10x slower than MR discussion sync because it
used a fully sequential pattern: fetch one issue's discussions, write to
DB, repeat. MR sync already used a prefetch pattern with concurrent HTTP
requests followed by sequential DB writes.
This commit brings issue discussion sync to parity with MRs:
Architecture (prefetch pattern):
1. HTTP phase: Concurrent fetches via `join_all()` with batch size
controlled by `dependent_concurrency` config (default 8)
2. Transform phase: Normalize discussions and notes during prefetch
3. DB phase: Sequential writes with proper transaction boundaries
Changes:
- gitlab/client.rs: Add `fetch_all_issue_discussions()` to mirror
the existing MR pattern for API consistency
- discussions.rs: Replace `ingest_issue_discussions()` with:
* `prefetch_issue_discussions()` - async HTTP fetch + transform
* `write_prefetched_issue_discussions()` - sync DB writes
* New structs: `PrefetchedIssueDiscussions`, `PrefetchedDiscussion`
- orchestrator.rs: Update `sync_discussions_sequential()` to use
concurrent prefetch for each batch instead of sequential calls
- surgical.rs: Update single-issue surgical sync to use new functions
- mod.rs: Update public exports
Expected improvement: 5-10x speedup on issue discussion sync (from ~50s
to ~5-10s for large projects) due to concurrent HTTP round-trips.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
41 lines
1.4 KiB
Rust
41 lines
1.4 KiB
Rust
pub mod dirty_tracker;
|
|
pub mod discussion_queue;
|
|
pub mod discussions;
|
|
pub mod issues;
|
|
pub mod merge_requests;
|
|
pub mod mr_diffs;
|
|
pub mod mr_discussions;
|
|
pub mod orchestrator;
|
|
pub(crate) mod surgical;
|
|
|
|
pub use discussions::{
|
|
IngestDiscussionsResult, prefetch_issue_discussions, write_prefetched_issue_discussions,
|
|
};
|
|
pub use issues::{IngestIssuesResult, IssueForDiscussionSync, ingest_issues};
|
|
pub use merge_requests::{
|
|
IngestMergeRequestsResult, MrForDiscussionSync, get_mrs_needing_discussion_sync,
|
|
ingest_merge_requests,
|
|
};
|
|
pub use mr_discussions::{IngestMrDiscussionsResult, ingest_mr_discussions};
|
|
/// Format a set of named counters as a compact human-readable summary,
|
|
/// filtering out zero values and joining with middle-dot separators.
|
|
/// Returns `"nothing to update"` when all values are zero.
|
|
pub(crate) fn nonzero_summary(pairs: &[(&str, usize)]) -> String {
|
|
let parts: Vec<String> = pairs
|
|
.iter()
|
|
.filter(|(_, v)| *v > 0)
|
|
.map(|(k, v)| format!("{v} {k}"))
|
|
.collect();
|
|
if parts.is_empty() {
|
|
"nothing to update".to_string()
|
|
} else {
|
|
parts.join(" \u{b7} ")
|
|
}
|
|
}
|
|
|
|
pub use orchestrator::{
|
|
DrainResult, IngestMrProjectResult, IngestProjectResult, ProgressCallback, ProgressEvent,
|
|
ingest_project_issues, ingest_project_issues_with_progress, ingest_project_merge_requests,
|
|
ingest_project_merge_requests_with_progress,
|
|
};
|