fix: throttle background prefetch and add circuit breaker (#67) - #72
Merged
Conversation
…humbnails The v1.2.0 background prefetch fix made things worse for users with consistently failing TV connections (Nitrowolf's 525-artwork catalog): - 27 batches each spawned 20 individual _tv_op calls (525 total) - Each call tried 3 attempts × 18s timeout = up to 58s per thumbnail - All serialized through one lock = hours of lock contention - Frontend retries piled up behind the same lock - Observed: 69-131s response times, endless ConnectionFailure spam Three changes to fix: 1. Circuit breaker: if get_thumbnail_list failed in the last 60s, skip it entirely and return cache state immediately (0ms instead of 8-60s) 2. Throttled prefetch: single attempt per thumbnail, 0.5s pause between successes, exponential backoff on failure (2s, 4s, 6s...), abort after 5 consecutive failures 3. Single prefetch task: don't spawn additional background tasks while one is already running (prevents request pile-up) Closes #67
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fix: Thumbnail prefetch hammering TV into ConnectionFailure spiral
What went wrong with v1.2.0
The background prefetch fix (PR #70) made things worse for users with consistently failing
get_thumbnail_list. Nitrowolf's v1.2.0 logs showed:POST /api/thumbnails 200 69.59s— response took 69 secondsPOST /api/thumbnails 200 131.01s— response took 131 secondsTV op attempt 1/3 failed (ConnectionFailure)per secondRoot cause: 27 batches of 20 IDs each spawned background prefetch tasks. Each task tried 20 individual
_tv_opcalls with 3 attempts × 18s timeout each, all serialized through one lock. That's potentially 525 × 3 × 18s = 8+ hours of lock contention. Meanwhile, the frontend's 8 retries also queued up behind the same lock, and each retry re-attempted the failingget_thumbnail_listcall (another 8s per try).Three fixes
Circuit breaker on
get_thumbnail_list: if it failed in the last 60s, skip it entirely and return the cache state in 0ms (vs 8-60s blocking)Throttled prefetch: single attempt per thumbnail (
attempts=1), 0.5s pause between successes, exponential backoff on failure (2s, 4s, 6s…), abort after 5 consecutive failuresSingle prefetch task: don't spawn additional background tasks while one is already running (prevents 27 prefetch tasks piling up)
Expected behavior for Nitrowolf
get_thumbnail_listfails → returns immediately withfallback: true→ spawns ONE prefetch taskCloses #67