Skip to content

fix: throttle background prefetch and add circuit breaker (#67) - #72

Merged
danmunz merged 1 commit into
mainfrom
fix/thumbnail-prefetch-throttle
Jun 2, 2026
Merged

fix: throttle background prefetch and add circuit breaker (#67)#72
danmunz merged 1 commit into
mainfrom
fix/thumbnail-prefetch-throttle

Conversation

@danmunz

@danmunz danmunz commented Jun 2, 2026

Copy link
Copy Markdown
Owner

Fix: Thumbnail prefetch hammering TV into ConnectionFailure spiral

What went wrong with v1.2.0

The background prefetch fix (PR #70) made things worse for users with consistently failing get_thumbnail_list. Nitrowolf's v1.2.0 logs showed:

  • POST /api/thumbnails 200 69.59s — response took 69 seconds
  • POST /api/thumbnails 200 131.01s — response took 131 seconds
  • Dozens of TV op attempt 1/3 failed (ConnectionFailure) per second
  • The TV was completely overwhelmed

Root cause: 27 batches of 20 IDs each spawned background prefetch tasks. Each task tried 20 individual _tv_op calls with 3 attempts × 18s timeout each, all serialized through one lock. That's potentially 525 × 3 × 18s = 8+ hours of lock contention. Meanwhile, the frontend's 8 retries also queued up behind the same lock, and each retry re-attempted the failing get_thumbnail_list call (another 8s per try).

Three fixes

  1. Circuit breaker on get_thumbnail_list: if it failed in the last 60s, skip it entirely and return the cache state in 0ms (vs 8-60s blocking)

  2. Throttled prefetch: single attempt per thumbnail (attempts=1), 0.5s pause between successes, exponential backoff on failure (2s, 4s, 6s…), abort after 5 consecutive failures

  3. Single prefetch task: don't spawn additional background tasks while one is already running (prevents 27 prefetch tasks piling up)

Expected behavior for Nitrowolf

  • First batch request: get_thumbnail_list fails → returns immediately with fallback: true → spawns ONE prefetch task
  • All subsequent batch requests (other 26 batches + retries): circuit breaker skips the TV call → returns cache state in <1ms
  • Background prefetch fetches thumbnails one at a time with pauses, gives up after 5 consecutive failures
  • No more ConnectionFailure spam, no more 69-131s response times

Closes #67

…humbnails

The v1.2.0 background prefetch fix made things worse for users with
consistently failing TV connections (Nitrowolf's 525-artwork catalog):

- 27 batches each spawned 20 individual _tv_op calls (525 total)
- Each call tried 3 attempts × 18s timeout = up to 58s per thumbnail
- All serialized through one lock = hours of lock contention
- Frontend retries piled up behind the same lock
- Observed: 69-131s response times, endless ConnectionFailure spam

Three changes to fix:

1. Circuit breaker: if get_thumbnail_list failed in the last 60s, skip
   it entirely and return cache state immediately (0ms instead of 8-60s)

2. Throttled prefetch: single attempt per thumbnail, 0.5s pause between
   successes, exponential backoff on failure (2s, 4s, 6s...), abort
   after 5 consecutive failures

3. Single prefetch task: don't spawn additional background tasks while
   one is already running (prevents request pile-up)

Closes #67
@danmunz
danmunz merged commit 862cda3 into main Jun 2, 2026
@danmunz
danmunz deleted the fix/thumbnail-prefetch-throttle branch June 2, 2026 19:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

1 participant