Research: WebRecorder/WACZ archival capture path (re-sited from watcher#4) #64

Re-sited from watcher#4 (opened 2026-03-21, before the #241 fetch-boundary cutover). The original targeted src/core/fetchers/webrecorder.py and a Watcher-side StorageBackend — neither exists. Watcher makes no origin request and holds no blob storage. Body rewritten against Replicator's actual seams.

Summary

Investigate a WebRecorder-based capture path (Browsertrix Crawler / ArchiveWeb.page) producing WACZ/WARC alongside the ordinary blob, for archival-quality preservation of a rendered source.

Why here, and on which seam

Capture is fetch-side. ContentReplicateCommand carries a blob_uri — the replicate path copies bytes the fetch path already produced (src/worker/replicate.py, AsyncGcsDriver.create_if_absent). It never captures. So a WACZ producer cannot live behind content.replicate; it is another implementation of the Fetcher Protocol in src/worker/handler.py, the same seam as #63.

That in turn means WACZ, once produced, is a permanent-store object flowing out through the ordinary replicate path — provider / credentials_alias / destination / media_type / object_options on ContentReplicateCommand, under the T3 alias-root guard. Nothing new needed on that side beyond a media type.

Open questions — these are the issue, not the implementation

Filed with the scoping concerns intact, because they were never resolved and they decide whether this is worth building:

1. What consumes a WACZ? Watcher's pipeline wants a fingerprint and diffable text. The original issue's own scope line — "determine how to extract diffable text content from WARC records" — proposes a format and then asks how to recover the only artefact downstream actually reads. If the answer is "the WACZ is for humans/replay, the diffable text comes from the ordinary blob", say so explicitly: that makes this a second artefact per fetch, not a substitute, and doubles the byte path's output.
2. Does Headless-browser fetch path for JS-rendered pages (re-sited from watcher#3) #63 dominate this? If a headless-browser fetcher lands, Browsertrix's marginal value is archival fidelity and replay, not rendering. Worth confirming that fidelity is actually wanted before paying for a crawler runtime.
3. Retention. The fetch path's blobs are deliberately transient — TTL sweeper, BlobUsage disk ceiling, host-local file:// blob_uri by contract. A WACZ is large and its whole point is permanence. It must not land in the temp tree on the normal expiry schedule; the hand-off to a permanent store has to be part of the design, not an afterthought.
4. Resource ceilings. Browsertrix is a crawler, not a request. Concurrency, memory, disk, and lifetime bounds on a shared VM — strictly heavier than Headless-browser fetch path for JS-rendered pages (re-sited from watcher#3) #63's.
5. Trigger. Same unresolved selection question as Headless-browser fetch path for JS-rendered pages (re-sited from watcher#3) #63: ContentFetchCommand (co-core 0.10.0) has no capture-mode field. Archival capture is plausibly not per-fetch at all but per-info_item_rep_spec_id on a slower cadence, which would be a different command shape entirely.

Scope (if the questions above resolve in favour)

• Research Browsertrix Crawler / ArchiveWeb.page for programmatic invocation.
• Capture path implementing the Fetcher Protocol, or a distinct command if (5) says per-RepSpec.
• WACZ → permanent store via content.replicate, with a media type and object options that survive the T3 destination guard.
• Text extraction from WARC records, per (1).
• Tests; one integration capture against a known target.

Out of scope

• Headless-browser rendering for change detection (Headless-browser fetch path for JS-rendered pages (re-sited from watcher#3) #63).
• Adaptive escalation (watcher#5, being evaluated).

Provenance

Original: CannObserv/watcher#4