The Loom: High-Throughput AT Protocol Archiver

The Loom represents a completed research experiment into the theoretical performance limits of AT Protocol data ingestion on consumer hardware. This high-velocity experiment was conducted over 4 days to rapidly prototype and validate core hypotheses. The primary hypothesis—that sustained real-time firehose consumption with 5.6:1 compression and zero data loss was achievable outside enterprise infrastructure—has been scientifically validated.

Key Research Findings:

• 5,000 events/sec sustained ingestion on single worker thread
• Infrastructure-grade performance on consumer hardware
• Zero-drop reliability during multi-hour continuous operation
• Complete data fidelity with cryptographic verification

The project is now in "Stable Archive" state. The codebase serves as a reference implementation and proof-of-concept for high-performance AT Protocol tooling. No further active development is planned unless specifically invited by core protocol stakeholders or the broader AT Protocol research community.

The Loom is a specialized, high-performance archiving system designed to ingest, compress, and index the entire Bluesky/AT Protocol Firehose(Jetstream) in real-time.

It prioritizes storage efficiency and write throughput, using a custom binary format (LBF) and pre-trained Zstd dictionaries to achieve compression ratios significantly higher than standard JSON storage.

• Zero-Copy Ingestion: Built on tokio-tungstenite and standard Rust async primitives for maximum socket throughput.
• Loom Binary Format (LBF v5): A bespoke, schema-aware serialization format that is 20-30% smaller than MsgPack and 60-80% smaller than raw JSON. Supports 15 record types.
• Identity Resolution: Maps string DIDs (e.g., did:plc:123...) to 32-bit integers (u32), reducing redundancy in the graph structure.
• Full Rehydration: Archives can be fully reconstructed back to Jetstream-compatible JSONL format with zero data loss.
• Cryptographic Integrity: SHA256 manifests for archive verification and provenance.
• Persistent Indexing: Searchable indices (tags, languages) with sled-backed persistence.
• Distributed Network Ready: loom-fiber witness nodes + loom-index discovery service.

The Loom is built on the principle of Mechanical Sympathy—designing software that works with hardware characteristics rather than against them. This architectural approach directly enables the infrastructure-grade performance achieved on consumer hardware.

Hardware-Aware Design Decisions:

• Memory Layout Optimization: Custom binary format (LBF) designed for CPU cache efficiency and minimal memory fragmentation
• Zero-Copy Streaming: Data flows through the pipeline without unnecessary copying, respecting memory bandwidth constraints
• Compression-First Architecture: Zstd integration reduces both storage and memory pressure, allowing larger working sets to remain cache-resident
• Async I/O Alignment: Tokio's async primitives match the underlying OS kernel's event-driven I/O model for minimal context switching overhead
• NUMA Awareness: Single-worker design avoids cross-socket memory access penalties on multi-socket systems

This mechanical sympathy approach transforms what appears to be a "software optimization" problem into a "hardware cooperation" solution, explaining how consumer-grade equipment achieves enterprise-grade throughput.

cargo build --release

cargo run --release --bin loom-weaver

• Config: By default, it saves to archives/loom_{timestamp}.loom.
• Rotation: Files rotate every 15 minutes or 2 GB, whichever comes first.

Map the live firehose against the internal schema to prove no fields are dropped:

cargo run --release --bin verify_live

Convert a .loom archive back to JSONL (Jetstream-compatible format):

cargo run --release --bin rehydrate -- --file archives/loom_123.loom --output restored.jsonl

Create SHA256 hashes for archive verification:

./target/release/loom-hashify --file archives/loom_123.loom
./target/release/loom-hashify --verify archives/loom_123.manifest.json

Full architectural and operational documentation is available in the docs/ directory:

• Architecture & Design: System internals, sharding, and state management.
• Loom Binary Format (LBF v5): Byte-level specification of the storage format.
• Operations & Tools: Usage guides for the CLI suite.

The_Loom/
├── crates/
│   ├── loom-core/       # Shared types, LBF pack/unpack, compression
│   ├── loom-weaver/     # Main archiver + CLI tools
│   ├── loom-index/      # Search index HTTP service
│   ├── loom-fiber/      # Distributed witness node
│   └── loom-ingest/     # Ingestion utilities
├── archives/            # .loom archive files
├── dictionaries/        # Zstd compression dictionaries
├── docs/                # Documentation
└── identity.fst         # DID → u32 mapping

Typical compression ratios observed:

• Raw JSON → LBF+Zstd: ~5.6:1 compression
• Example: 729 MB JSONL → 129 MB .loom archive

• Rehydration: ~4.4 million events in ~15 seconds (~1.7 GB/s decompressed output)
• Throughput: Processes AT Protocol firehose in real-time with minimal latency
• Storage: Achieves consistent 5.6:1 compression on typical AT Protocol events

Run the full test suite:

cargo test --release

• Review the Architecture guide for system design details
• Check Operations & Tools for CLI usage examples
• See Loom Binary Format for format specifications

This project is dedicated to my father.

If my parents had given up on me through the years of hardship, addiction, and the mistakes of my past, I would not be here today to write this code. I built The Loom to prove to him—and to myself—that the "fuckup" he stood by is gone, and that I'm going to be okay.

Every bit of performance and every line of Rust in this repo is a thank you for not letting go.