# stream

> stream is a Jetstream V2-compatible firehose for the AT Protocol with the
> whole network archived behind it: sealed segments back to seq 1, appended
> live from bsky.network. Subscribe like any Jetstream, or read history out
> of the archive by segment. One instance at https://stream.waow.tech
> (experimental, no SLA). Live counts: https://stream.waow.tech/status

Things to remember:

- **the archive API needs a bearer key** (as of 2026-08-13, matching Bluesky
  Protocol Services' hosted Jetstream v2 instances): listSegments, getSegment,
  getBlock, and planSnapshot answer 401 without `Authorization: Bearer <key>`.
  The live tail (/subscribe, subscribeEvents) and getZstdDictionary stay open,
  no key ever. Keys are granted on request — ask @zzstoatzz.io.

- cursor semantics on /subscribe split by magnitude: >= 1e15 is a `time_us`
  microsecond timestamp, below that a local sequence number. cursor 0 means
  oldest retained; future cursors go silently live; **too-old cursors clamp
  silently to oldest retained** — check the first event you receive if gaps
  matter.
- the subscribe replay window is bounded (~1.5 days; see `cursor lookback`
  on /status). deeper history is the archive API's job, not subscribe's.
- archive blocks are PLAIN zstd frames — the getZstdDictionary endpoint is
  only for subscribeEvents wire compression.
- the archive's retrieval unit is a ~4096-event block: right for dense
  collections, whole-network drains, per-DID history, and deleted or
  historical records; wrong for sparse scattered collections (measured
  ~1,300x read amplification — use a collection directory such as
  lightrail.microcosm.blue plus per-repo listRecords instead, then follow
  live here).
- trust model: record CBOR (v2) is CID-checkable; commit signatures are
  NOT re-verifiable from any jetstream-class stream, this one included.
  need cryptographic authenticity? consume a relay's subscribeRepos.
- coverage is relay-known repos, not deep-crawl: bridged accounts whose
  repos no PDS serves are absent.
- a restart pauses the live tail briefly (minutes, not hours; the archive
  keeps serving throughout); the tail replays its whole gap from the relay
  on return, gap-free. cursor consumers resume smoothly across it.

## Surfaces

- `wss://stream.waow.tech/subscribe`: Jetstream-compatible JSON websocket — `wantedCollections` (prefix patterns, cap 100),
  `wantedDids`, `cursor`
- `wss://stream.waow.tech/xrpc/network.bsky.jetstream.subscribeEvents`:
  proposal-0015 lexicon frames —
  strict per-event sequence cursors, raw record CBOR, #sync events
- listSegments (bearer key required):
  sealed-segment metadata (seq ranges, checksums, event counts), paginated
- `getSegment?name=<seg_....jss>`: one sealed segment (~277 MB),
  Range/206 + ETag (etag = checksum)
- `getBlock?segment=<name>&blockIndex=<n>`: one compressed block
- planSnapshot (POST, bearer key required; json `{"collections":[...]}`,
  `{"dids":[...]}`, and/or `{"kinds":["commit",...]}` — kinds filtering per
  upstream v0.2.0):
  returns `plannedThroughSeq` + the segment/block list for a range —
  gap-free recovery: plan from your last seq, fetch, resume subscribeEvents
  at `plannedThroughSeq`
- [/status](https://stream.waow.tech/status): plain-text field report for
  scripts (browsers get HTML — don't parse that variant)
- [/api/endpoints.json](https://stream.waow.tech/api/endpoints.json): this
  endpoint list, machine-readable (the homepage templates from it)
- [/stats](https://stream.waow.tech/stats) ·
  [/metrics](https://stream.waow.tech/metrics): human dashboard ·
  prometheus series behind it

## Source docs

- [jss format spec](https://tangled.org/zat.dev/stream/blob/main/docs/jss-format-v1.md):
  byte-exact segment/block layout — everything needed to decode
- [subscribe protocol](https://tangled.org/zat.dev/stream/blob/main/docs/upstream-subscribe-protocol.md):
  exact v1/v2 wire semantics and divergences
- [working decoder](https://tangled.org/zat.dev/stream/blob/main/examples/backfill-collection-sqlite.py):
  plan → fetch → decode → sqlite, runnable
- [architecture](https://tangled.org/zat.dev/stream/blob/main/docs/architecture.md):
  how the subsystems fit
- [source](https://tangled.org/zat.dev/stream): zig; wire-compatible
  rewrite of bluesky-social/jetstream V2, proven against the official Go
  client. running build: the `build` line on /status

## Optional

- [semantic parity audit](https://tangled.org/zat.dev/stream/blob/main/docs/semantic-parity.md):
  row-by-row proof ledger against the upstream pin
- [expected event rate](https://relay-eval.waow.tech/api/rate-profile):
  hour-of-day rate quantiles for judging whether the tail is current