Part of Stem. This page defines how a daemon processes data. There is one pipeline, and every blob enters it the same way whether it came from a peer, from the local app, from the CLI or from an HM24 client. The pipeline is driven by Kind descriptors, so a new resource kind is a new document rather than new code.
The goal, restated
Today the daemon has one indexing routine per blob type (Change, Ref, Capability, Comment, Profile, Contact, DagPB), and sync and privacy each know those types by name. The goal recorded on the Seed Resources board was a unified runtime model for resources, permissions and sync, with no special cases for Seed's own data. Stem reaches it with three moves: every resource is declared by a Node blob; every rule a type needs is in its Kind descriptor; and every access question is answered by one evaluator over the authority graph.
The pipeline
flowchart TD
T[Transfer: blobs + claimed scope + sender] --> V[Validate: hash, signature, structure]
V --> A[Authorize: signer's level over the scope's nodes]
A -->|not yet| S[Stash with reason]
A --> I[Index: handler reads the Kind descriptor]
I --> F[Facts]
I --> L[Links]
I --> N[Affected nodes]
N --> R[Re-evaluate: fold state, recompute readers]
R --> B[Materialise blob access]
B --> W[Notify watchers]
B --> P[Policy: keep, pin or allow collection]
S -.->|a dependency or grant arrives| V1. Transfer
Every batch of blobs arrives as a transfer: the blobs, the peer (or local client) that sent them, the account the sender had authenticated as if any, and the scope the sender claims they belong to. A Fetch the daemon made writes a transfer with the scope it was syncing; an accepted Offer writes one with the offered scope; a local Publish writes one with the client's scope or a derived one. The claimed scope is the context the roadmap asked uploads to carry: it says where the blobs are meant to live, so the daemon can check that the sender is allowed to put them there before it does any other work. A transfer whose scope the local policy ignores is refused in full.
2. Validate
Each blob is decoded and checked against things that need no other blob: its bytes hash to the CID the sender named; it is DAG-CBOR or a file codec; if it has a type it is one the daemon knows (the six Stem types, the five HM24 types, or a kind-defined extension); its signature verifies with the zero-sig rule from Signed blobs; its structure matches the type's schema (strict for the six Stem types; for a Snapshot, the value is validated against the schema named in the blob, which must be the kind's schema). A blob that fails is rejected with reason invalid and the whole transfer is not rolled back: unlike today, where one bad blob fails the batch, Stem rejects per blob, because a transfer is a claim by one sender and other blobs in it may still be good.
3. Authorize
Authorization asks the authority graph one question per blob: does the signer hold the level this blob requires over the node it affects? A Node blob requires write on its node, or on its parent when it has no id and so creates the node; the signer may be any key with that level, not only the owner, and the node then belongs to the owner's space. A Change or Snapshot requires that some head Node blob for a node the signer may write names it, directly or through dep and prev; a Change that nobody's Node blob has claimed yet is held until one does. A Grant requires that the signer hold its access over its subject. A Revocation requires issuer, delegate or admin standing. A Group requires nothing but a valid signature. A file requires that some authorized state embeds it.
The blob must also belong to the claimed scope: a Node blob for a node outside the scope's subtree, or a Grant whose subject is elsewhere, is rejected as out-of-scope. This is what stops a sender from using a scope it may write to as a carrier for blobs about a scope it may not.
A blob that fails authorization because of something that may still arrive is stashed, not rejected: the signer's grant has not been seen yet (PermissionDenied), a dependency is missing (FailedPrecondition), a parent node does not exist yet. The blob is stored, recorded as stashed with its reason and the identifiers it is waiting on, and re-run when a matching blob is indexed. This is today's stash mechanism, generalised: today a stashed Ref is retried when a Capability naming its signer arrives; in Stem any grant, revocation, group change, Node blob or dependency can unstash. Stashed blobs are not advertised, not served and not counted in any scope set.
4. Index
The handler is one routine. For a Node blob it reads the Kind descriptor named by kind, checks the descriptor's rules (state matches the target kind, naming matches whether a name is present, children allows the parent relationship for any existing children, target names a field that exists in the state when access is target), and emits three things.
Facts. Indexed statements about the node with provenance: signed facts read from the state (the title, the attributes the kind's schema declares as indexable, the target of a comment) and derived facts the peer computes (heads, child count, comment count, last activity). See fact.
Links. Typed edges, built in for structure (dep from Change deps and Node heads, prev from Snapshot prev and Node prev, proof from Node and Grant proof, parent from Node parent, schema from Snapshot schema and Node kind) and declared by the descriptor's link rules for content (embed, link, mention, target, file found at the JSON Pointers the rules name). See Links.
Affected nodes. The node the blob declares; for a Grant, Revocation or Group, every node whose readers or writers may change; for a Node blob that changes placement, the node and its subtree.
For a Change or Snapshot the handler emits links and facts the same way, using the schema named by the kind of whichever node claims it. For a file blob it emits nothing but its presence.
Nothing in this step mentions documents, comments or profiles by name. The core kinds are published descriptors like any other; the daemon ships with their CIDs pinned so that it can start before it has synced them.
5. Re-evaluate
Each affected node is re-folded as Resources and nodes describes, producing its head declarations, state, version and placement. Then readers are recomputed for the affected subtree as Authority and Privacy describe. Evaluation is incremental: a Change affects one node; a Grant on a subtree affects that subtree; a Node blob that moves a folder affects the folder's subtree in both old and new positions. Nothing is recomputed that the new blob does not reach.
Re-evaluation may unstash: a Node blob whose signer is now authorized, a Change whose dependency just landed, a comment whose target now exists. Unstashed blobs re-enter at step 2.
6. Materialise blob access
Every blob reachable from an affected node's state through structural links (dep, prev, file, and the Node blob itself) is mapped to that node in the blob access relation. Grants, Revocations and Groups are mapped to their subjects. From this point a request for any of these blobs is a join against the node's readers, on every surface.
7. Notify and keep
Peers with a live Watch on a scope that the affected nodes fall in are sent the CIDs that entered the scope set and that they may read. The local policy then decides retention: blobs in a followed or pinned scope are kept; blobs fetched on demand are eligible for collection when nothing retained links to them; blobs in an ignored scope were refused at step 1.
What the pipeline produces
A processed transfer results in new or changed node states, new readers, new blob access rows, new facts and links, and a transfer record with per-blob verdicts (indexed, stashed with reason, rejected with reason). The board's phrase was that a processed batch should result in new named roots; in Stem it results in re-folded nodes.
Senders
The board listed four kinds of sender and asked whether anonymous web pushes should go. In Stem:
Sender | How it enters | What it may claim |
|---|---|---|
A peer answering our | the scope we asked for | nothing more than we asked |
A peer calling | the offered scope, with proof | only scopes our policy accepts and the peer can show authority for |
The local app or CLI via | the client's scope or a derived one | anything the signing key is authorized for |
An HTTP client via a site's publish endpoint | the client's scope | anything the signing key is authorized for; the site is just a peer calling |
An anonymous push of arbitrary blobs no longer exists. A blob with no signer (a file) enters only when an authorized state embeds it, which is the ownership claim the permissions investigation asked for on raw uploads.
The migration box
HM24 blobs are accepted for as long as the transition lasts, and converted into Stem records between validation and authorization. The team's name for this step is the magic box: old data is signature-validated as it is, then passed through a conversion into what the normal indexer expects. Conversion is deterministic, so every peer derives the same nodes from the same HM24 blobs, and the original blobs are kept so signatures remain verifiable.
HM24 blob | Stem record it becomes |
|---|---|
Ref, version shape, at path P | Node whose id is the SHA-256 CID of the earliest Ref blob for P and this genesis (that Ref is treated as the creating blob), kind document, parent the migrated id of P's parent path (or the root id), name the last segment of P, target |
Ref, tombstone shape | Node for the same id, target |
Ref, redirect shape | Node for the same id, target |
Ref with | the node gets |
Home document (path empty) and its Changes | The space root, whose id is derived from the owner's deterministic creating blob, kind space, target |
Profile | attributes (name, icon, description, alias) on the space root's state; |
Capability, role WRITER, path P | Grant: subject the migrated node of P with subtree (or the space root when P is empty), audience key(delegate), access |
Capability, role AGENT | Grant: subject the space root with subtree, audience key(delegate), access |
Comment with TSID T | Node whose id is the SHA-256 CID of the first Comment blob carrying T, in the author's space, kind comment, parent the root id, no name, |
Contact with TSID T | Node whose id is the SHA-256 CID of the first Contact blob carrying T, kind contact, parent the root id, no name, target |
Public space (any public Ref) | Grant: subject the space root, audience everyone, access |
|
|
Derived Snapshots and derived Grants carry the migrated blob's CID in a source note so their origin is auditable, and are signed by nobody: they are index records that stand in for a signature the original blob already carries. The roadmap records that sub-document capabilities are expected to be discarded by the migration; the table above keeps them as write grants on the migrated node, because the id exists and the grant is harmless, but the app stops issuing them.
Changes need no conversion. Files need no conversion.
Reindex
A reindex replays every stored blob through the pipeline in storage order. Because the pipeline stashes rather than failing on missing dependencies, and because every rule is deterministic, a reindex reproduces the same state regardless of the order blobs originally arrived. The derived tables listed in Database structure are dropped and rebuilt; the two ledgers are not, since they record history rather than derive it.
Today (HM24)
Today | Stem |
|---|---|
Seven indexers, one per blob type | One handler driven by Kind descriptors |
A batch fails as a whole on one bad blob | Per-blob verdicts; transfers record them |
Visibility propagated by a rules table over link types | Blob access filled from each node's structural links; readers from the authority graph |
Stashed Refs retried on a matching Capability | Any blob stashed on any missing authority or dependency, retried on any matching arrival |
Push of arbitrary blobs, validated after the fact | Transfers carry a claimed scope and are authorized against it first |
No record of what was served or received beyond metrics | Disclosure ledger and transfer log |
The shipping pipeline is documented in Signed blobs under "What happens to a blob".
Open questions
Open: whether derived index records for migrated HM24 blobs should be materialised as real signed blobs by the owner's device once it comes online, so that a Stem-only peer can hold a migrated space without the HM24 originals.
Open: how a daemon learns about a Kind descriptor it has never seen (a third-party kind). Stem's position: the descriptor is fetched like any dependency through the schema link, and until it arrives the Node blob is stashed.
Do you like what you are reading? Subscribe to receive updates.
Unsubscribe anytime