reticiulum-specification/todo.md

# TODO

Outstanding work for the spec repo.

## Outreach

- [ ] **File a community-documentation issue on `markqvist/Reticulum`.**
      Link this repo as a community-maintained byte-level spec. Ask
      whether the maintainer would like to bless / link from the
      official Reticulum manual. Frame it as a complement to (not a
      replacement for) the existing operator-focused docs.

## Test infrastructure

- [x] **Bootstrap `test-vectors/identities.json`** — Alice + Bob
      identities populated against RNS 1.2.0. Regenerator at
      `tools/regen_identities.py`.
- [ ] **Bootstrap remaining test-vectors files** (`announces.json`,
      `lxmf.json`, `links.json`) with the existing vectors from
      `reticulum-mobile-app/reference/test-vectors.json`. Convert to
      the proposed JSON format documented in `test-vectors/README.md`,
      adding the regenerator scripts so future contributors can
      verify vectors against newer upstream RNS releases.

- [ ] **Write the priority verifier scripts** listed in
      `tools/README.md`, in this order (highest interop value first):
      1. [x] `verify_destination_hash.py` — pure-function check, no RNS state needed
      2. [x] `verify_packet_header.py` — bit layout + HEADER_1/HEADER_2 round-trip + originator HEADER_1→HEADER_2 conversion
      3. [ ] `verify_announce_roundtrip.py` — closes the SPEC.md §4 gap (partial coverage in `verify_announce_app_data.py`)
      4. [ ] `verify_token_crypto.py` — closes SPEC.md §3 gap
      5. [ ] `verify_lxmf_opportunistic.py` — closes SPEC.md §5 gap
      6. [ ] `verify_link_handshake.py` — closes SPEC.md §6 gap
      7. [x] `verify_path_request.py` — closes SPEC.md §7.1, §7.2 gaps
      8. [ ] `verify_msgpack_quirk.py` — closes SPEC.md §9.3 gap

      Each verifier should remove its corresponding `⚠️ UNVERIFIED` /
      `🔮 SPECULATION` callout in `SPEC.md` (per `agent.md` §1).

## Open `⚠️ UNVERIFIED` items in SPEC.md

These need either a runtime test or a stronger upstream source citation
to remove their markers:

- [x] **§2.3 Originator HEADER_1 → HEADER_2 conversion.** Verified
      against RNS 1.2.0 by `tools/verify_packet_header.py`, which
      seeds `Transport.path_table` with a multi-hop entry and confirms
      the converted wire bytes via stubbed `Transport.transmit`.
      Citation updated to `RNS/Transport.py:1074-1083`.

- [x] **§4.3 The 3-element `[name, stamp_cost, [capabilities]]`
      app_data variant.** Verified against LXMF 0.9.6 by
      `tools/verify_announce_app_data.py`. Finding: in this LXMF
      version the producer emits a 2-element form only (the
      `supported_functionality` line at `LXMF/LXMRouter.py:999` is
      dead code); the parser is prepared for a 3-element form via
      `compression_support_from_app_data`. SPEC.md §4.3 updated to
      describe the actual current behavior.

- [x] **§7.1 path? always precedes LXMF DATA.** Verified against
      LXMF 0.9.6 by `tools/verify_path_request.py`. Finding: the
      preamble fires only when `not has_path()` AND method is
      OPPORTUNISTIC; the retry path can fire a second `request_path`
      after `MAX_PATHLESS_TRIES` (`LXMRouter.py:2571+`). SPEC.md §7.1
      rewritten accordingly. Also fixed a documentation bug in §1.2
      (path-request name_hash column).

- [x] **§7.4 Ratchet ring count default = 8.** False — actual upstream
      default is `Destination.RATCHET_COUNT = 512` at
      `RNS/Destination.py:85` in RNS 1.2.0, with
      `RATCHET_INTERVAL = 30*60` (line 90) and
      `RATCHET_EXPIRY = 60*60*24*30` (`RNS/Identity.py:69`).
      SPEC.md §7.4 corrected.

## Open `⚠️` items needing a runtime verifier

- [ ] **`tools/verify_rnode_split.py` to lock in §8.3.** The RNode
      air-frame split-packet protocol is now documented in SPEC.md §8.3
      against direct citations in `markqvist/RNode_Firmware/Framing.h`,
      `Config.h`, `Utilities.h`, and `RNode_Firmware.ino`, plus the
      clean-room reimplementation in `thatSFguy/reticulum-lora-repeater/src/Radio.cpp`.
      A runtime verifier would: build a 300-byte synthetic Reticulum
      packet, run it through a Python implementation of the TX-side
      header rules, and confirm the byte-level frames match what
      `RNode_Firmware.ino:716-742` would emit (header byte high nibble
      random + low-nibble FLAG_SPLIT bit, both frames sharing the same
      header, split point at 255 bytes total per LoRa frame). RX-side
      verifier should drive the state-table at SPEC.md §8.3 and confirm
      the four reassembly cases.

- [ ] **Lock in the §6.2 / §6.3 corrections with `verify_link_handshake.py`.**
      The wire-byte order of the LRPROOF body (`signature || responder_X25519_pub || signalling`,
      not `link_id || responder_X25519_pub || signature || signalling`) and
      the `link_id` derivation offsets (`N=2` for HEADER_1, `N=18` for HEADER_2,
      not 18/34) were corrected against direct upstream source citations
      (`RNS/Link.py:376`, `RNS/Packet.py:354-361`) in `SPEC.md` §6.2/§6.3
      while writing `flows/send-link-lxmf.md`. They are source-cited but
      not yet exercised by a runtime verifier. Add `tools/verify_link_handshake.py`
      that drives an upstream LINKREQUEST → LRPROOF → ACTIVE handshake and
      asserts byte-level layouts + `link_id` invariance under HEADER_1↔HEADER_2.

## Spec gaps for a functional client (priority-ordered)

The items below are missing pieces that prevent a client built only from
this spec (plus the existing flows/) from interoperating with upstream.
Tier 1 = required to talk at all to the mesh as a leaf LXMF client.
Tier 2 = required for a client that's actually useful (chat that works
in the wild). Tier 3 = required to act as a transport node / relay.

Where I've already done the source reading, I've left the file/line
citations inline so whoever picks the item up can start without
re-research.

### Tier 1 — required for a barebones leaf LXMF client to interop

- [x] **`flows/receive-announce.md` + SPEC.md §4.5 announce validation
      rules.** Done. SPEC.md §4.5 covers the MUST validation rules
      (body parse with `context_flag` branch, signed_data
      reconstruction, signature verification, dest_hash recomputation,
      public-key collision rejection, blackhole list, cache update
      order, PATH_RESPONSE handling). `flows/receive-announce.md` walks
      the chronology end-to-end. Side fixes: SPEC.md §4.1 corrected
      (`random_hash` is 5 random bytes + 5 bytes big-endian uint40
      unix_seconds, not 10 random bytes); SPEC.md §2.5 contexts table
      now lists `0x0B PATH_RESPONSE`.
- [x] **SPEC.md §10 / `flows/send-resource.md`: Reticulum Resource
      fragmentation.** Done. SPEC.md §10 covers the wire-level MUST
      rules: 13 sub-sections from "when Resource runs" through wire
      contexts (ADV / REQ / RESOURCE / HMU / PRF / ICL / RCL),
      hashmap collision-guard, sliding window, multi-segment cutover
      at MAX_EFFICIENT_SIZE = 1 MiB - 1, and the encryption-then-split
      layering. `flows/send-resource.md` walks the chronology in 10
      steps with a wire-byte ladder diagram. Side fixes during the
      drafting: SPEC.md §2.5 contexts table now lists ALL upstream
      contexts (was missing all RESOURCE_*, REQUEST/RESPONSE,
      COMMAND, CHANNEL, LINKIDENTIFY, LINKCLOSE, LRRTT entries) and
      corrects KEEPALIVE from 0xFD (which is actually LINKPROOF) to
      0xFA per `RNS/Packet.py:87`. SPEC.md §6.5 wording updated to
      use the correct LINKPROOF context name. The previously-existing
      §10 "Test vectors" and §11 "Source map" were renumbered to §11
      and §12 to put §10 in the protocol-stack flow.
- [ ] **SPEC.md §6.5 expansion: regular (non-LRPROOF) PROOF body.** The
      mandatory PROOF receipt for every CTX_NONE Link DATA packet. Body
      is `packet_hash(32) || signature(64)` (`RNS/Link.py::prove_packet`
      line 384-393), with a hardcoded explicit-mode comment hinting at
      a future implicit-mode toggle that elides the packet_hash prefix.
      Adding a `tools/verify_proof_packet.py` that runs a real link
      transfer and asserts the proof body shape is the right
      verification.
- [ ] **SPEC.md §6 sub-section: 3-byte MTU/mode signalling field.**
      Present on LINKREQUEST and LRPROOF iff
      `Reticulum.link_mtu_discovery() == True` and the next-hop
      interface advertises an HW MTU. Encode/decode helpers at
      `RNS/Link.py::signalling_bytes` line 148; consumers at
      `mtu_from_lr_packet` / `mode_from_lr_packet` /
      `mtu_from_lp_packet` / `mode_from_lp_packet`. Spec currently
      shows this slot as "[signalling(3)]" with no byte definition —
      a client that emits a wrong format gets wrong MTU on the link.
- [ ] **SPEC.md §7.2 expansion + new flow `flows/path-discovery.md`:
      path-response announce vs periodic announce.** When a node
      fulfills a `path?` request it emits an announce with
      `path_response=True`, which sets `context = PATH_RESPONSE = 0x0B`
      on the announce packet (`RNS/Packet.py:83`). Receivers
      distinguish via `packet.context == RNS.Packet.PATH_RESPONSE`
      (`RNS/Transport.py:1989-1991`); announce handlers default to
      ignoring path-responses unless they set
      `receive_path_responses = True` on themselves. Spec mentions §7.2
      "respond by re-announcing" but doesn't name the wire context.
- [ ] **SPEC.md §1.3 expansion: identity on-disk format.** §1.3 names
      the byte order (Ed25519 first, X25519 second, opposite of the
      public-key concat) but not the file structure. `RNS/Identity.py::to_file`
      is the reference. Without this, identities can't be exported /
      imported across implementations.

### Tier 2 — required for a client to be useful in the wild

- [ ] **SPEC.md: Propagation node protocol.** Offline message retrieval
      via store-and-forward propagation nodes. Without this, every
      message requires both peers online simultaneously. Authoritative
      source: `LXMF/LXMRouter.py::process_propagated`, the
      `lxmf.propagation` peering exchange (`peer()` / `sync()` between
      nodes — `LXMRouter.py:1892+, 2118+`). The `propagated` method is
      already in `LXMessage.py` but the wire protocol between
      propagation nodes is undocumented. Cross-flow:
      `flows/send-propagated-lxmf.md` (already a `⏳` entry in
      `flows/README.md`).
- [ ] **SPEC.md §6 expansion: KEEPALIVE / link teardown protocol.**
      `CTX_KEEPALIVE = 0xfd` packets — exact wire body, exact cadence
      (`Link.KEEPALIVE` constant), exact teardown packet (`Link.PROOF`
      context). Real clients drop links incorrectly without this.
- [ ] **SPEC.md §5.x (new): LXMF stamps + tickets for spam control.**
      `LXMF.Stamp` (proof-of-work field in the optional 5th element of
      the msgpack payload), `FIELD_TICKET` lookup. Modern Sideband 1.x
      treats missing-stamp messages as spam in the UI. Spec currently
      doesn't mention stamps at all. Authoritative source:
      `LXMF/LXMessage.py::validate_stamp`, `LXMF/LXMRouter.py:1741-1774`
      (the stamp-check branch in `lxmf_delivery`).
- [ ] **SPEC.md §13 (new): NomadNet page protocol.** Distinct from
      LXMF — pages fetched over a Link with `context = CTX_REQUEST (0x09)`
      / `CTX_RESPONSE (0x0a)` (already in §2.5 contexts table). Request
      body is a path string + field map; response is a body bytes blob.
      Without this, a client can do LXMF chat but can't render NomadNet
      content (nodes serving content, telemetry, micron pages).
- [ ] **SPEC.md §1.4 (new): GROUP destinations.** `RNS.Destination.GROUP`
      type uses symmetric AES-256-CBC with a pre-shared key; different
      encrypt/decrypt paths in `RNS/Destination.py:601+` (`prv` is a
      symmetric-key wrapper, not an X25519 priv). Almost no clients
      implement this but the protocol allows it.
- [ ] **SPEC.md §8.4 (new): CSMA / airtime tracking.** LoRa-only —
      carrier-sense + random backoff that prevents transmitter
      collisions on shared channel. The clean-room repeater explicitly
      flags "no CSMA" as a phase-2 simplification. A serious LoRa
      client needs `RNS.Reticulum.ANNOUNCE_CAP`-aware backoff and the
      `airtime_bins` accounting from `RNode_Firmware.ino:683-712`.
- [ ] **SPEC.md §8.5 (new): RNode KISS configuration handshake.**
      Beyond §8.3 (split-packet protocol), a client opening an RNode
      drives `CMD_DETECT` / `CMD_FREQUENCY` / `CMD_BANDWIDTH` /
      `CMD_SF` / `CMD_CR` / `CMD_TXPOWER` / `CMD_RADIO_STATE` over KISS
      to bring up the radio. All defined in `RNode_Firmware/Framing.h:24-95`.
      Spec just says "send Reticulum packets via CMD_DATA" — that's
      not enough.
- [ ] **SPEC.md §6.5 second sub-bullet: implicit vs explicit proof
      mode.** `RNS.Reticulum.should_use_implicit_proof()` mode trims
      the proof body to just the signature (no `packet_hash` prefix),
      saving 32 bytes. `RNS/Link.py:386-389` has the explicit form
      hard-coded with the implicit branch commented out, but at least
      one upstream branch toggles it — a client that hard-codes the
      explicit form will eventually meet a peer in implicit mode.

### Tier 3 — required to act as a transport node / relay

- [ ] **SPEC.md §7.7 (new): DATA forwarding rules.** Forwarding non-
      local DATA per `path_table[dest][NEXT_HOP]`, with hop increment,
      MTU-fit check, blackhole avoidance, and IFAC re-signing.
      Currently mentioned only obliquely in §2.3 / §7.6. The full
      forwarding logic is the bulk of `RNS/Transport.py::inbound`'s
      ~800-line dispatch table at lines 1499-1620. The repeater repo
      patches microReticulum to enable this — see commit
      `Add DATA and PROOF forwarding patches for transport repeating`.
- [ ] **SPEC.md §4.6 (new): ANNOUNCE rebroadcasting.** Including the
      announce-cap (`RNS.Reticulum.ANNOUNCE_CAP`, default 2% airtime),
      the announce queue, the `path_responses` cache, and the
      `random_blob` history that lets a relay drop replays. Most of
      `RNS/Transport.py:1196-1300, 1810-1969`.
- [ ] **SPEC.md §7.8 (new): path table management.** TTL-based expiry
      (`Transport.AP_PATH_TIME`, `ROAMING_PATH_TIME`, `DESTINATION_TIMEOUT`),
      eviction on stale-link, persistence-across-reboot file format.
      Hooks: `RNS/Transport.py:747-769` (stale_paths accumulator) and
      the `paths` file under `storagepath`.
- [ ] **SPEC.md §7.9 (new): tunnels and shared-instance protocol.**
      `tunnels`, `discovery_path_requests`, `RNS/Reticulum.py::is_connected_to_shared_instance`
      — how a process talks to a co-resident `rnsd`. Spec's §7.6
      covers one symptom (TCP OUT default) but not the actual
      shared-instance wire format.
- [ ] **SPEC.md §6.x (new): reverse table + link transport.** When a
      Link's path crosses a relay, the relay must forward both
      directions of every Link DATA + PROOF using
      `Transport.reverse_table` (`RNS/Transport.py:2087-2204`).
      Distinct from path-table forwarding — different lookup, different
      lifecycle.

## Spec polishing (lower priority)

- [ ] **Split `SPEC.md` into per-layer files** as the document grows
      past ~1500 lines. Suggested layout per `README.md`:
      `00-overview.md`, `01-packet-header.md`, `02-identity.md`,
      `03-announce.md`, `04-token-crypto.md`, `05-lxmf.md`,
      `06-link.md`, `07-resource.md`, `08-transport.md`,
      `09-paths-and-discovery.md`, `10-implementation-gotchas.md`.

- [ ] **Add a "last-verified-against-rns" line** to SPEC.md
      frontmatter (per `agent.md` §7) so readers know which RNS
      version the spec was tested against.