95823ad840
Closes Tier 1 #2. Without this, a client can't send any LXMF body larger than LINK_PACKET_MAX_CONTENT ≈ 360 B, can't receive a NomadNet page that doesn't fit in one MTU, and can't transfer files via rncp. SPEC.md §10 (new): full Resource fragmentation protocol with citations to RNS/Resource.py. 13 sub-sections covering preparation pipeline (metadata prefix → optional bz2 → random_hash prefix → SHA-256 over data||random_hash → link.encrypt of the WHOLE blob → part-split into SDU-sized chunks → 4-byte map_hash hashmap with collision guard within COLLISION_GUARD_SIZE = 2*WINDOW_MAX + HASHMAP_MAX_LEN), wire context inventory (RESOURCE_ADV / RESOURCE / RESOURCE_REQ / RESOURCE_HMU / RESOURCE_PRF / RESOURCE_ICL / RESOURCE_RCL), the msgpack dict for the advertisement (t/d/n/h/r/o/i/l/q/f/m), the request payload format with the hashmap_exhausted sentinel, the lazy-hashmap RESOURCE_HMU continuation that lets large hashmaps avoid breaking small-MTU links, the proof body resource_hash(32) || full_proof = SHA256(data||hash) (32) returned in a PROOF-type packet, the sliding window dynamics (WINDOW=4 → WINDOW_MAX_FAST=75 / WINDOW_MAX_VERY_SLOW=4 with rate detection), multi-segment cutover at MAX_EFFICIENT_SIZE = 1 MiB - 1 with the lazy `__prepare_next_segment` pattern, and the encryption-before-split layering that means a missing part can't be decrypted in isolation. flows/send-resource.md: 10-step chronology from RNS.Resource() construction through advertise → req/parts loop → HMU continuation → final RESOURCE_PRF → multi-segment fan-out, with a wire-byte ladder diagram and a per-step source map. Side fixes found while drafting: - SPEC.md §2.5 contexts table was wildly incomplete and had a real bug: KEEPALIVE was listed as 0xFD; upstream is 0xFA per RNS/Packet.py:87. 0xFD is actually LINKPROOF (the regular DATA-receipt context, §6.5). Replaced with the full upstream context inventory: NONE, RESOURCE_*, CACHE_REQUEST, REQUEST, RESPONSE, PATH_RESPONSE, COMMAND, COMMAND_STATUS, CHANNEL, KEEPALIVE, LINKIDENTIFY, LINKCLOSE, LINKPROOF, LRRTT, LRPROOF. - SPEC.md §6.5 reworded: "send back a PROOF packet (no context byte specifics)" → "send back a PROOF-type packet with context = LINKPROOF (0xFD)" for clarity. - The previously-numbered §10 "Test vectors" and §11 "Source map" are renumbered to §11 / §12 so the new Resource section lands in its correct protocol-stack position. agent.md §5 audit table updated accordingly. flows/README.md status table updated; receive-resource.md added as the next pending flow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
272 lines
15 KiB
Markdown
272 lines
15 KiB
Markdown
# TODO
|
|
|
|
Outstanding work for the spec repo.
|
|
|
|
## Outreach
|
|
|
|
- [ ] **File a community-documentation issue on `markqvist/Reticulum`.**
|
|
Link this repo as a community-maintained byte-level spec. Ask
|
|
whether the maintainer would like to bless / link from the
|
|
official Reticulum manual. Frame it as a complement to (not a
|
|
replacement for) the existing operator-focused docs.
|
|
|
|
## Test infrastructure
|
|
|
|
- [x] **Bootstrap `test-vectors/identities.json`** — Alice + Bob
|
|
identities populated against RNS 1.2.0. Regenerator at
|
|
`tools/regen_identities.py`.
|
|
- [ ] **Bootstrap remaining test-vectors files** (`announces.json`,
|
|
`lxmf.json`, `links.json`) with the existing vectors from
|
|
`reticulum-mobile-app/reference/test-vectors.json`. Convert to
|
|
the proposed JSON format documented in `test-vectors/README.md`,
|
|
adding the regenerator scripts so future contributors can
|
|
verify vectors against newer upstream RNS releases.
|
|
|
|
- [ ] **Write the priority verifier scripts** listed in
|
|
`tools/README.md`, in this order (highest interop value first):
|
|
1. [x] `verify_destination_hash.py` — pure-function check, no RNS state needed
|
|
2. [x] `verify_packet_header.py` — bit layout + HEADER_1/HEADER_2 round-trip + originator HEADER_1→HEADER_2 conversion
|
|
3. [ ] `verify_announce_roundtrip.py` — closes the SPEC.md §4 gap (partial coverage in `verify_announce_app_data.py`)
|
|
4. [ ] `verify_token_crypto.py` — closes SPEC.md §3 gap
|
|
5. [ ] `verify_lxmf_opportunistic.py` — closes SPEC.md §5 gap
|
|
6. [ ] `verify_link_handshake.py` — closes SPEC.md §6 gap
|
|
7. [x] `verify_path_request.py` — closes SPEC.md §7.1, §7.2 gaps
|
|
8. [ ] `verify_msgpack_quirk.py` — closes SPEC.md §9.3 gap
|
|
|
|
Each verifier should remove its corresponding `⚠️ UNVERIFIED` /
|
|
`🔮 SPECULATION` callout in `SPEC.md` (per `agent.md` §1).
|
|
|
|
## Open `⚠️ UNVERIFIED` items in SPEC.md
|
|
|
|
These need either a runtime test or a stronger upstream source citation
|
|
to remove their markers:
|
|
|
|
- [x] **§2.3 Originator HEADER_1 → HEADER_2 conversion.** Verified
|
|
against RNS 1.2.0 by `tools/verify_packet_header.py`, which
|
|
seeds `Transport.path_table` with a multi-hop entry and confirms
|
|
the converted wire bytes via stubbed `Transport.transmit`.
|
|
Citation updated to `RNS/Transport.py:1074-1083`.
|
|
|
|
- [x] **§4.3 The 3-element `[name, stamp_cost, [capabilities]]`
|
|
app_data variant.** Verified against LXMF 0.9.6 by
|
|
`tools/verify_announce_app_data.py`. Finding: in this LXMF
|
|
version the producer emits a 2-element form only (the
|
|
`supported_functionality` line at `LXMF/LXMRouter.py:999` is
|
|
dead code); the parser is prepared for a 3-element form via
|
|
`compression_support_from_app_data`. SPEC.md §4.3 updated to
|
|
describe the actual current behavior.
|
|
|
|
- [x] **§7.1 path? always precedes LXMF DATA.** Verified against
|
|
LXMF 0.9.6 by `tools/verify_path_request.py`. Finding: the
|
|
preamble fires only when `not has_path()` AND method is
|
|
OPPORTUNISTIC; the retry path can fire a second `request_path`
|
|
after `MAX_PATHLESS_TRIES` (`LXMRouter.py:2571+`). SPEC.md §7.1
|
|
rewritten accordingly. Also fixed a documentation bug in §1.2
|
|
(path-request name_hash column).
|
|
|
|
- [x] **§7.4 Ratchet ring count default = 8.** False — actual upstream
|
|
default is `Destination.RATCHET_COUNT = 512` at
|
|
`RNS/Destination.py:85` in RNS 1.2.0, with
|
|
`RATCHET_INTERVAL = 30*60` (line 90) and
|
|
`RATCHET_EXPIRY = 60*60*24*30` (`RNS/Identity.py:69`).
|
|
SPEC.md §7.4 corrected.
|
|
|
|
## Open `⚠️` items needing a runtime verifier
|
|
|
|
- [ ] **`tools/verify_rnode_split.py` to lock in §8.3.** The RNode
|
|
air-frame split-packet protocol is now documented in SPEC.md §8.3
|
|
against direct citations in `markqvist/RNode_Firmware/Framing.h`,
|
|
`Config.h`, `Utilities.h`, and `RNode_Firmware.ino`, plus the
|
|
clean-room reimplementation in `thatSFguy/reticulum-lora-repeater/src/Radio.cpp`.
|
|
A runtime verifier would: build a 300-byte synthetic Reticulum
|
|
packet, run it through a Python implementation of the TX-side
|
|
header rules, and confirm the byte-level frames match what
|
|
`RNode_Firmware.ino:716-742` would emit (header byte high nibble
|
|
random + low-nibble FLAG_SPLIT bit, both frames sharing the same
|
|
header, split point at 255 bytes total per LoRa frame). RX-side
|
|
verifier should drive the state-table at SPEC.md §8.3 and confirm
|
|
the four reassembly cases.
|
|
|
|
- [ ] **Lock in the §6.2 / §6.3 corrections with `verify_link_handshake.py`.**
|
|
The wire-byte order of the LRPROOF body (`signature || responder_X25519_pub || signalling`,
|
|
not `link_id || responder_X25519_pub || signature || signalling`) and
|
|
the `link_id` derivation offsets (`N=2` for HEADER_1, `N=18` for HEADER_2,
|
|
not 18/34) were corrected against direct upstream source citations
|
|
(`RNS/Link.py:376`, `RNS/Packet.py:354-361`) in `SPEC.md` §6.2/§6.3
|
|
while writing `flows/send-link-lxmf.md`. They are source-cited but
|
|
not yet exercised by a runtime verifier. Add `tools/verify_link_handshake.py`
|
|
that drives an upstream LINKREQUEST → LRPROOF → ACTIVE handshake and
|
|
asserts byte-level layouts + `link_id` invariance under HEADER_1↔HEADER_2.
|
|
|
|
## Spec gaps for a functional client (priority-ordered)
|
|
|
|
The items below are missing pieces that prevent a client built only from
|
|
this spec (plus the existing flows/) from interoperating with upstream.
|
|
Tier 1 = required to talk at all to the mesh as a leaf LXMF client.
|
|
Tier 2 = required for a client that's actually useful (chat that works
|
|
in the wild). Tier 3 = required to act as a transport node / relay.
|
|
|
|
Where I've already done the source reading, I've left the file/line
|
|
citations inline so whoever picks the item up can start without
|
|
re-research.
|
|
|
|
### Tier 1 — required for a barebones leaf LXMF client to interop
|
|
|
|
- [x] **`flows/receive-announce.md` + SPEC.md §4.5 announce validation
|
|
rules.** Done. SPEC.md §4.5 covers the MUST validation rules
|
|
(body parse with `context_flag` branch, signed_data
|
|
reconstruction, signature verification, dest_hash recomputation,
|
|
public-key collision rejection, blackhole list, cache update
|
|
order, PATH_RESPONSE handling). `flows/receive-announce.md` walks
|
|
the chronology end-to-end. Side fixes: SPEC.md §4.1 corrected
|
|
(`random_hash` is 5 random bytes + 5 bytes big-endian uint40
|
|
unix_seconds, not 10 random bytes); SPEC.md §2.5 contexts table
|
|
now lists `0x0B PATH_RESPONSE`.
|
|
- [x] **SPEC.md §10 / `flows/send-resource.md`: Reticulum Resource
|
|
fragmentation.** Done. SPEC.md §10 covers the wire-level MUST
|
|
rules: 13 sub-sections from "when Resource runs" through wire
|
|
contexts (ADV / REQ / RESOURCE / HMU / PRF / ICL / RCL),
|
|
hashmap collision-guard, sliding window, multi-segment cutover
|
|
at MAX_EFFICIENT_SIZE = 1 MiB - 1, and the encryption-then-split
|
|
layering. `flows/send-resource.md` walks the chronology in 10
|
|
steps with a wire-byte ladder diagram. Side fixes during the
|
|
drafting: SPEC.md §2.5 contexts table now lists ALL upstream
|
|
contexts (was missing all RESOURCE_*, REQUEST/RESPONSE,
|
|
COMMAND, CHANNEL, LINKIDENTIFY, LINKCLOSE, LRRTT entries) and
|
|
corrects KEEPALIVE from 0xFD (which is actually LINKPROOF) to
|
|
0xFA per `RNS/Packet.py:87`. SPEC.md §6.5 wording updated to
|
|
use the correct LINKPROOF context name. The previously-existing
|
|
§10 "Test vectors" and §11 "Source map" were renumbered to §11
|
|
and §12 to put §10 in the protocol-stack flow.
|
|
- [ ] **SPEC.md §6.5 expansion: regular (non-LRPROOF) PROOF body.** The
|
|
mandatory PROOF receipt for every CTX_NONE Link DATA packet. Body
|
|
is `packet_hash(32) || signature(64)` (`RNS/Link.py::prove_packet`
|
|
line 384-393), with a hardcoded explicit-mode comment hinting at
|
|
a future implicit-mode toggle that elides the packet_hash prefix.
|
|
Adding a `tools/verify_proof_packet.py` that runs a real link
|
|
transfer and asserts the proof body shape is the right
|
|
verification.
|
|
- [ ] **SPEC.md §6 sub-section: 3-byte MTU/mode signalling field.**
|
|
Present on LINKREQUEST and LRPROOF iff
|
|
`Reticulum.link_mtu_discovery() == True` and the next-hop
|
|
interface advertises an HW MTU. Encode/decode helpers at
|
|
`RNS/Link.py::signalling_bytes` line 148; consumers at
|
|
`mtu_from_lr_packet` / `mode_from_lr_packet` /
|
|
`mtu_from_lp_packet` / `mode_from_lp_packet`. Spec currently
|
|
shows this slot as "[signalling(3)]" with no byte definition —
|
|
a client that emits a wrong format gets wrong MTU on the link.
|
|
- [ ] **SPEC.md §7.2 expansion + new flow `flows/path-discovery.md`:
|
|
path-response announce vs periodic announce.** When a node
|
|
fulfills a `path?` request it emits an announce with
|
|
`path_response=True`, which sets `context = PATH_RESPONSE = 0x0B`
|
|
on the announce packet (`RNS/Packet.py:83`). Receivers
|
|
distinguish via `packet.context == RNS.Packet.PATH_RESPONSE`
|
|
(`RNS/Transport.py:1989-1991`); announce handlers default to
|
|
ignoring path-responses unless they set
|
|
`receive_path_responses = True` on themselves. Spec mentions §7.2
|
|
"respond by re-announcing" but doesn't name the wire context.
|
|
- [ ] **SPEC.md §1.3 expansion: identity on-disk format.** §1.3 names
|
|
the byte order (Ed25519 first, X25519 second, opposite of the
|
|
public-key concat) but not the file structure. `RNS/Identity.py::to_file`
|
|
is the reference. Without this, identities can't be exported /
|
|
imported across implementations.
|
|
|
|
### Tier 2 — required for a client to be useful in the wild
|
|
|
|
- [ ] **SPEC.md: Propagation node protocol.** Offline message retrieval
|
|
via store-and-forward propagation nodes. Without this, every
|
|
message requires both peers online simultaneously. Authoritative
|
|
source: `LXMF/LXMRouter.py::process_propagated`, the
|
|
`lxmf.propagation` peering exchange (`peer()` / `sync()` between
|
|
nodes — `LXMRouter.py:1892+, 2118+`). The `propagated` method is
|
|
already in `LXMessage.py` but the wire protocol between
|
|
propagation nodes is undocumented. Cross-flow:
|
|
`flows/send-propagated-lxmf.md` (already a `⏳` entry in
|
|
`flows/README.md`).
|
|
- [ ] **SPEC.md §6 expansion: KEEPALIVE / link teardown protocol.**
|
|
`CTX_KEEPALIVE = 0xfd` packets — exact wire body, exact cadence
|
|
(`Link.KEEPALIVE` constant), exact teardown packet (`Link.PROOF`
|
|
context). Real clients drop links incorrectly without this.
|
|
- [ ] **SPEC.md §5.x (new): LXMF stamps + tickets for spam control.**
|
|
`LXMF.Stamp` (proof-of-work field in the optional 5th element of
|
|
the msgpack payload), `FIELD_TICKET` lookup. Modern Sideband 1.x
|
|
treats missing-stamp messages as spam in the UI. Spec currently
|
|
doesn't mention stamps at all. Authoritative source:
|
|
`LXMF/LXMessage.py::validate_stamp`, `LXMF/LXMRouter.py:1741-1774`
|
|
(the stamp-check branch in `lxmf_delivery`).
|
|
- [ ] **SPEC.md §13 (new): NomadNet page protocol.** Distinct from
|
|
LXMF — pages fetched over a Link with `context = CTX_REQUEST (0x09)`
|
|
/ `CTX_RESPONSE (0x0a)` (already in §2.5 contexts table). Request
|
|
body is a path string + field map; response is a body bytes blob.
|
|
Without this, a client can do LXMF chat but can't render NomadNet
|
|
content (nodes serving content, telemetry, micron pages).
|
|
- [ ] **SPEC.md §1.4 (new): GROUP destinations.** `RNS.Destination.GROUP`
|
|
type uses symmetric AES-256-CBC with a pre-shared key; different
|
|
encrypt/decrypt paths in `RNS/Destination.py:601+` (`prv` is a
|
|
symmetric-key wrapper, not an X25519 priv). Almost no clients
|
|
implement this but the protocol allows it.
|
|
- [ ] **SPEC.md §8.4 (new): CSMA / airtime tracking.** LoRa-only —
|
|
carrier-sense + random backoff that prevents transmitter
|
|
collisions on shared channel. The clean-room repeater explicitly
|
|
flags "no CSMA" as a phase-2 simplification. A serious LoRa
|
|
client needs `RNS.Reticulum.ANNOUNCE_CAP`-aware backoff and the
|
|
`airtime_bins` accounting from `RNode_Firmware.ino:683-712`.
|
|
- [ ] **SPEC.md §8.5 (new): RNode KISS configuration handshake.**
|
|
Beyond §8.3 (split-packet protocol), a client opening an RNode
|
|
drives `CMD_DETECT` / `CMD_FREQUENCY` / `CMD_BANDWIDTH` /
|
|
`CMD_SF` / `CMD_CR` / `CMD_TXPOWER` / `CMD_RADIO_STATE` over KISS
|
|
to bring up the radio. All defined in `RNode_Firmware/Framing.h:24-95`.
|
|
Spec just says "send Reticulum packets via CMD_DATA" — that's
|
|
not enough.
|
|
- [ ] **SPEC.md §6.5 second sub-bullet: implicit vs explicit proof
|
|
mode.** `RNS.Reticulum.should_use_implicit_proof()` mode trims
|
|
the proof body to just the signature (no `packet_hash` prefix),
|
|
saving 32 bytes. `RNS/Link.py:386-389` has the explicit form
|
|
hard-coded with the implicit branch commented out, but at least
|
|
one upstream branch toggles it — a client that hard-codes the
|
|
explicit form will eventually meet a peer in implicit mode.
|
|
|
|
### Tier 3 — required to act as a transport node / relay
|
|
|
|
- [ ] **SPEC.md §7.7 (new): DATA forwarding rules.** Forwarding non-
|
|
local DATA per `path_table[dest][NEXT_HOP]`, with hop increment,
|
|
MTU-fit check, blackhole avoidance, and IFAC re-signing.
|
|
Currently mentioned only obliquely in §2.3 / §7.6. The full
|
|
forwarding logic is the bulk of `RNS/Transport.py::inbound`'s
|
|
~800-line dispatch table at lines 1499-1620. The repeater repo
|
|
patches microReticulum to enable this — see commit
|
|
`Add DATA and PROOF forwarding patches for transport repeating`.
|
|
- [ ] **SPEC.md §4.6 (new): ANNOUNCE rebroadcasting.** Including the
|
|
announce-cap (`RNS.Reticulum.ANNOUNCE_CAP`, default 2% airtime),
|
|
the announce queue, the `path_responses` cache, and the
|
|
`random_blob` history that lets a relay drop replays. Most of
|
|
`RNS/Transport.py:1196-1300, 1810-1969`.
|
|
- [ ] **SPEC.md §7.8 (new): path table management.** TTL-based expiry
|
|
(`Transport.AP_PATH_TIME`, `ROAMING_PATH_TIME`, `DESTINATION_TIMEOUT`),
|
|
eviction on stale-link, persistence-across-reboot file format.
|
|
Hooks: `RNS/Transport.py:747-769` (stale_paths accumulator) and
|
|
the `paths` file under `storagepath`.
|
|
- [ ] **SPEC.md §7.9 (new): tunnels and shared-instance protocol.**
|
|
`tunnels`, `discovery_path_requests`, `RNS/Reticulum.py::is_connected_to_shared_instance`
|
|
— how a process talks to a co-resident `rnsd`. Spec's §7.6
|
|
covers one symptom (TCP OUT default) but not the actual
|
|
shared-instance wire format.
|
|
- [ ] **SPEC.md §6.x (new): reverse table + link transport.** When a
|
|
Link's path crosses a relay, the relay must forward both
|
|
directions of every Link DATA + PROOF using
|
|
`Transport.reverse_table` (`RNS/Transport.py:2087-2204`).
|
|
Distinct from path-table forwarding — different lookup, different
|
|
lifecycle.
|
|
|
|
## Spec polishing (lower priority)
|
|
|
|
- [ ] **Split `SPEC.md` into per-layer files** as the document grows
|
|
past ~1500 lines. Suggested layout per `README.md`:
|
|
`00-overview.md`, `01-packet-header.md`, `02-identity.md`,
|
|
`03-announce.md`, `04-token-crypto.md`, `05-lxmf.md`,
|
|
`06-link.md`, `07-resource.md`, `08-transport.md`,
|
|
`09-paths-and-discovery.md`, `10-implementation-gotchas.md`.
|
|
|
|
- [ ] **Add a "last-verified-against-rns" line** to SPEC.md
|
|
frontmatter (per `agent.md` §7) so readers know which RNS
|
|
version the spec was tested against.
|