diff --git a/README.md b/README.md index 190c7ba..a09c97c 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,7 @@ Each finding is grounded in upstream source citations (file + line) so it can be - [`SPEC.md`](SPEC.md) — the single combined spec document, organized by protocol layer - [`flows/`](flows/) — chronological end-to-end narratives (e.g. "send a message"), cross-referencing SPEC.md sections -- [`tools/`](tools/) — self-contained Python verifier scripts that test SPEC.md claims against upstream RNS / LXMF +- [`tools/`](tools/) — self-contained Python verifier scripts that test SPEC.md claims against upstream RNS / LXMF. Pinned via [`tools/requirements.txt`](tools/requirements.txt) to the upstream versions the scripts were last re-verified against - [`test-vectors/`](test-vectors/) — known-good byte sequences each implementation should be able to round-trip (intent: grow into a compliance suite) As content grows, `SPEC.md` will be split into per-layer files (packet header, identity, announce, token-crypto, LXMF, link, resource, transport). diff --git a/SPEC.md b/SPEC.md index cb0d353..19ea9cd 100644 --- a/SPEC.md +++ b/SPEC.md @@ -2,7 +2,7 @@ A byte-level reference for implementing Reticulum-compatible clients. This document focuses on what implementations need to interop with the canonical Python implementation ([`markqvist/Reticulum`](https://github.com/markqvist/Reticulum) and [`markqvist/LXMF`](https://github.com/markqvist/LXMF)) plus the existing client ecosystem (Sideband, Nomadnet, MeshChat, the various firmware projects). -**Last verified against:** `RNS 1.2.0` / `LXMF 0.9.6` / `RNode_Firmware` (master at the spec's last revision date). Each section's source citations were re-checked against these versions; runtime verifiers in [`tools/`](tools/) lock the wire-format claims in against actually-running upstream code. When you upgrade past these, re-run every `tools/verify_*.py` and look for `FAIL`s. +**Last verified against:** `RNS 1.2.4` / `LXMF 0.9.7` / `RNode_Firmware` (master at the spec's last revision date). Each section's source citations were re-checked against these versions; runtime verifiers in [`tools/`](tools/) lock the wire-format claims in against actually-running upstream code. When you upgrade past these, re-run every `tools/verify_*.py` and look for `FAIL`s. Source citations refer to the standard `pip install rns lxmf` install layout (`RNS/`, `LXMF/`). @@ -208,7 +208,7 @@ Common pre-computed `name_hash` values: prv_bytes_blob = X25519_priv(32) || Ed25519_priv(32) // 64 bytes total ``` -`Identity.get_private_key()` at `RNS/Identity.py:694-698` returns this exact concatenation: +`Identity.get_private_key()` at `RNS/Identity.py:723-728` returns this exact concatenation: ```python def get_private_key(self): @@ -243,7 +243,7 @@ The reason: `X25519PrivateKey.from_private_bytes` and `Ed25519PrivateKey.from_pr #### Cross-implementation portability -The format is portable across implementations because there's nothing in it but the raw bytes. A 64-byte file written by Python RNS is byte-identical to one written by any clean-room implementation that follows this section, and both produce the same `identity_hash` and `lxmf.delivery` `destination_hash` when fed back through §1.1 and §1.2 — test vectors at [`test-vectors/identities.json`](test-vectors/identities.json) demonstrate the round-trip against RNS 1.2.0. +The format is portable across implementations because there's nothing in it but the raw bytes. A 64-byte file written by Python RNS is byte-identical to one written by any clean-room implementation that follows this section, and both produce the same `identity_hash` and `lxmf.delivery` `destination_hash` when fed back through §1.1 and §1.2 — test vectors at [`test-vectors/identities.json`](test-vectors/identities.json) demonstrate the round-trip against RNS 1.2.4. > ⚠️ **Spec correction:** Earlier revisions of this section described the on-disk order as Ed25519 first, X25519 second ("opposite of the public_key concatenation"). That was wrong — verified by re-running `Identity.to_file` and reading back the bytes against the test vector at `test-vectors/identities.json`, the actual order is X25519 first, Ed25519 second, identical to the public_key order. Implementations following the prior spec wording would have corrupted identity files when interoperating with upstream Python RNS. @@ -324,7 +324,7 @@ bit 3-2 : destination_type (0 = SINGLE, 1 = GROUP, 2 = PLAIN, 3 = LINK) bit 1-0 : packet_type (0 = DATA, 1 = ANNOUNCE, 2 = LINKREQUEST, 3 = PROOF) ``` -Each subfield is **1 bit** (or 2 for `destination_type` / `packet_type`). Upstream's parser extracts them with these masks (`RNS/Packet.py:246-250` in RNS 1.2.0): +Each subfield is **1 bit** (or 2 for `destination_type` / `packet_type`). Upstream's parser extracts them with these masks (`RNS/Packet.py:246-250` in RNS 1.2.4): ```python self.header_type = (self.flags & 0b01000000) >> 6 # bit 6 only @@ -334,7 +334,7 @@ self.destination_type = (self.flags & 0b00001100) >> 2 self.packet_type = (self.flags & 0b00000011) ``` -Bit 7 (`ifac_flag`) is set by `Transport.transmit` immediately before transmission when the egress interface has an IFAC identity attached (`RNS/Transport.py:993-1024`). The setter is unambiguous: +Bit 7 (`ifac_flag`) is set by `Transport.transmit` immediately before transmission when the egress interface has an IFAC identity attached (`RNS/Transport.py:994-1024`). The setter is unambiguous: ```python # Set IFAC flag @@ -343,7 +343,7 @@ new_header = bytes([raw[0] | 0x80, raw[1]]) # 0x80 = bit 7 new_raw = new_header + ifac + raw[2:] # IFAC field is inserted between header and addresses ``` -When `ifac_flag = 1`, an `ifac_size`-byte IFAC field appears immediately after byte 2 of the header — i.e. between the `hops` byte and the start of `ADDRESSES`. `ifac_size` is interface-configured and ranges from `IFAC_MIN_SIZE = 1` byte (`RNS/Reticulum.py:151-154`) up to 64 bytes (full Ed25519 signature). The receiving side strips the IFAC after verification and rebuilds the un-IFACed packet for upstream processing (`RNS/Transport.py:1342-1369`). +When `ifac_flag = 1`, an `ifac_size`-byte IFAC field appears immediately after byte 2 of the header — i.e. between the `hops` byte and the start of `ADDRESSES`. `ifac_size` is interface-configured and ranges from `IFAC_MIN_SIZE = 1` byte (`RNS/Reticulum.py:148-152`) up to 64 bytes (full Ed25519 signature). The receiving side strips the IFAC after verification and rebuilds the un-IFACed packet for upstream processing (`RNS/Transport.py:1343-1369`). > ⚠️ **Spec correction.** Earlier revisions of this section (through commit [`8c4d550`](https://github.com/thatSFguy/reticulum-specifications/commit/8c4d550)) treated `header_type` as a 2-bit field occupying bits 7-6, with bit 7 reserved. That was wrong: bit 7 has always been the IFAC flag (`Transport.transmit` line 1003), and `header_type` is 1 bit at position 6. The official manual §4.6.3 documents the correct layout. Implementations that followed the prior wording will mis-parse IFAC-protected packets as `header_type = 2 or 3` and reject them. Surfaced by the reporter on [issue #4](https://github.com/thatSFguy/reticulum-specifications/issues/4) item #1. @@ -358,7 +358,7 @@ HEADER_2: flags(1) hops(1) transport_id(16) dest_hash(16) context(1) data(...) ### 2.3 Originator HEADER_1 → HEADER_2 conversion -This is non-obvious and matters: when an **originator** (not a relay) sends a packet to a destination known to be more than 1 hop away, the originator MUST also do the HEADER_2 conversion. From `RNS/Transport.py::outbound` (lines 1074-1083 in RNS 1.2.0; verified by `tools/verify_packet_header.py`): +This is non-obvious and matters: when an **originator** (not a relay) sends a packet to a destination known to be more than 1 hop away, the originator MUST also do the HEADER_2 conversion. From `RNS/Transport.py::outbound` (lines 1077-1108 in RNS 1.2.4; verified by `tools/verify_packet_header.py`): ```python if path_entry[IDX_PT_HOPS] > 1: @@ -370,7 +370,7 @@ if path_entry[IDX_PT_HOPS] > 1: new_raw += packet.raw[2:] # original dest_hash + context + payload ``` -For destinations 0 or 1 hops away, the originator may stay HEADER_1 — the receiving rnsd auto-fills the transport_id when the destination matches a local client (`for_local_client` branch at `RNS/Transport.py:1451` in RNS 1.2.0). Implementations that always emit HEADER_1 will silently fail to deliver to multi-hop destinations even with a known path. +For destinations 0 or 1 hops away, the originator may stay HEADER_1 — the receiving rnsd auto-fills the transport_id when the destination matches a local client (`for_local_client` branch at `RNS/Transport.py:1454` in RNS 1.2.4). Implementations that always emit HEADER_1 will silently fail to deliver to multi-hop destinations even with a known path. ### 2.4 Hop count @@ -380,7 +380,7 @@ Byte 1 is `hops`, an 8-bit counter that each transit relay increments by 1. `0` Single byte after the destination hash (offset 18 for HEADER_1, offset 34 for HEADER_2). Common values: -Full context inventory from `RNS/Packet.py:72-92` (RNS 1.2.0): +Full context inventory from `RNS/Packet.py:74-92` (RNS 1.2.4): | Hex | Name | Used for | |---|---|---| @@ -402,7 +402,7 @@ Full context inventory from `RNS/Packet.py:72-92` (RNS 1.2.0): | `0xFA` | KEEPALIVE | Link keepalive (sent periodically while a Link is idle) | | `0xFB` | LINKIDENTIFY | Backchannel-identify proof on an established Link (§5 backchannel) | | `0xFC` | LINKCLOSE | Link teardown notification | -| `0xFD` | LINKPROOF | Defined but **not actually emitted** by upstream RNS 1.2.0 in this revision. Both `Identity.prove` and `Link.prove_packet` build their proof packets with `context = NONE (0x00)` — the proof-ness is conveyed by `packet_type = PROOF (3)`, not by this context byte. Reserved for a future revision; see §6.5 | +| `0xFD` | LINKPROOF | Defined but **not actually emitted** by upstream RNS 1.2.4 in this revision. Both `Identity.prove` and `Link.prove_packet` build their proof packets with `context = NONE (0x00)` — the proof-ness is conveyed by `packet_type = PROOF (3)`, not by this context byte. Reserved for a future revision; see §6.5 | | `0xFE` | LRRTT | Link RTT measurement reply | | `0xFF` | LRPROOF | Link request proof (§6.2) | @@ -466,7 +466,7 @@ The 64-byte `public_key` is the X25519 || Ed25519 concat described in section 1. random_hash = RNS.Identity.get_random_hash()[0:5] + int(time.time()).to_bytes(5, "big") ``` -Transit relays read the timestamp portion via `Transport.timebase_from_random_blob(random_blob) = int.from_bytes(random_blob[5:10], "big")` (`RNS/Transport.py:3100-3101`) to make ordering decisions when an inbound announce carries a higher hop count than the cached path: only newer-emitted announces can refresh the path table (see §4.5). 5 bytes of seconds covers ~34,000 years, so wraparound is not a near-term concern. Implementations MUST emit this exact format, including a clock value that's monotonically non-decreasing across announces from the same destination — clockless sender devices (per §9.6) may end up locked out of long-range path table updates. +Transit relays read the timestamp portion via `Transport.timebase_from_random_blob(random_blob) = int.from_bytes(random_blob[5:10], "big")` (`RNS/Transport.py:3106-3107`) to make ordering decisions when an inbound announce carries a higher hop count than the cached path: only newer-emitted announces can refresh the path table (see §4.5). 5 bytes of seconds covers ~34,000 years, so wraparound is not a near-term concern. Implementations MUST emit this exact format, including a clock value that's monotonically non-decreasing across announces from the same destination — clockless sender devices (per §9.6) may end up locked out of long-range path table updates. > ⚠️ **UNVERIFIED — Known deviation:** `attermann/microReticulum/src/Destination.cpp:270-272` (and therefore every project that uses microReticulum unmodified, including [`thatSFguy/reticulum-lora-repeater`](https://github.com/thatSFguy/reticulum-lora-repeater) and the Faketec sibling project) currently emits 10 fully-random bytes for `random_hash` — the timestamp half is a TODO that never landed: > @@ -476,7 +476,7 @@ Transit relays read the timestamp portion via `Transport.timebase_from_random_bl > Bytes random_hash = Cryptography::random(Type::Identity::RANDOM_HASH_LENGTH/8); > ``` > -> Python RNS receivers interpret `random_hash[5:10]` as a big-endian uint40 unix_seconds. A uniformly-random uint40 has median value ~5.5×10¹¹ ≈ year 19403 AD, so a microReticulum announce will (with overwhelming probability) appear "far-future" to a Python receiver. Effect: once one such announce populates `path_table[dest][IDX_PT_RANDBLOBS]`, the equal-or-greater-hop branch at `RNS/Transport.py:1721-1745` will reject any real-timestamped announce as "stale" until the path TTL expires. First-contact path-table population is unaffected; the bug only surfaces on path replacement under §4.5 step 6.3. The microReticulum receive side does NOT consult the timestamp half so microReticulum-to-microReticulum traffic is unaffected. The repeater repo's `pre_build.py` patches several microReticulum protocol bugs but not this one (as of [`thatSFguy/reticulum-lora-repeater@95823ad`-vintage upstream](https://github.com/thatSFguy/reticulum-lora-repeater)). Verifying by capture-and-decode against an actual mixed-vendor mesh is the work that would let this callout be removed. +> Python RNS receivers interpret `random_hash[5:10]` as a big-endian uint40 unix_seconds. A uniformly-random uint40 has median value ~5.5×10¹¹ ≈ year 19403 AD, so a microReticulum announce will (with overwhelming probability) appear "far-future" to a Python receiver. Effect: once one such announce populates `path_table[dest][IDX_PT_RANDBLOBS]`, the equal-or-greater-hop branch at `RNS/Transport.py:1723-1755` will reject any real-timestamped announce as "stale" until the path TTL expires. First-contact path-table population is unaffected; the bug only surfaces on path replacement under §4.5 step 6.3. The microReticulum receive side does NOT consult the timestamp half so microReticulum-to-microReticulum traffic is unaffected. The repeater repo's `pre_build.py` patches several microReticulum protocol bugs but not this one (as of [`thatSFguy/reticulum-lora-repeater@95823ad`-vintage upstream](https://github.com/thatSFguy/reticulum-lora-repeater)). Verifying by capture-and-decode against an actual mixed-vendor mesh is the work that would let this callout be removed. The optional 32-byte `ratchet_pub` (an X25519 public key) is present iff the packet header's `context_flag` bit is 1. Indexing through this layout accordingly is mandatory; see `RNS/Identity.py::validate_announce` for the canonical parser. @@ -491,10 +491,10 @@ Note that `dest_hash` is INCLUDED in the signed data even though it's not in the ### 4.3 `app_data` format for LXMF delivery destinations -Upstream `LXMF/LXMRouter.py::get_announce_app_data` produces a 2-element msgpack array (verified against LXMF 0.9.6 by `tools/verify_announce_app_data.py`): +Upstream `LXMF/LXMRouter.py::get_announce_app_data` produces a 2-element msgpack array (verified against LXMF 0.9.7 by `tools/verify_announce_app_data.py`): ```python -# LXMF/LXMRouter.py:986-1002 in LXMF 0.9.6 +# LXMF/LXMRouter.py:985-1002 in LXMF 0.9.7 peer_data = [display_name, stamp_cost] # stamp_cost = None unless 1 ≤ N ≤ 254 return msgpack.packb(peer_data) ``` @@ -510,7 +510,7 @@ c0 # nil (stamp_cost) Encoding the display name as msgpack `bin` (`0xc4 NN`) is required for upstream interop — see section 9.3 below. The stamp_cost field can be `int 0` (`0x00`) or `nil` (`0xc0`); upstream's `stamp_cost_from_app_data` doesn't strict-type-check. -**A third optional `[capability_flags]` element** (e.g. `[SF_COMPRESSION]`, the only flag currently defined at `LXMF/LXMF.py:108`) is **read by the parser** (`compression_support_from_app_data` at `LXMF/LXMF.py:154-167`) but is **not emitted by the LXMF 0.9.6 producer** — `LXMRouter.py:999` computes `supported_functionality = [SF_COMPRESSION]` but never appends it to `peer_data`. Implementations should accept the 3-element form on inbound (a future LXMF version may re-enable it; older deployments may emit it) but should not rely on receiving it. +**A third optional `[capability_flags]` element** (e.g. `[SF_COMPRESSION]`, the only flag currently defined at `LXMF/LXMF.py:108`) is **read by the parser** (`compression_support_from_app_data` at `LXMF/LXMF.py:154-167`) but is **not emitted by the LXMF 0.9.7 producer** — `LXMRouter.py:998` computes `supported_functionality = [SF_COMPRESSION]` but never appends it to `peer_data`. Implementations should accept the 3-element form on inbound (a future LXMF version may re-enable it; older deployments may emit it) but should not rely on receiving it. The parser also tolerates a 1-element msgpack array (just the name) and a raw UTF-8 string ("original announce format" branch at `LXMF/LXMF.py:138-139`) — see `LXMF/LXMF.py::display_name_from_app_data` for all four accepted shapes. @@ -528,7 +528,7 @@ Treating every announce as a contact (the naive default) populates the UI with h ### 4.5 Announce validation rules (receive side) -These are the MUST rules a receiver applies to every inbound announce before considering the announced destination "known". The canonical implementation is `RNS/Identity.py::validate_announce` (line 496-598 in RNS 1.2.0); the dispatch site that calls it is `RNS/Transport.py::inbound` line 1623-1650. +These are the MUST rules a receiver applies to every inbound announce before considering the announced destination "known". The canonical implementation is `RNS/Identity.py::validate_announce` (line 509-612 in RNS 1.2.4); the dispatch site that calls it is `RNS/Transport.py::inbound` line 1631-1655. #### 1. Body parse — branch on `context_flag` @@ -574,33 +574,33 @@ identity_hash = SHA256(public_key)[:16] expected_hash = SHA256(name_hash || identity_hash)[:16] ``` -Reject the announce iff `expected_hash != packet.destination_hash` (the value from the outer header). This catches both random hash collisions and active spoofing attempts that pair a valid signature with an unrelated dest_hash. (`RNS/Identity.py:548-551`). +Reject the announce iff `expected_hash != packet.destination_hash` (the value from the outer header). This catches both random hash collisions and active spoofing attempts that pair a valid signature with an unrelated dest_hash. (`RNS/Identity.py:562-565`). #### 4. Public-key collision rejection -If the receiver already has a different public_key cached for this `destination_hash` (from a prior announce), the new announce MUST be rejected with a critical-severity log even if the signature is otherwise valid. Per the upstream comment: "In reality, this should never occur, but in the odd case that someone manages a hash collision, we reject the announce" (`RNS/Identity.py:554-560`). +If the receiver already has a different public_key cached for this `destination_hash` (from a prior announce), the new announce MUST be rejected with a critical-severity log even if the signature is otherwise valid. Per the upstream comment: "In reality, this should never occur, but in the odd case that someone manages a hash collision, we reject the announce" (`RNS/Identity.py:569-575`). This rule means: **first-announcer-wins for any given destination_hash** within a receiver's lifetime. A peer who loses their identity material and regenerates with the same display name + app_name will produce a different identity_hash → different destination_hash → no collision. A peer who tries to *replace* their announced public key under the same destination_hash, however, gets rejected — the real defense against this class of attack. #### 5. Blackhole list check -Before everything else, check `RNS.Transport.blackholed_identities`. An identity_hash on the blackhole list is dropped silently regardless of signature validity (`RNS/Identity.py:538-541`). This is operator-controlled state, not a wire feature. +Before everything else, check `RNS.Transport.blackholed_identities`. An identity_hash on the blackhole list is dropped silently regardless of signature validity (`RNS/Identity.py:551-554`). This is operator-controlled state, not a wire feature. #### 6. Caching the announce contents On a fully validated announce, the receiver MUST update its caches in this order: -1. **`known_destinations[destination_hash]`** ← `[recv_time, packet_hash, public_key, app_data, last_used]` — populates the table that `RNS.Identity.recall(dest_hash)` reads when constructing outbound destinations (`RNS/Identity.py::remember`, line 100-112). Without this, every subsequent outbound message to this peer fails because no public key is available for Token encryption. +1. **`known_destinations[destination_hash]`** ← `[recv_time, packet_hash, public_key, app_data, last_used]` — populates the table that `RNS.Identity.recall(dest_hash)` reads when constructing outbound destinations (`RNS/Identity.py::remember`, line 101-113). Without this, every subsequent outbound message to this peer fails because no public key is available for Token encryption. 2. **`known_ratchets[destination_hash]`** ← `ratchet_pub` (only if `context_flag == 1` and `ratchet_pub != b""`) — `Identity._remember_ratchet`, line 395-428. The ratchet is also persisted to disk under `{storagepath}/ratchets/{hexhash}` for use across restarts. 3. **`path_table`** entry update or insertion (see §4.6 — TBD when the relay rebroadcast spec lands), gated by: - - `random_blob` (= `random_hash`) not in the cached `random_blobs` history for this destination — cheap replay defence (`RNS/Transport.py:1707, 1732, 1745`). - - Hop count comparison against any existing entry: equal-or-fewer hops always win; more hops win only if the cached path has expired or the new announce's emission timestamp (from `random_hash[5:10]`) is more recent than every cached blob's timestamp (`RNS/Transport.py:1700-1745`). + - `random_blob` (= `random_hash`) not in the cached `random_blobs` history for this destination — cheap replay defence (`RNS/Transport.py:1710, 1735, 1748`). + - Hop count comparison against any existing entry: equal-or-fewer hops always win; more hops win only if the cached path has expired or the new announce's emission timestamp (from `random_hash[5:10]`) is more recent than every cached blob's timestamp (`RNS/Transport.py:1700-1755`). #### 7. `PATH_RESPONSE` distinction An announce whose outer packet `context == PATH_RESPONSE (0x0B)` is the responder's reply to a recent `path?` request, not a periodic re-announce. Validation is identical (rules 1-6 above), but listener dispatch differs: -- The default behavior of `Transport.announce_handlers` registered via `RNS.Transport.register_announce_handler` is to **skip** path-response announces unless the handler sets `receive_path_responses = True` on itself (`RNS/Transport.py:1989-1991`). +- The default behavior of `Transport.announce_handlers` registered via `RNS.Transport.register_announce_handler` is to **skip** path-response announces unless the handler sets `receive_path_responses = True` on itself (`RNS/Transport.py:1991-1995`). - The path table population path is the same either way — both regular and path-response announces refresh the path entry — so a leaf client that ignores PATH_RESPONSE entirely at the application layer still benefits from the path-table side effect. #### 8. Implementation-private behavior (SHOULD) @@ -608,18 +608,18 @@ An announce whose outer packet `context == PATH_RESPONSE (0x0B)` is the responde These are not wire-spec MUST rules but most working clients implement them; without them the implementation will misbehave in busy meshes: - **Per-interface ingress rate limiting.** When the inbound announce rate on an interface exceeds `IC_BURST_FREQ_NEW = 6 Hz` (interfaces less than 2 hours old) or `IC_BURST_FREQ = 35 Hz` (older), and the announced destination is **not** in `path_table` and **not** in `path_requests`, the announce is held in the interface's `held_announces` dict for later release rather than processed immediately. Released later in lowest-hop-count-first order. (`RNS/Interfaces/Interface.py:60-200`.) Without this, a flood of unknown-destination announces can drown out everything else. -- **`random_blob` history cap.** The cached `random_blobs` list per destination is bounded by `Transport.MAX_RANDOM_BLOBS` to keep the path table from growing without bound under a long-lived destination's announce stream (`RNS/Transport.py:1820`). +- **`random_blob` history cap.** The cached `random_blobs` list per destination is bounded by `Transport.MAX_RANDOM_BLOBS` (= 64 in RNS 1.2.4 at `RNS/Transport.py:97`) to keep the path table from growing without bound under a long-lived destination's announce stream (`RNS/Transport.py:1823`). - **Self-announce filter.** §9.5 — drop announces where `destination_hash` matches one of the receiver's own destinations to avoid populating its own contact list with itself. #### 9. Source map for §4.5 | File | What it pins down | |---|---| -| `RNS/Identity.py:496-598` | `validate_announce` — body parse, signed_data, sig verify, dest_hash recompute, collision check | -| `RNS/Identity.py:100-112` | `Identity.remember` — `known_destinations` update | -| `RNS/Identity.py:395-428` | `_remember_ratchet` — ratchet persistence | -| `RNS/Transport.py:1623-2024` | inbound dispatch for `packet_type == ANNOUNCE`: quick sig check, ingress limiting, path table population, handler dispatch | -| `RNS/Transport.py:3100-3117` | `timebase_from_random_blob`, `announce_emitted` | +| `RNS/Identity.py:509-612` | `validate_announce` — body parse, signed_data, sig verify, dest_hash recompute, collision check | +| `RNS/Identity.py:101-113` | `Identity.remember` — `known_destinations` update | +| `RNS/Identity.py:408-441` | `_remember_ratchet` — ratchet persistence | +| `RNS/Transport.py:1631-2030` | inbound dispatch for `packet_type == ANNOUNCE`: quick sig check, ingress limiting, path table population, handler dispatch | +| `RNS/Transport.py:3106-3122` | `timebase_from_random_blob`, `announce_emitted` | | `RNS/Interfaces/Interface.py:60-200` | ingress-limit constants, `should_ingress_limit`, `hold_announce`, `process_held_announces` | | `RNS/Packet.py:83` | `PATH_RESPONSE = 0x0B` context constant | @@ -938,10 +938,10 @@ Receivers parse this via `pn_announce_data_is_valid` (`LXMF/LXMF.py:191-206`), w | File | What | |---|---| | `LXMF/LXMRouter.py:173` | propagation_destination construction | -| `LXMF/LXMRouter.py:307-319` | propagation announce app_data shape | -| `LXMF/LXMRouter.py:651-655` | `/offer` and `/get` handler registration | -| `LXMF/LXMRouter.py:1427-1500` | `message_get_request` handler (client `/get`) | -| `LXMF/LXMRouter.py:2142-2192` | `offer_request` handler (peer `/offer`) | +| `LXMF/LXMRouter.py:306-322` | propagation announce app_data shape | +| `LXMF/LXMRouter.py:650-651` | `/offer` and `/get` handler registration | +| `LXMF/LXMRouter.py:1426-1500` | `message_get_request` handler (client `/get`) | +| `LXMF/LXMRouter.py:2145-2200` | `offer_request` handler (peer `/offer`) | | `LXMF/LXMPeer.py:14-50` | path constants and error-response constants | | `LXMF/LXMPeer.py:370-486` | initiator-side `/offer` flow | | `LXMF/LXStamper.py::validate_peering_key` | peering-key PoW validation | @@ -971,13 +971,13 @@ Both initiator-side keys are **fresh ephemeral keys** (not the initiator's long- A `packet_type = PROOF (3)` with `context = 0xff`, addressed to the link itself — i.e. `dest_hash` in the packet header is the 16-byte `link_id` (`RNS/Packet.py:182-184`: when context is `LRPROOF`, `header += destination.link_id` and the body is appended unencrypted). -Body (`proof_data` at `RNS/Link.py:376`): +Body (`proof_data` at `RNS/Link.py:371`): ``` signature(64) || responder_X25519_pub(32) || [signalling(3)] ``` -Only the responder's X25519 is fresh-ephemeral; the responder signs with its **long-term** Ed25519 private key (asymmetric with the initiator). The responder's long-term Ed25519 public key is **not** sent on the wire — both sides already know it from the responder's prior announce, and it is included implicitly in the signature input. Signature input (`RNS/Link.py:373` for the signer, `:417` for the validator): +Only the responder's X25519 is fresh-ephemeral; the responder signs with its **long-term** Ed25519 private key (asymmetric with the initiator). The responder's long-term Ed25519 public key is **not** sent on the wire — both sides already know it from the responder's prior announce, and it is included implicitly in the signature input. Signature input (`RNS/Link.py:373-374` for the signer, `:417` for the validator): ``` signed_data = link_id || responder_X25519_pub || responder_long_term_Ed25519_pub || [signalling] @@ -1003,7 +1003,7 @@ hashable_part = byte(flags & 0x0F) || raw[N:] The "hashable part" deliberately strips `header_type`, `context_flag`, `transport_type` (top 4 bits of flags — modifiable by transit relays), the `hops` byte (modified by every relay), and (for HEADER_2) the `transport_id` (added by the originator and re-written by each relay). What remains in both cases is the low nibble of flags + dest_hash + context + body, so the resulting `link_id` is the same whether the LINKREQUEST is hashed at the initiator (HEADER_1) or at the responder after one or more transport relays (HEADER_2). Both sides agree on the 16-byte ID. -For LINKREQUEST packets specifically, the trailing signalling bytes (if present, indicated by `len(packet.data) > Link.ECPUBSIZE` in `link_id_from_lr_packet` at `RNS/Link.py:340-347`) are stripped from the END of `hashable_part` before hashing, so the link_id is invariant under MTU-discovery signalling. +For LINKREQUEST packets specifically, the trailing signalling bytes (if present, indicated by `len(packet.data) > Link.ECPUBSIZE` in `link_id_from_lr_packet` at `RNS/Link.py:341-348`) are stripped from the END of `hashable_part` before hashing, so the link_id is invariant under MTU-discovery signalling. ### 6.4 Session key derivation @@ -1050,7 +1050,7 @@ The two forms are distinguished **purely by length** at the receiver. `PacketRec Sender side, two distinct policies: -**Opportunistic DATA addressed to a SINGLE destination** — `RNS.Identity.prove(packet, destination)` at `RNS/Identity.py:912-923`: +**Opportunistic DATA addressed to a SINGLE destination** — `RNS.Identity.prove(packet, destination)` at `RNS/Identity.py:943-954`: ```python def prove(self, packet, destination=None): @@ -1064,7 +1064,7 @@ def prove(self, packet, destination=None): proof.send() ``` -The default upstream value is `Reticulum.__use_implicit_proof = True` (`RNS/Reticulum.py:259`), so **upstream emits the 64-byte implicit form by default**. The 96-byte explicit form is only emitted when the operator's `[reticulum]` config sets `use_implicit_proof = No`. A clean-room implementation that hardcodes either single form will fail to interop with peers running the other one — receiver-side validators handle both, but a hardcoded sender writing the wrong length to the wire is not negotiable. +The default upstream value is `Reticulum.__use_implicit_proof = True` (`RNS/Reticulum.py:256`), so **upstream emits the 64-byte implicit form by default**. The 96-byte explicit form is only emitted when the operator's `[reticulum]` config sets `use_implicit_proof = No`. A clean-room implementation that hardcodes either single form will fail to interop with peers running the other one — receiver-side validators handle both, but a hardcoded sender writing the wrong length to the wire is not negotiable. **DATA on an active Link** — `RNS.Link.prove_packet(packet)` at `RNS/Link.py:383-394`: @@ -1076,7 +1076,7 @@ def prove_packet(self, packet): proof.send() ``` -with the upstream comment `# TODO: Hardcoded as explicit proof for now`. Link DATA proofs are **always** the 96-byte explicit form in RNS 1.2.0 regardless of the `use_implicit_proof` setting, and the matching `validate_link_proof` at `RNS/Packet.py:449-494` has the implicit-form branch commented out with the same note. Today, Link DATA proofs are explicit-only on both ends; an implementation may match this behavior with a single hardcoded length on the link path, but should be ready to revisit if upstream re-enables the implicit branch (no fixed timeline). +with the upstream comment `# TODO: Hardcoded as explicit proof for now`. Link DATA proofs are **always** the 96-byte explicit form in RNS 1.2.4 regardless of the `use_implicit_proof` setting, and the matching `validate_link_proof` at `RNS/Packet.py:449-494` has the implicit-form branch commented out with the same note. Today, Link DATA proofs are explicit-only on both ends; an implementation may match this behavior with a single hardcoded length on the link path, but should be ready to revisit if upstream re-enables the implicit branch (no fixed timeline). #### 6.5.3 Where the proof packet is addressed @@ -1098,7 +1098,7 @@ implicit form (64 bytes total body): [ 64B Ed25519_signature(SHA256(get_hashable_part(original_packet))) ] ``` -Note `context = 0x00 (NONE)` in both cases — the proof-ness is conveyed by `packet_type = PROOF (3)` in the flag byte, not by a context. This is in contrast to LRPROOF (which uses `context = 0xFF`) and RESOURCE_PRF (which uses `context = 0x05`). The `LINKPROOF (0xFD)` context constant defined at `RNS/Packet.py:90` is reserved but not actually used by either prove path in RNS 1.2.0. +Note `context = 0x00 (NONE)` in both cases — the proof-ness is conveyed by `packet_type = PROOF (3)` in the flag byte, not by a context. This is in contrast to LRPROOF (which uses `context = 0xFF`) and RESOURCE_PRF (which uses `context = 0x05`). The `LINKPROOF (0xFD)` context constant defined at `RNS/Packet.py:90` is reserved but not actually used by either prove path in RNS 1.2.4. #### 6.5.5 Receiver tolerance @@ -1124,7 +1124,7 @@ byte 1 : m m m m m m m m — mtu[15..8] byte 2 : m m m m m m m m — mtu[7..0] ``` -Encoded by `RNS/Link.py:147-151`: +Encoded by `RNS/Link.py:148-152`: ```python @staticmethod @@ -1148,9 +1148,9 @@ The mtu decode trick: the full 24-bit value of all three bytes is masked with th #### 6.6.2 Mode field -3-bit value (0..7) at the top of byte 0. Defined values, with `RNS/Link.py:125-142`: +3-bit value (0..7) at the top of byte 0. Defined values, with `RNS/Link.py:126-145`: -| Mode | Name | Status in RNS 1.2.0 | Derived key length | +| Mode | Name | Status in RNS 1.2.4 | Derived key length | |---|---|---|---| | `0x00` | `MODE_AES128_CBC` | Defined, NOT enabled (sender-side will raise `TypeError`) | 32 bytes | | `0x01` | `MODE_AES256_CBC` | Default; the only enabled mode (`ENABLED_MODES = [0x01]`) | 64 bytes | @@ -1158,7 +1158,7 @@ The mtu decode trick: the full 24-bit value of all three bytes is masked with th | `0x03` | `MODE_OTP_RESERVED` | Reserved, not enabled | — | | `0x04`–`0x07` | `MODE_PQ_RESERVED_*` | Reserved for the post-quantum migration; not enabled | — | -The `derived_key_length` at `RNS/Link.py:358-360` is what the HKDF in §6.4 produces, split as `signing_key(32) || encrypt_key(32)` for the AES-256 path or `signing_key(16) || encrypt_key(16)` for the AES-128 path. +The `derived_key_length` at `RNS/Link.py:359-361` is what the HKDF in §6.4 produces, split as `signing_key(32) || encrypt_key(32)` for the AES-256 path or `signing_key(16) || encrypt_key(16)` for the AES-128 path. A receiver MUST tolerate seeing any 3-bit value in the mode field on inbound traffic — `mode_from_lr_packet` returns the raw integer without validating it against `ENABLED_MODES`. The mode is enforced at handshake time (`Link.handshake` at line 353-368): unknown / disabled modes raise `TypeError` and the link transitions to `CLOSED` rather than `ACTIVE`. Senders MUST NOT emit any mode value not in `ENABLED_MODES` — `signalling_bytes()` raises if you try. @@ -1178,7 +1178,7 @@ else: signalling_bytes = Link.signalling_bytes(RNS.Reticulum.MTU, self.mode) ``` -When the responder emits an LRPROOF with signalling, the encoded `mtu` is the **min** of its own next-hop view and what arrived in the LINKREQUEST, computed during validation (`RNS/Transport.py:2042-2051`): +When the responder emits an LRPROOF with signalling, the encoded `mtu` is the **min** of its own next-hop view and what arrived in the LINKREQUEST. In RNS 1.2.4 the clamp is performed by transit relays in the DATA forwarding path (`RNS/Transport.py:1539-1556`), which rewrite the LINKREQUEST's signalling bytes in place before forwarding so the responder's `Link.validate_request` (`RNS/Link.py:186-200`) sees the already-clamped value: ```python path_mtu = Link.mtu_from_lr_packet(packet) or Reticulum.MTU @@ -1214,7 +1214,7 @@ Per §6.2, the LRPROOF's signed_data when signalling is present is: signed_data = link_id || responder_X25519_pub || responder_long_term_Ed25519_pub || signalling ``` -A clean-room implementation that omits the signalling bytes when present (or includes them when absent) computes a different signed_data than the responder did, fails signature validation, and the link never establishes. This is the most common interop break in this area; cross-check against `RNS/Link.py:373` (signer) and `:417` (validator). +A clean-room implementation that omits the signalling bytes when present (or includes them when absent) computes a different signed_data than the responder did, fails signature validation, and the link never establishes. This is the most common interop break in this area; cross-check against `RNS/Link.py:373-374` (signer) and `:417` (validator). #### 6.6.6 Disabling MTU discovery @@ -1228,7 +1228,7 @@ A Link goes through five states (`RNS/Link.py:110-114`): `PENDING → HANDSHAKE #### 6.7.1 KEEPALIVE (`context = 0xFA`) -Cadence (`RNS/Link.py:844-846`): +Cadence (`RNS/Link.py:845-847`): ```python def __update_keepalive(self): @@ -1248,7 +1248,7 @@ def send_keepalive(self): Body is a single byte `0xFF` — the "ping" sentinel. The packet is Token-encrypted with the link's session key per §3.1 link-derived form, so the wire body is `iv(16) || ciphertext(...) || hmac(32)`; the decrypted plaintext is just `b'\xff'`. -The **responder** receives this in `Link.receive` at `RNS/Link.py:1149-1153` and answers with the "pong" sentinel: +The **responder** receives this in `Link.receive` at `RNS/Link.py:1149-1153` and answers with the "pong" sentinel (in 1.2.4 the body is `bytes([0xFE])`): ```python elif packet.context == RNS.Packet.KEEPALIVE: @@ -1281,7 +1281,7 @@ elif self.status == Link.STALE: `teardown_reason` is set to `Link.TIMEOUT` (constant value `0x01`) so the application's `link_closed_callback` can distinguish "the peer went dark" from "the peer cleanly closed". -There is also an explicit-cleanup path: after a STALE-induced teardown the watchdog adds a final grace period of `RTT × KEEPALIVE_TIMEOUT_FACTOR + STALE_GRACE` (= `RTT × 4 + 5s`) at line 797 to allow a delayed reply to bring the link back into ACTIVE before final teardown — but in upstream RNS 1.2.0 the `STALE → CLOSED` transition runs immediately on the next watchdog pass without consulting that grace period. The grace constant lives in case a future revision restores the soft-stale window. +There is also an explicit-cleanup path: after a STALE-induced teardown the watchdog adds a final grace period of `RTT × KEEPALIVE_TIMEOUT_FACTOR + STALE_GRACE` (= `RTT × 4 + 5s`) at line 797 to allow a delayed reply to bring the link back into ACTIVE before final teardown — but in upstream RNS 1.2.4 the `STALE → CLOSED` transition runs immediately on the next watchdog pass without consulting that grace period. The grace constant lives in case a future revision restores the soft-stale window. #### 6.7.3 LINKCLOSE (`context = 0xFC`) @@ -1297,7 +1297,7 @@ Wire form: - `packet_type = DATA (0)`, `context = 0xFC`, `dest_hash = link_id`. - Body is the **16-byte link_id**, Token-encrypted by the link's session key. -The peer's receiver path at `RNS/Link.py:1061-1063` calls `teardown_packet(packet)` (line 710-722): +The peer's receiver path at `RNS/Link.py:1061-1063` calls `teardown_packet(packet)` (line 710-728): ```python def teardown_packet(self, packet): @@ -1426,7 +1426,7 @@ A clean-room client that only implements opportunistic LXMF can ignore Channel e ### 7.1 Path requests: peers send `path?` before opportunistic LXMF when no path is known -The path-request preamble in upstream LXMF is **conditional, not unconditional** (verified by `tools/verify_path_request.py` against LXMF 0.9.6): +The path-request preamble in upstream LXMF is **conditional, not unconditional** (verified by `tools/verify_path_request.py` against LXMF 0.9.7): ```python # LXMF/LXMRouter.py::handle_outbound, ~line 1672 @@ -1436,7 +1436,7 @@ if not RNS.Transport.has_path(destination_hash) and lxmessage.method == LXMessag lxmessage.next_delivery_attempt = time.time() + LXMRouter.PATH_REQUEST_WAIT ``` -In other words: a `path?` is sent before the LXM **only when no entry exists in `Transport.path_table`** for the target — `has_path()` is just a key-presence check (`RNS/Transport.py:2570-2576`). Existing-but-stale path entries are NOT replaced by this preamble; LXMF instead leans on the periodic `Transport.jobs` cycle to evict expired path entries (`stale_paths` accumulator at `RNS/Transport.py:747+`), after which the next outbound LXM rediscovers the unknown-path branch and triggers the `request_path`. A second `request_path` is issued from the retry path (`LXMRouter.py:2571+`) once `lxmessage.delivery_attempts >= MAX_PATHLESS_TRIES`, so on a flaky path peers can see multiple `path?` retransmits without intervening DATA — that matches BLE-trace observations. +In other words: a `path?` is sent before the LXM **only when no entry exists in `Transport.path_table`** for the target — `has_path()` is just a key-presence check via `LXMRouter.handle_outbound`. Existing-but-stale path entries are NOT replaced by this preamble; LXMF instead leans on the periodic `Transport.jobs` cycle to evict expired path entries (`stale_paths` accumulator at `RNS/Transport.py:750+`), after which the next outbound LXM rediscovers the unknown-path branch and triggers the `request_path`. A second `request_path` is issued from the retry path (`LXMRouter.py:2568+`) once `lxmessage.delivery_attempts >= MAX_PATHLESS_TRIES`, so on a flaky path peers can see multiple `path?` retransmits without intervening DATA — that matches BLE-trace observations. A `path?` request itself is a regular DATA packet (verified by `tools/verify_path_request.py`): @@ -1452,7 +1452,7 @@ A `path?` request itself is a regular DATA packet (verified by `tools/verify_pat #### 7.2.1 Path-request packet parse rules -The path-request handler at `RNS/Transport.py:2800-2843` parses inbound packets addressed to `path_request_destination` (the dest_hash in §7.1). The handler is registered as the destination's `packet_callback` at `Transport.py:237-240`, so any DATA packet to that dest_hash flows through it. +The path-request handler at `RNS/Transport.py:2806-2850` parses inbound packets addressed to `path_request_destination` (the dest_hash in §7.1). The handler is registered as the destination's `packet_callback` at `Transport.py:241`, so any DATA packet to that dest_hash flows through it. The path-request destination is a **PLAIN destination** with no identity attached, which is why its `dest_hash` derives only from the name: `dest_hash = SHA256(SHA256("rnstransport.path.request")[:10])[:16]` per the PLAIN/GROUP recipe in §1.4.3 (the `identity == None` branch of `Destination.hash` at `RNS/Destination.py:121-130`). The result is a constant — `6b9f66014d9853faab220fba47d02761` — that every node on the mesh resolves identically without needing to discover a per-peer identity first. @@ -1481,7 +1481,7 @@ Three observations that matter for interop: #### 7.2.2 Tag-based deduplication -The handler builds `unique_tag = destination_hash || tag_bytes` and consults `Transport.discovery_pr_tags` (`Transport.py:2829-2839`): +The handler builds `unique_tag = destination_hash || tag_bytes` and consults `Transport.discovery_pr_tags` (`Transport.py:2835-2845`): ```python unique_tag = destination_hash + tag_bytes @@ -1498,13 +1498,13 @@ with Transport.discovery_pr_tags_lock: # ignore duplicate path request ``` -`discovery_pr_tags` is bounded at `Transport.max_pr_tags = 32000` entries (`Transport.py:126`); older entries are aged out by the periodic `Transport.jobs` cycle. **Every node — leaf or transport — that wants to respond to path requests MUST maintain this dedup table** or it will respond to every retransmit, and a transport-enabled node will additionally re-forward to all other interfaces, generating a broadcast storm. +`discovery_pr_tags` is bounded at `Transport.max_pr_tags = 32000` entries (`Transport.py:127`); older entries are aged out by the periodic `Transport.jobs` cycle. **Every node — leaf or transport — that wants to respond to path requests MUST maintain this dedup table** or it will respond to every retransmit, and a transport-enabled node will additionally re-forward to all other interfaces, generating a broadcast storm. The `unique_tag = dest_hash || tag` format means the same tag bytes against different destination_hashes are distinct — so two different requesters racing for the same target with happenstance-identical random tags don't suppress each other. Senders MUST use a fresh random tag per fresh request (the upstream emitter calls `Identity.get_random_hash()`); reusing tags across requests for the same destination_hash makes the second request appear to be a duplicate. #### 7.2.3 The five-way dispatch in `Transport.path_request` -`RNS/Transport.py:2846-2973`. After dedup, the handler calls into `path_request()` which decides how to respond. Five mutually-exclusive branches in priority order: +`RNS/Transport.py:2852-2980`. After dedup, the handler calls into `path_request()` which decides how to respond. Five mutually-exclusive branches in priority order: 1. **`destination_hash` is local** (i.e. it's one of our own registered destinations, line 2873-2875): ```python @@ -1538,11 +1538,11 @@ A `tag` argument hands a previously-built path-response announce body back uncha #### 7.2.5 Timing: `PATH_REQUEST_GRACE` and roaming -When branch 2 fires (transit relay answering on behalf of a remote destination), the rebroadcast is delayed by `PATH_REQUEST_GRACE = 0.4s` (`Transport.py:80, 2917`) — extra grace to let directly-reachable peers respond first if they're in earshot. On `MODE_ROAMING` interfaces an additional `PATH_REQUEST_RG = 1.5s` is added on top (`Transport.py:81, 2922-2923`) so well-connected fixed nodes get a chance to answer before mobile ones. +When branch 2 fires (transit relay answering on behalf of a remote destination), the rebroadcast is delayed by `PATH_REQUEST_GRACE = 0.4s` (`Transport.py:80`) — extra grace to let directly-reachable peers respond first if they're in earshot. On `MODE_ROAMING` interfaces an additional `PATH_REQUEST_RG = 1.5s` is added on top (`Transport.py:81`) so well-connected fixed nodes get a chance to answer before mobile ones. Branch 1 (local destination answers) fires immediately with no grace, since the leaf is the authoritative source for its own destination — there's no point waiting for someone else to potentially answer faster. -Local-client originators also bypass the grace period (`Transport.py:2909-2910`): a relay answering for a destination that lives on a local-client interface can send back the cached announce instantly because the answer doesn't need to compete with peer-mesh announces. +Local-client originators also bypass the grace period (`Transport.py:2915-2920`): a relay answering for a destination that lives on a local-client interface can send back the cached announce instantly because the answer doesn't need to compete with peer-mesh announces. #### 7.2.6 Minimum responsibility for a leaf @@ -1565,7 +1565,7 @@ For a chronological walk-through of the full request → response → path-table The 32-byte `ratchet_pub` field in announces is meant to rotate periodically. The **purpose** is forward secrecy: rotating the ECDH key on a regular cadence limits the plaintext window an adversary can decrypt if a single ratchet privkey leaks. It is **not** what makes your announces visible to the mesh. -The actual replay-and-loop defence in upstream is keyed on **`random_hash`**, not on `ratchet_pub` — see §4.5 step 6.3 (path-table replacement check `not random_blob in random_blobs` at `RNS/Transport.py:1707, 1732, 1745`). Verified by `tools/verify_ratchet_dedup.py`: two announces sharing a `ratchet_pub` but differing in `random_hash[:5]` are both accepted by upstream's replay machinery. +The actual replay-and-loop defence in upstream is keyed on **`random_hash`**, not on `ratchet_pub` — see §4.5 step 6.3 (path-table replacement check `not random_blob in random_blobs` at `RNS/Transport.py:1710, 1735, 1748`). Verified by `tools/verify_ratchet_dedup.py`: two announces sharing a `ratchet_pub` but differing in `random_hash[:5]` are both accepted by upstream's replay machinery. > ⚠️ **Spec correction:** Earlier revisions of this section claimed transit nodes dedup announces on `(destination_hash, ratchet_pub)` tuples and that a non-rotating client becomes invisible to the mesh after one announce. That was wrong on the mechanism: upstream's `RATCHET_INTERVAL = 30 min` × `ANNOUNCE_INTERVAL = 5–15 min` means most upstream announces share a ratchet across 2–6 emissions, so if relays really dropped on `ratchet_pub` equality, upstream wouldn't function. The actual win observed in the bootstrap test (per `agent.md` §5) was incidental — the fix that rotated ratchets per announce also rotated `random_hash`, and it was the latter that mattered. @@ -1611,7 +1611,7 @@ Emitting `context_flag = 0` (no `ratchet_pub` field, body layout per §4.5 step Senders cache the most recent ratchet they've seen for each destination. If you rotate your ratchet faster than relays propagate the announce, in-flight messages may arrive encrypted to your *previous* ratchet. To decrypt these, keep a ring of recent ratchet privkeys and try each in order during decrypt. The fallback to the long-term identity privkey is the ultimate safety net. -Upstream's default ring size is **`Destination.RATCHET_COUNT = 512`** (`RNS/Destination.py:85` in RNS 1.2.0), with a minimum rotation interval of `RATCHET_INTERVAL = 30*60` seconds (line 90) and per-ratchet `RATCHET_EXPIRY = 60*60*24*30` seconds (`RNS/Identity.py:69`). A new ratchet is generated on each `rotate_ratchets()` call and prepended to the in-memory list; `_clean_ratchets` truncates back to `RATCHET_COUNT`. The 512 figure is generous and not a hard interop requirement — it's an in-memory bound on the inbound-decrypt try-list. +Upstream's default ring size is **`Destination.RATCHET_COUNT = 512`** (`RNS/Destination.py:85` in RNS 1.2.4), with a minimum rotation interval of `RATCHET_INTERVAL = 30*60` seconds (line 90) and per-ratchet `RATCHET_EXPIRY = 60*60*24*30` seconds (`RNS/Identity.py:69`). A new ratchet is generated on each `rotate_ratchets()` call and prepended to the in-memory list; `_clean_ratchets` truncates back to `RATCHET_COUNT`. The 512 figure is generous and not a hard interop requirement — it's an in-memory bound on the inbound-decrypt try-list. A minimal client may keep just the current ratchet privkey, accepting that the brief window between rotation and announce-propagation will lose some messages. Mention the trade-off in your implementation notes. @@ -1956,7 +1956,7 @@ Effective HW_MTU is `1196` bytes (`AutoInterface.HW_MTU`, line 44) — chosen to #### 8.6.6 IFAC integration -If the AutoInterface is configured with an `ifac_identity` (out-of-band-shared key), every Reticulum packet on the data port is IFAC-sealed and unmasked using the standard Transport-level IFAC mechanism (`RNS/Transport.py:1338-1387`). Peers with mismatched IFAC keys can see each other's discovery announces but can't decode each other's data — a pragmatic privacy boundary on a shared LAN. +If the AutoInterface is configured with an `ifac_identity` (out-of-band-shared key), every Reticulum packet on the data port is IFAC-sealed and unmasked using the standard Transport-level IFAC mechanism (`RNS/Transport.py:1338-1390`). Peers with mismatched IFAC keys can see each other's discovery announces but can't decode each other's data — a pragmatic privacy boundary on a shared LAN. #### 8.6.7 Source map @@ -2020,7 +2020,7 @@ LoRa devices without an RTC will populate the LXMF `timestamp` field with second Even after a successful initial announce, paths in the mesh expire within minutes. Without a 5–15 minute re-announce loop, the second message any peer tries to send you will fail because the relay's path table has aged out. (See also §7.5.) -There is **no upstream-mandated default** — `RNS/Reticulum.py:745` uses `6*60*60` (6 h) for interface-level *discovery* announces and `RNS/Transport.py:162` uses `2*60*60` (2 h) for transport-management announces, but those are not the cadence end-user destinations announce at. Sideband emits roughly every 30 minutes; the upstream manual recommends 30–60 minutes for a desktop client. Practical guidance for application destinations: +There is **no upstream-mandated default** — `RNS/Reticulum.py:764` uses `6*60*60` (6 h) for interface-level *discovery* announces and `RNS/Transport.py:192` uses `2*60*60` (2 h) for transport-management announces, but those are not the cadence end-user destinations announce at. Sideband emits roughly every 30 minutes; the upstream manual recommends 30–60 minutes for a desktop client. Practical guidance for application destinations: | Deployment | RECOMMENDED cadence | |---|---| @@ -2048,7 +2048,7 @@ logged before any filtering converts hours of "messages aren't arriving" debuggi Real interop bug to plan around: `attermann/microReticulum`'s `Destination::announce` emits 10 fully-random bytes for the announce `random_hash` field rather than the upstream Python form of `5 random bytes || big-endian uint40 unix_seconds` (see §4.1). The Python form is preserved as a comment in the C++ source with a `TODO add in time to random hash` next to it; the timestamp half was never implemented. -Effect on a mixed-vendor mesh: a Python RNS receiver parses `random_hash[5:10]` of a microReticulum announce as a far-future timestamp (median ~year 19403 AD because the random uint40 is uniformly distributed across `0..2^40-1`). The path-table replacement rule at `RNS/Transport.py:1721-1745` rejects subsequent real-timestamped announces from Python sources as "stale" until the path TTL expires. +Effect on a mixed-vendor mesh: a Python RNS receiver parses `random_hash[5:10]` of a microReticulum announce as a far-future timestamp (median ~year 19403 AD because the random uint40 is uniformly distributed across `0..2^40-1`). The path-table replacement rule at `RNS/Transport.py:1723-1755` rejects subsequent real-timestamped announces from Python sources as "stale" until the path TTL expires. Symptom: a microReticulum repeater works fine when it's the only path; in a mesh that also has Python relays, paths "stick" to the microReticulum side even when shorter / fresher Python paths come up, until natural TTL expiry. First-contact path-table population is unaffected — the bug only surfaces on path replacement. @@ -2064,7 +2064,7 @@ The repeater repo's `pre_build.py` patches several other microReticulum protocol A **Resource** transfers a payload that exceeds the per-packet content limit of an established Reticulum Link. It is the only way to carry an LXMF body, NomadNet page, or file larger than ~360 bytes (`LINK_PACKET_MAX_CONTENT`) over a Link. Resource is built **on top of** an active Link — it relies on the Link's session key for encryption (§3.1 link-derived form) and on the Link's bidirectional DATA channel for control traffic. -The complete reference is `RNS/Resource.py` (1383 lines in RNS 1.2.0); `RNS/Packet.py:72-78` defines the context constants. This section describes the wire-level invariants a clean-room implementation must respect; many implementation choices (window sizing heuristics, watchdog timers, EIFR computation) are private and listed only when their absence would cause an interop break. +The complete reference is `RNS/Resource.py` (1380 lines in RNS 1.2.4); `RNS/Packet.py:74-79` defines the context constants. This section describes the wire-level invariants a clean-room implementation must respect; many implementation choices (window sizing heuristics, watchdog timers, EIFR computation) are private and listed only when their absence would cause an interop break. ### 10.1 When Resource runs @@ -2163,6 +2163,20 @@ The advertisement is sent once on `Resource.advertise()`; if no part requests ar > implementing `delivery_resource_advertised(resource)` returning > `False` (§5.8.3 / §16.9) is the upstream-blessed way to refuse > oversized advertisements. +> +> Upstream RNS adopted this cap in 1.1.9 after a CVE-class report: +> `Resource.assemble` uses `bz2.BZ2Decompressor.decompress(data, +> max_length=self.max_decompressed_size)` and rejects the resource +> if `decompressor.eof` is False after the bounded read +> (`RNS/Resource.py:686-691`). The Channel-mode counterpart is +> `Buffer.RawChannelReader` which caps each chunk at +> `RawChannelWriter.MAX_CHUNK_LEN` via the same `max_length` mechanism +> (`RNS/Buffer.py:95-97`). Clean-room implementations should mirror +> this — a `bz2.BZ2Decompressor.decompress(data, max_length=N)` plus +> `eof` check is the minimum. **Do not use the one-shot +> `bz2.decompress()` API for resource bodies** — it has no output +> bound and will allocate as much memory as the input legitimately +> expands to. ### 10.5 RESOURCE_REQ — receiver requests parts @@ -2497,7 +2511,7 @@ Registered via `Destination.register_request_handler(path, response_generator, a ### 11.5 RequestReceipt — initiator-side state machine -`RNS/Link.py:1348-1448`. When `Link.request()` returns a `RequestReceipt`, the initiator can attach: +`RNS/Link.py:1335-1530`. When `Link.request()` returns a `RequestReceipt`, the initiator can attach: - `response_callback(receipt)` — fires when the response has fully arrived (single packet OR resource concluded). - `failed_callback(receipt)` — fires on timeout or link teardown. @@ -2654,7 +2668,7 @@ None of these are wire-spec — they're caller conventions layered on top of §1 | `RNS/Link.py:478-527` | `Link.request()` — initiator-side packing and dispatch by size | | `RNS/Link.py:853-904` | `Link.handle_request()` — server-side path lookup + auth + response dispatch | | `RNS/Link.py:906-925` | `Link.handle_response()` — initiator-side response correlation | -| `RNS/Link.py:1348-1448` | `RequestReceipt` — callback machinery | +| `RNS/Link.py:1335-1530` | `RequestReceipt` — callback machinery | | `RNS/Destination.py::register_request_handler` | Server-side handler registration | | `RNS/Destination.py:35-40` | `ALLOW_NONE/ALLOW_LIST/ALLOW_ALL` constants | | `RNS/Packet.py:81-82` | `REQUEST = 0x09`, `RESPONSE = 0x0A` context constants | @@ -2685,7 +2699,7 @@ For an inbound DATA packet (`packet_type == DATA`, `destination_type` not LINK) - `packet.transport_id == Transport.identity.hash` (the originator picked us as the next hop), AND - `packet.destination_hash` is in `Transport.path_table`, -the relay rewrites the wire bytes according to `path_table[dest][HOPS]` and re-transmits on `path_table[dest][RVCD_IF]`. From `RNS/Transport.py:1497-1573`, three cases by `remaining_hops`: +the relay rewrites the wire bytes according to `path_table[dest][HOPS]` and re-transmits on `path_table[dest][RVCD_IF]`. From `RNS/Transport.py:1500-1580`, three cases by `remaining_hops`: #### 12.2.1 `remaining_hops > 1` — forward as HEADER_2 @@ -2719,7 +2733,7 @@ The destination is registered on the relay itself (it's both our path-table next #### 12.2.4 LINKREQUEST forwarding extras -When the forwarded packet is a `LINKREQUEST`, the relay also writes a `link_table` entry keyed by the link_id (computed via §6.3's `link_id_from_lr_packet`). Entry contents (`Transport.py:1553-1561`): +When the forwarded packet is a `LINKREQUEST`, the relay also writes a `link_table` entry keyed by the link_id (computed via §6.3's `link_id_from_lr_packet`). Entry contents (`Transport.py:1556-1565`): ``` [ now, # 0 IDX_LT_TIMESTAMP @@ -2739,7 +2753,7 @@ The relay also performs the §6.6 MTU clamp at this point: if the LINKREQUEST ca #### 12.2.5 Non-LINKREQUEST DATA — reverse_table entry -For any other forwarded DATA (the much-more-common opportunistic LXMF case), the relay writes a `reverse_table` entry keyed by `packet.getTruncatedHash()` (`Transport.py:1567-1571`): +For any other forwarded DATA (the much-more-common opportunistic LXMF case), the relay writes a `reverse_table` entry keyed by `packet.getTruncatedHash()` (`Transport.py:1570-1574`): ``` [ packet.receiving_interface, # 0 IDX_RT_RCVD_IF — interface to send PROOF back through @@ -2751,7 +2765,7 @@ The reverse_table is what lets the eventual PROOF receipt (§6.5) trace its way ### 12.3 ANNOUNCE rebroadcasting -When an inbound ANNOUNCE validates (per §4.5) AND the destination is non-local AND `transport_enabled OR is_from_local_client`, the relay queues a rebroadcast. From `Transport.py:1810-1890`: +When an inbound ANNOUNCE validates (per §4.5) AND the destination is non-local AND `transport_enabled OR is_from_local_client`, the relay queues a rebroadcast. From `Transport.py:1810-1895`: ```python if (Reticulum.transport_enabled() or is_from_local_client) and packet.context != PATH_RESPONSE: @@ -2781,7 +2795,7 @@ The cap is per-interface, not global — a relay with multiple interfaces budget ### 12.4 Path table management -`Transport.path_table[destination_hash]` entry shape (`Transport.py:3439-3446`): +`Transport.path_table[destination_hash]` entry shape (`Transport.py:3457-3464`): ``` [ timestamp, # 0 IDX_PT_TIMESTAMP — when last refreshed @@ -2807,7 +2821,7 @@ The wide spread of defaults reflects expected churn rates: AP-mode interfaces ha #### 12.4.2 Eviction -`Transport.jobs` runs a `stale_paths` accumulator that walks `path_table` and pops entries whose `expires` timestamp has passed (`Transport.py:747-769`). Eviction is silent — no notification to the application; the next outbound message to the destination just re-discovers it via `request_path` per §7.1. +`Transport.jobs` runs a `stale_paths` accumulator that walks `path_table` and pops entries whose `expires` timestamp has passed (`Transport.py:750-770`). Eviction is silent — no notification to the application; the next outbound message to the destination just re-discovers it via `request_path` per §7.1. A relay also evicts path entries whose underlying interface has been removed (`receiving_interface not in Transport.interfaces`). This handles the case where a TCP client disconnects. @@ -2821,7 +2835,7 @@ Once a Link's LINKREQUEST has been forwarded by a relay (§12.2.4 wrote the `lin #### 12.5.1 LRPROOF forwarding -When an LRPROOF arrives whose `dest_hash` (= link_id) is in the relay's `link_table` AND the proof arrives on the next-hop interface (`packet.receiving_interface == link_entry[IDX_LT_NH_IF]`), the relay validates the signature against the destination's known long-term public key (recalled via `Identity.recall(link_entry[DSTHASH])`) and forwards on the receive interface (`Transport.py:2110-2138`): +When an LRPROOF arrives whose `dest_hash` (= link_id) is in the relay's `link_table` AND the proof arrives on the next-hop interface (`packet.receiving_interface == link_entry[IDX_LT_NH_IF]`), the relay validates the signature against the destination's known long-term public key (recalled via `Identity.recall(link_entry[DSTHASH])`) and forwards on the receive interface (`Transport.py:2110-2145`): ```python new_raw = packet.raw[0:1] + struct.pack("!B", packet.hops) + packet.raw[2:] @@ -2837,7 +2851,7 @@ For a `DATA` packet with `destination_type == LINK` whose `dest_hash` is in `lin #### 12.5.3 PROOF receipt forwarding via `reverse_table` -`Transport.py:2196-2205`. When a PROOF arrives whose `dest_hash` is in `reverse_table` (i.e. an opportunistic-DATA proof being routed back to its originator), the relay pops the entry, checks the proof arrived on the correct outbound interface (`receiving_interface == reverse_entry[IDX_RT_OUTB_IF]`), and forwards on the originally-receiving interface: +`Transport.py:2199-2208`. When a PROOF arrives whose `dest_hash` is in `reverse_table` (i.e. an opportunistic-DATA proof being routed back to its originator), the relay pops the entry, checks the proof arrived on the correct outbound interface (`receiving_interface == reverse_entry[IDX_RT_OUTB_IF]`), and forwards on the originally-receiving interface: ```python new_raw = packet.raw[0:1] + struct.pack("!B", packet.hops) + packet.raw[2:] @@ -2852,7 +2866,7 @@ Two related state mechanisms a transport node maintains: #### 12.6.1 `discovery_path_requests` -When a transport-enabled relay receives a path? for a destination it doesn't know AND doesn't have a local client to forward to, it records a `discovery_path_requests[dest_hash]` entry (`Transport.py:2949-2963`): +When a transport-enabled relay receives a path? for a destination it doesn't know AND doesn't have a local client to forward to, it records a `discovery_path_requests[dest_hash]` entry (`Transport.py:2955-2970`): ```python pr_entry = { @@ -2867,7 +2881,7 @@ Then forwards the path? to every other interface preserving the original tag. Th #### 12.6.2 `tunnels` -A tunnel is an interface-level path mechanism for handling temporarily-disconnected interfaces (e.g. a mobile peer that comes and goes). The `tunnels[interface_tunnel_id]` state lets the relay reconstruct paths through the interface when it reconnects, without requiring all paths to be re-discovered from scratch. The shape (`Transport.py:783-832`): +A tunnel is an interface-level path mechanism for handling temporarily-disconnected interfaces (e.g. a mobile peer that comes and goes). The `tunnels[interface_tunnel_id]` state lets the relay reconstruct paths through the interface when it reconnects, without requiring all paths to be re-discovered from scratch. The shape (`Transport.py:792-832`): ``` [ now, # 0 IDX_TT_TIMESTAMP @@ -2884,8 +2898,8 @@ Each path inside the tunnel's `paths_dict` mirrors a `path_table` entry. When th When multiple processes on one host share a single Reticulum stack (via `share_instance = Yes` in the rnsd config), one process owns `Transport` and the others connect to it as **local clients** via a small TCP loopback interface. The shared instance treats local-client traffic specially: -- `from_local_client` and `for_local_client` are computed on every inbound packet (`Transport.py:1450-1454`). -- Path-table entries with `IDX_PT_HOPS == 0` mean "destination is a local client" — the §2.3 originator-side HEADER_1 conversion applies for hops==1 too, so the shared instance gets a transport_id-tagged packet (`Transport.py:1094-1105`). +- `from_local_client` and `for_local_client` are computed on every inbound packet (`Transport.py:1453-1456`). +- Path-table entries with `IDX_PT_HOPS == 0` mean "destination is a local client" — the §2.3 originator-side HEADER_1 conversion applies for hops==1 too, so the shared instance gets a transport_id-tagged packet (`Transport.py:1097-1108`). - Local-client originated path? requests are forwarded to every external interface, fanning out the search across the shared mesh (§7.2 dispatch branch 3). The wire protocol for shared-instance loopback is just the same Reticulum packets over a TCP loopback interface — no special framing or commands. What's "shared" is the path_table and announce dispatch, not the wire format. @@ -2894,15 +2908,15 @@ The wire protocol for shared-instance loopback is just the same Reticulum packet | File | What | |---|---| -| `RNS/Transport.py:1497-1573` | DATA forwarding (HEADER_1↔HEADER_2 conversion for relay) | -| `RNS/Transport.py:1553-1561` | `link_table` entry shape | -| `RNS/Transport.py:1567-1571` | `reverse_table` entry shape | -| `RNS/Transport.py:1810-1969` | ANNOUNCE rebroadcast queue and per-interface dispatch | -| `RNS/Transport.py:2110-2138` | LRPROOF forwarding via `link_table` | -| `RNS/Transport.py:2196-2205` | PROOF receipt forwarding via `reverse_table` | -| `RNS/Transport.py:3439-3446` | `path_table` entry shape (IDX_PT_*) | -| `RNS/Transport.py:747-832` | stale-path / stale-tunnel eviction | -| `RNS/Transport.py:2949-2963` | `discovery_path_requests` recursive search | +| `RNS/Transport.py:1500-1580` | DATA forwarding (HEADER_1↔HEADER_2 conversion for relay) | +| `RNS/Transport.py:1556-1565` | `link_table` entry shape | +| `RNS/Transport.py:1570-1574` | `reverse_table` entry shape | +| `RNS/Transport.py:1810-1975` | ANNOUNCE rebroadcast queue and per-interface dispatch | +| `RNS/Transport.py:2110-2145` | LRPROOF forwarding via `link_table` | +| `RNS/Transport.py:2199-2208` | PROOF receipt forwarding via `reverse_table` | +| `RNS/Transport.py:3457-3464` | `path_table` entry shape (IDX_PT_*) | +| `RNS/Transport.py:750-832` | stale-path / stale-tunnel eviction | +| `RNS/Transport.py:2955-2970` | `discovery_path_requests` recursive search | | `RNS/Interfaces/Interface.py:232-272` | per-interface `announce_queue` and `ANNOUNCE_CAP` enforcement | --- @@ -2917,14 +2931,14 @@ Upstream RNS spawns the following persistent daemon threads at `Transport.start( | Thread | Source | Cadence | Purpose | |---|---|---|---| -| **`Transport.jobloop`** | `RNS/Transport.py:280, 483-486` | every `job_interval = 0.250s` | Runs `Transport.jobs()` — the catch-all maintenance pass: link state checks, announce-queue drain, stale-path eviction, hashlist cleanup, reverse-table cleanup, tunnels housekeeping. | -| **`Transport.count_traffic_loop`** | `RNS/Transport.py:281, 449-480` | every 1s | Snapshots per-interface RX/TX byte counters into rolling-window deques for bandwidth/airtime accounting. | -| **`Link.__watchdog_job`** | `RNS/Link.py:746-821` | per-link, RTT-driven | One per active Link. Drives keepalive emission (initiator side), STALE→CLOSED transitions, and link-establishment timeouts. Sleeps `min(WATCHDOG_MAX_SLEEP=5s, RTT-derived)` between iterations. | -| **`Resource.__watchdog_job`** | `RNS/Resource.py:564-642` | per-resource | One per in-progress Resource. Detects retransmit timeouts, advertisement retries, and PRF-wait timeouts. | -| **`AnnounceHandler` callbacks** | `RNS/Transport.py:1995-2016` | per inbound announce | Each accepted announce fires its registered handler **on a fresh daemon thread** — the dispatcher does not serialize. Two announces from the same destination back-to-back run two handler threads concurrently. | +| **`Transport.jobloop`** | `RNS/Transport.py:283, 486-489` | every `job_interval = 0.250s` | Runs `Transport.jobs()` — the catch-all maintenance pass: link state checks, announce-queue drain, stale-path eviction, hashlist cleanup, reverse-table cleanup, tunnels housekeeping. | +| **`Transport.count_traffic_loop`** | `RNS/Transport.py:284, 452-483` | every 1s | Snapshots per-interface RX/TX byte counters into rolling-window deques for bandwidth/airtime accounting. | +| **`Link.__watchdog_job`** | `RNS/Link.py:751-828` | per-link, RTT-driven | One per active Link. Drives keepalive emission (initiator side), STALE→CLOSED transitions, and link-establishment timeouts. Sleeps `min(WATCHDOG_MAX_SLEEP=5s, RTT-derived)` between iterations. | +| **`Resource.__watchdog_job`** | `RNS/Resource.py:564-670` | per-resource | One per in-progress Resource. Detects retransmit timeouts, advertisement retries, and PRF-wait timeouts. | +| **`AnnounceHandler` callbacks** | `RNS/Transport.py:1995-2025` | per inbound announce | Each accepted announce fires its registered handler **on a fresh daemon thread** — the dispatcher does not serialize. Two announces from the same destination back-to-back run two handler threads concurrently. | | **Per-interface RX threads** | `RNS/Interfaces/*Interface.py` | always | Each interface (TCP, KISS, RNode, AutoInterface) has its own blocking-read RX thread that calls `Transport.inbound(raw, self)` on each complete frame. | | **`process_announce_queue`** | `RNS/Interfaces/Interface.py:266-267` | one-shot timer per drain | Per-interface `announce_queue` drain uses `threading.Timer` to schedule the next emission at the airtime-cap-derived wait time. Not a long-running thread but a chain of one-shots. | -| **`Resource.__advertise_job`** | `RNS/Resource.py:520-541` | per-resource | One-shot daemon thread that performs the resource hashmap construction (which can take seconds on a large body) so the calling thread doesn't block. | +| **`Resource.__advertise_job`** | `RNS/Resource.py:520-542` | per-resource | One-shot daemon thread that performs the resource hashmap construction (which can take seconds on a large body) so the calling thread doesn't block. | A clean-room implementation with cooperative scheduling (e.g. asyncio, embedded RTOS task model) needs to provide equivalent behavior for each row. The key invariants — not the exact thread inventory — are what matter for interop: @@ -2967,7 +2981,7 @@ What upstream **guarantees** to application-level callbacks: - **`Link.set_link_established_callback`** — fires once when a link transitions PENDING → ACTIVE. On the receive thread. - **`Link.set_link_closed_callback`** — fires once when a link transitions to CLOSED, regardless of cause (timeout, peer close, local teardown). On the watchdog thread or the receive thread depending on which path triggered the close. - **`PacketReceipt.set_delivery_callback`** — fires once when a PROOF arrives matching this receipt. On the receive thread. -- **`AnnounceHandler.received_announce`** — fires once per accepted announce, **on a fresh daemon thread**. This is the only callback that's NOT on the receive thread (`Transport.py:1995-2016`). +- **`AnnounceHandler.received_announce`** — fires once per accepted announce, **on a fresh daemon thread**. This is the only callback that's NOT on the receive thread (`Transport.py:1995-2025`). - **`Resource.callback`** — fires once on resource conclude, on the assembly thread. Implications for a clean-room implementation: @@ -2997,15 +3011,15 @@ A client running on a constrained device (less RAM, slower CPU) can scale all of | File | What | |---|---| -| `RNS/Transport.py:280-281` | top-level thread spawn at startup | -| `RNS/Transport.py:128-148` | the lock inventory (Transport-side) | -| `RNS/Transport.py:172, 175, 186` | `job_interval`, `links_check_interval`, `tables_last_culled` | -| `RNS/Transport.py:483-486` | `jobloop` — the periodic driver | -| `RNS/Transport.py:489+` | `jobs()` body (held under `jobs_lock`) | -| `RNS/Transport.py:1995-2016` | announce-handler dispatch (fresh thread per callback) | -| `RNS/Link.py:746-821` | per-link `__watchdog_job` | -| `RNS/Resource.py:564-642` | per-resource `__watchdog_job` | -| `RNS/Resource.py:520-541` | one-shot `__advertise_job` | +| `RNS/Transport.py:283-284` | top-level thread spawn at startup | +| `RNS/Transport.py:132-148` | the lock inventory (Transport-side) | +| `RNS/Transport.py:174, 176, 187` | `job_interval`, `links_check_interval`, `tables_last_culled` | +| `RNS/Transport.py:486-489` | `jobloop` — the periodic driver | +| `RNS/Transport.py:492+` | `jobs()` body (held under `jobs_lock`) | +| `RNS/Transport.py:1995-2025` | announce-handler dispatch (fresh thread per callback) | +| `RNS/Link.py:751-828` | per-link `__watchdog_job` | +| `RNS/Resource.py:564-670` | per-resource `__watchdog_job` | +| `RNS/Resource.py:520-542` | one-shot `__advertise_job` | | `RNS/Interfaces/*Interface.py` | per-interface RX thread | --- @@ -3209,7 +3223,7 @@ Embedded clean-room implementations need to know up front which data structures | Structure | Cap | Where | Notes | |---|---|---|---| | `Transport.path_table` | (unbounded — count grows with mesh size) | §12.4 | Grows with the number of distinct destinations the node has heard about. Bounded effectively by TTL eviction (§12.4.2): AP_PATH_TIME (1h), ROAMING_PATH_TIME (4h), PATHFINDER_E (30d). On a tiny LoRa mesh this is dozens of entries; on a global Reticulum mesh routed through a TCP backbone it can be thousands. | -| `path_table[dest][IDX_PT_RANDBLOBS]` (per-destination random_blob history) | `Transport.MAX_RANDOM_BLOBS = 32` | §4.5 step 6.3, §12.3.2 | Sliding window. Caps memory growth from one destination's announce stream. | +| `path_table[dest][IDX_PT_RANDBLOBS]` (per-destination random_blob history) | `Transport.MAX_RANDOM_BLOBS = 64` (RNS 1.2.4) | §4.5 step 6.3, §12.3.2 | Sliding window. Caps memory growth from one destination's announce stream. | | `Transport.announce_table` | (unbounded — populated only for in-flight announces awaiting rebroadcast) | §12.3 | Drains via `Transport.jobs` retransmit timer, capped at `PATHFINDER_R = 4` retries each. Effective cap: number of announces seen × time. | | `Transport.discovery_pr_tags` | `Transport.max_pr_tags = 32000` | §7.2.2 | Path-request dedup table. Older entries aged out by `Transport.jobs`. | | `Transport.path_requests` | (unbounded — one entry per recently-issued path? request) | §7.1 | Aged out at `Transport.PATH_REQUEST_GATE_TIMEOUT = 120s`. | @@ -3327,7 +3341,7 @@ A reader hitting §2.3 might wonder "do I need this?" Three different answers: - **Cat 1 (e.g. MeshChat, Sideband):** No — `RNS.Transport.outbound` at lines 1074-1083 does the conversion automatically when you call `Packet.send()` to a destination with `path_table[dest][HOPS] > 1`. Your app just calls `LXMessage.send()` or `Packet.send()` and §2.3 happens invisibly. You can read §2.3 to understand WHY some captures show HEADER_2 with a transport_id, but you have no code to write. - **Cat 2 (wrappers):** Same as Cat 1 — the wrapped Python RNS does the conversion. Your wrapper is just relaying API calls. -- **Cat 3 (clean-room):** **Yes, you implement §2.3 yourself.** Failure to do so means your packets aren't forwarded by transit relays — they're processed and dropped silently per `Transport.py:1497` (only HEADER_2 packets with `transport_id == relay.identity.hash` enter the forwarding branch). The symptom is "messages I send through a relay never arrive, but direct-link messages do." Sideband works in shared-instance and direct-TCP modes both because **upstream** does the conversion; a clean-room app working only via shared-instance is masking the missing §2.3. +- **Cat 3 (clean-room):** **Yes, you implement §2.3 yourself.** Failure to do so means your packets aren't forwarded by transit relays — they're processed and dropped silently per `Transport.py:1500` (only HEADER_2 packets with `transport_id == relay.identity.hash` enter the forwarding branch). The symptom is "messages I send through a relay never arrive, but direct-link messages do." Sideband works in shared-instance and direct-TCP modes both because **upstream** does the conversion; a clean-room app working only via shared-instance is masking the missing §2.3. ### 17.4 Pragmatic implication diff --git a/agent.md b/agent.md index d86c0bf..a0986ca 100644 --- a/agent.md +++ b/agent.md @@ -100,7 +100,7 @@ Initial confidence assessment (subjective, not authoritative — re-do this audi | §7.6 `TCPServerInterface.OUT` override | Source-cited; matches behavior observed in the mobile-app's local-transport experiments. | | §8 KISS / HDLC framing | High — both work in production on the reference clients | | §9.1–§9.8 Implementation gotchas | Each was a real bug that bit a real implementation. High confidence each is real; some lack formal test scripts. | -| §10 Resource fragmentation | Source-cited from `RNS/Resource.py` against RNS 1.2.0; not yet runtime-verified in this repo's `tools/`. | +| §10 Resource fragmentation | Source-cited from `RNS/Resource.py` against RNS 1.2.4; not yet runtime-verified in this repo's `tools/`. | | §11 Test vectors | The vectors themselves are verified; the test-vectors/ directory needs to be populated in this repo (currently partially populated). | | §12 Source map | High | diff --git a/flows/forward-announce.md b/flows/forward-announce.md index 7046ec1..832d5f0 100644 --- a/flows/forward-announce.md +++ b/flows/forward-announce.md @@ -1,6 +1,6 @@ # Flow: forward an announce (transport-node rebroadcast) -What a transport-mode node does when it receives an inbound announce destined for a non-local destination. This is the flow that makes the mesh actually mesh — without it, announces never propagate beyond direct radio range. Pinned against **RNS 1.2.0**; cross-references [`../SPEC.md`](../SPEC.md) §4.5 (validation), §12.3 (rebroadcast rules), §12.4 (path table). +What a transport-mode node does when it receives an inbound announce destined for a non-local destination. This is the flow that makes the mesh actually mesh — without it, announces never propagate beyond direct radio range. Pinned against **RNS 1.2.4**; cross-references [`../SPEC.md`](../SPEC.md) §4.5 (validation), §12.3 (rebroadcast rules), §12.4 (path table). This flow only runs on a node with `enable_transport = Yes` per §12.1. Leaf clients can ignore it entirely. @@ -14,7 +14,7 @@ The receive-announce flow ([`receive-announce.md`](receive-announce.md)) runs fi ### 2. Eligibility checks -`RNS/Transport.py:1822`. Three conditions all must hold: +`RNS/Transport.py:1825`. Three conditions all must hold: ```python if (Reticulum.transport_enabled() or is_from_local_client) \ @@ -72,7 +72,7 @@ The actual code is in `Transport.py:1196-1300, 1810-1969`; the structure above i ### 5. Per-interface `announce_queue` drain -Each interface independently throttles its outbound announces against `interface.announce_cap` (default `Reticulum.ANNOUNCE_CAP = 2.0` = 2% airtime). `Interface.process_announce_queue` (`RNS/Interfaces/Interface.py:232-272`) drains the queue at a rate the cap permits, **picking the lowest-hop-count entry first** so closer destinations propagate before further ones: +Each interface independently throttles its outbound announces against `interface.announce_cap` (default `Reticulum.ANNOUNCE_CAP = 2.0` = 2% airtime). `Interface.process_announce_queue` (`RNS/Interfaces/Interface.py:237-272`) drains the queue at a rate the cap permits, **picking the lowest-hop-count entry first** so closer destinations propagate before further ones: ```python min_hops = min(e["hops"] for e in self.announce_queue) diff --git a/flows/path-discovery.md b/flows/path-discovery.md index 93fcf56..92acff2 100644 --- a/flows/path-discovery.md +++ b/flows/path-discovery.md @@ -2,7 +2,7 @@ What happens chronologically when one node wants to reach another whose path is missing from `Transport.path_table`. This flow drives the well-known `rnstransport.path.request` destination (dest_hash `6b9f66014d9853faab220fba47d02761`, see [`../SPEC.md`](../SPEC.md) §1.2 and §7.1) and the path-response announce that returns along the reverse path. -Pinned against **RNS 1.2.0**. +Pinned against **RNS 1.2.4**. Out of scope: the LXMF outbound retry that gates this flow (`flows/send-opportunistic-lxmf.md` step 4); the periodic `Transport.jobs` cycle that ages out stale paths; transport-mode rebroadcast of a known path on behalf of a remote requester (Tier 3 spec gap, see `../todo.md`). @@ -20,7 +20,7 @@ Out of scope: the LXMF outbound retry that gates this flow (`flows/send-opportun ### 1. Initiator: `RNS.Transport.request_path(dest_hash)` -`RNS/Transport.py:2707-2745`. Triggered by: +`RNS/Transport.py:2713-2750`. Triggered by: - `LXMRouter.handle_outbound` when `not has_path(dest) and method == OPPORTUNISTIC` (see `send-opportunistic-lxmf.md` step 4). - `LXMRouter.process_outbound` retry path after `MAX_PATHLESS_TRIES`. @@ -45,7 +45,7 @@ Transport.path_requests[destination_hash] = time.time() The well-known dest hash `6b9f66014d9853faab220fba47d02761` is computed as `SHA256(SHA256("rnstransport.path.request")[:10])[:16]` per [`../SPEC.md`](../SPEC.md) §1.2 (the `identity=None` branch of `Destination.hash`). It is the same on every node — no per-node uniqueness — because the destination is `PLAIN` with no identity attached. -The packet is `DATA` (not ANNOUNCE), `BROADCAST` transport, `HEADER_1`, and `context = NONE`. The body is **unencrypted** because the outer destination is `PLAIN` (`RNS/Packet.py:192-194` skips encryption for PLAIN-type destinations). +The packet is `DATA` (not ANNOUNCE), `BROADCAST` transport, `HEADER_1`, and `context = NONE`. The body is **unencrypted** because the outer destination is `PLAIN` — `Packet.pack` falls through to `self.destination.encrypt(self.data)` at `RNS/Packet.py:215`, and `Destination.encrypt` returns `plaintext` unchanged for `Destination.PLAIN` (`RNS/Destination.py:592-593`). Initiator records `Transport.path_requests[destination_hash] = time.time()` for `PATH_REQUEST_GATE_TIMEOUT = 120s` rate-limiting (prevents the same node from repeating identical path? requests faster than `PATH_REQUEST_MI = 20s`). diff --git a/flows/receive-announce.md b/flows/receive-announce.md index 75f26c8..ce0424c 100644 --- a/flows/receive-announce.md +++ b/flows/receive-announce.md @@ -2,7 +2,7 @@ What happens chronologically on a node when an `ANNOUNCE` packet arrives on one of its interfaces. Without this flow working, nothing else does — no peer is ever known, every outbound message fails at `Identity.recall(dest_hash)`, and no path table entry exists for routing. -Pinned against **RNS 1.2.0**. Line numbers below are from that version. +Pinned against **RNS 1.2.4**. Line numbers below are from that version. This flow covers the **receive + ingest** path that every Reticulum node runs (leaf clients and transport nodes alike). The **rebroadcast** logic that transport nodes additionally run is covered separately in `forward-announce.md` (TODO — Tier 3 in `../todo.md`). @@ -22,12 +22,12 @@ This flow covers the **receive + ingest** path that every Reticulum node runs (l Steps 1-6 of [`receive-opportunistic-lxmf.md`](receive-opportunistic-lxmf.md) apply unchanged: -1. KISS / HDLC deframer (and RNode 2-frame split-packet reassembly per [`../SPEC.md`](../SPEC.md) §8.3 if applicable) hands the raw Reticulum packet bytes to `RNS.Transport.inbound(raw, interface)` (`RNS/Transport.py:1327`). +1. KISS / HDLC deframer (and RNode 2-frame split-packet reassembly per [`../SPEC.md`](../SPEC.md) §8.3 if applicable) hands the raw Reticulum packet bytes to `RNS.Transport.inbound(raw, interface)` (`RNS/Transport.py:1330`). 2. IFAC unmask if the interface has `ifac_identity` configured. 3. `packet = RNS.Packet(None, raw); packet.unpack(); packet.hops += 1`. 4. RSSI / SNR / Q stats are attached to the packet. 5. `Transport.packet_filter(packet)` dedup check; the packet's hash is added to `Transport.packet_hashlist` unless it belongs to a link or is an LRPROOF. -6. Dispatch by `packet.packet_type`. For an announce: `packet_type == ANNOUNCE (1)`, `destination_type == SINGLE (0)`, `transport_type == BROADCAST (0)`. Control reaches `RNS/Transport.py:1628`: +6. Dispatch by `packet.packet_type`. For an announce: `packet_type == ANNOUNCE (1)`, `destination_type == SINGLE (0)`, `transport_type == BROADCAST (0)`. Control reaches `RNS/Transport.py:1631`: ```python if packet.packet_type == RNS.Packet.ANNOUNCE: @@ -36,7 +36,7 @@ if packet.packet_type == RNS.Packet.ANNOUNCE: ### 2. Quick signature-only validation (cheap pre-filter) -`RNS/Transport.py:1629`: +`RNS/Transport.py:1632`: ```python if interface != None and RNS.Identity.validate_announce(packet, only_validate_signature=True): @@ -52,7 +52,7 @@ If the signature fails, the announce is silently dropped here without the deque ### 3. Ingress rate limiting (defer-and-hold) -`RNS/Transport.py:1632-1643`: +`RNS/Transport.py:1635-1646`: ```python if not packet.destination_hash in Transport.path_table: @@ -74,7 +74,7 @@ Held announces are released later by `Interface.process_held_announces()` (`RNS/ ### 4. Local-destination short-circuit -`RNS/Transport.py:1645-1648`: +`RNS/Transport.py:1648-1651`: ```python local_destination = None @@ -87,7 +87,7 @@ If the announce's `destination_hash` matches one of our own registered destinati ### 5. Full announce validation -`RNS/Transport.py:1650`: +`RNS/Transport.py:1653`: ```python if local_destination == None and RNS.Identity.validate_announce(packet): @@ -108,7 +108,7 @@ If any check fails, the function returns `False` and the wider dispatch skips ev ### 6. `random_blob` extraction and `received_from` resolution -`RNS/Transport.py:1651-1677`. The 10-byte `random_hash` is re-extracted as `random_blob` (same bytes, different name in the routing-table code): +`RNS/Transport.py:1654-1680`. The 10-byte `random_hash` is re-extracted as `random_blob` (same bytes, different name in the routing-table code): ```python random_blob = packet.data[64+10 : 64+10+10] # bytes 74..84 of the announce body @@ -120,7 +120,7 @@ The trailing 5 bytes of `random_blob` are the emission timestamp (SPEC.md §4.1) ### 7. Path table population -`RNS/Transport.py:1679-1969`. The full decision tree for whether and how to update `path_table[destination_hash]` is roughly: +`RNS/Transport.py:1682-1975`. The full decision tree for whether and how to update `path_table[destination_hash]` is roughly: ``` local_and_hops_condition := (packet.hops < PATHFINDER_M+1) # default 128 @@ -160,7 +160,7 @@ A leaf client that **doesn't** maintain a path table can still send opportunisti ### 8. Announce handler dispatch -`RNS/Transport.py:1970-2024`. With path-table updated, the receiver fans the announce out to any application-level listeners that registered via `RNS.Transport.register_announce_handler`: +`RNS/Transport.py:1976-2030`. With path-table updated, the receiver fans the announce out to any application-level listeners that registered via `RNS.Transport.register_announce_handler`: ```python for handler in Transport.announce_handlers: @@ -183,7 +183,7 @@ Three details that bite implementers: 1. **`aspect_filter` is a name string, not a name_hash.** A handler that wants only `lxmf.delivery` announces sets `self.aspect_filter = "lxmf.delivery"` and Reticulum recomputes the expected dest_hash from `(name, announced_identity)` for each candidate. So a single handler can match across all peers' `lxmf.delivery` destinations without enumerating them. 2. **`PATH_RESPONSE` is filtered OUT by default.** A handler must opt in via `handler.receive_path_responses = True` to see path-response announces; otherwise it sees only periodic re-announces. The path-table update in step 7 still happens regardless. -3. **The handler is called in a fresh daemon thread.** A slow handler doesn't block subsequent inbound packets, but it also can't assume serialised execution — two announces from the same destination arriving back-to-back will run two handler threads concurrently. Handlers that mutate shared state need their own locks. (`RNS/Transport.py:1995-2016`.) +3. **The handler is called in a fresh daemon thread.** A slow handler doesn't block subsequent inbound packets, but it also can't assume serialised execution — two announces from the same destination arriving back-to-back will run two handler threads concurrently. Handlers that mutate shared state need their own locks. (`RNS/Transport.py:1995-2025`.) LXMF registers two handlers at `LXMF/LXMRouter.py:207-208`: @@ -198,7 +198,7 @@ so any client built on the LXMF router gets contacts/propagation-node discovery If `RNS.Reticulum.transport_enabled() == True`, the same `Transport.inbound` dispatch then walks an additional code path that constructs a rebroadcast announce, queues it on each suitable interface (`announce_queue`), and emits it later subject to the per-interface `announce_cap` (default 2% of airtime). This entire path is skipped on a leaf client. -The rebroadcast logic is the bulk of `RNS/Transport.py:1810-1969` and is documented separately in `forward-announce.md` (TODO). +The rebroadcast logic is the bulk of `RNS/Transport.py:1810-1975` and is documented separately in `forward-announce.md` (TODO). --- diff --git a/flows/receive-link-lxmf.md b/flows/receive-link-lxmf.md index a47c87b..985aea7 100644 --- a/flows/receive-link-lxmf.md +++ b/flows/receive-link-lxmf.md @@ -1,6 +1,6 @@ # Flow: receive an LXMF message over a Reticulum Link -The inverse of [`send-link-lxmf.md`](send-link-lxmf.md), covering both halves of the responder side: accepting the inbound LINKREQUEST, sending the LRPROOF, then handling LXMF DATA on the established link. Pinned against **RNS 1.2.0**; cross-references [`../SPEC.md`](../SPEC.md) §6 (Link), §6.5 (PROOF), §6.6 (signalling), §6.7 (KEEPALIVE/teardown), §10 (Resource). +The inverse of [`send-link-lxmf.md`](send-link-lxmf.md), covering both halves of the responder side: accepting the inbound LINKREQUEST, sending the LRPROOF, then handling LXMF DATA on the established link. Pinned against **RNS 1.2.4 / LXMF 0.9.7**; cross-references [`../SPEC.md`](../SPEC.md) §6 (Link), §6.5 (PROOF), §6.6 (signalling), §6.7 (KEEPALIVE/teardown), §10 (Resource). --- @@ -14,7 +14,7 @@ A Reticulum DATA packet with `packet_type = LINKREQUEST (2)`, addressed to the r initiator_X25519_pub(32) || initiator_Ed25519_pub(32) || [signalling(3)] ``` -`Transport.inbound` (`RNS/Transport.py:2027-2057`) recognizes `packet_type == LINKREQUEST + destination_type == SINGLE`, looks up the destination in `destinations_map`, and calls `Destination.receive(packet)` which routes to `Destination.incoming_link_request(data, packet)` per `RNS/Destination.py:403-406`: +`Transport.inbound` (`RNS/Transport.py:2030-2060`) recognizes `packet_type == LINKREQUEST + destination_type == SINGLE`, looks up the destination in `destinations_map`, and calls `Destination.receive(packet)` which routes to `Destination.incoming_link_request(data, packet)` per `RNS/Destination.py:403-450` (`receive` at 403, dispatches to `incoming_link_request` at 420): ```python def receive(self, packet): @@ -24,7 +24,7 @@ def receive(self, packet): ### 2. Responder builds Link state via `Link.validate_request` -`RNS/Link.py:186-227`. Length-checks the body (`ECPUBSIZE` or `ECPUBSIZE + LINK_MTU_SIZE`), rejects otherwise. On success: +`RNS/Link.py:186-230`. Length-checks the body (`ECPUBSIZE` or `ECPUBSIZE + LINK_MTU_SIZE`), rejects otherwise. On success: 1. Build a `Link` object with `peer_pub_bytes = data[:32]` and `peer_sig_pub_bytes = data[32:64]`. 2. `set_link_id(packet)` per §6.3 — the link_id derives from `Packet.get_hashable_part`, invariant under HEADER_1↔HEADER_2 conversion. @@ -47,7 +47,7 @@ where `signature = sign(link_id || responder_X25519_pub || responder_long_term_E After the initiator validates the LRPROOF and sends `Link.LRRTT (0xFE)` carrying its measured RTT, the responder receives it at `Link.receive` line 1056-1059 and calls `rtt_packet`. The responder's RTT cache updates and `Link.STATUS = ACTIVE` triggers the `link_established_callback` registered via `Destination.set_link_established_callback`. -For LXMF, that callback is `LXMRouter.delivery_link_established` (`LXMF/LXMRouter.py:1849-1856`): +For LXMF, that callback is `LXMRouter.delivery_link_established` (`LXMF/LXMRouter.py:1852-1858`): ```python link.track_phy_stats(True) @@ -63,7 +63,7 @@ This is what makes inbound DATA on this link route into LXMF processing. ### 5. Inbound LXMF DATA — single-packet (PACKET representation) -A regular DATA packet on the link (`context = NONE`, Token-encrypted with link session key per §3.1). `Link.receive` decrypts and passes the plaintext to `delivery_packet(data, packet)` (`LXMF/LXMRouter.py:1819-1847`) — the same handler used by opportunistic delivery. Differences: +A regular DATA packet on the link (`context = NONE`, Token-encrypted with link session key per §3.1). `Link.receive` decrypts and passes the plaintext to `delivery_packet(data, packet)` (`LXMF/LXMRouter.py:1822-1850`) — the same handler used by opportunistic delivery. Differences: - `packet.destination_type == LINK` so `method = DIRECT`. - The LXMF body arrives **with** the recipient's dest_hash (§5.2), so no re-prepend like the opportunistic path does at step 9 of `receive-opportunistic-lxmf.md`. @@ -72,7 +72,7 @@ The handler calls `packet.prove()` immediately (mandatory PROOF receipt per §6. ### 6. Inbound LXMF DATA — Resource representation -A larger LXMF body arrives as a Resource transfer per `flows/receive-resource.md`. The Link's `resource_strategy = ACCEPT_APP` triggers `delivery_resource_advertised(resource)` (`LXMF/LXMRouter.py:1864-1871`): +A larger LXMF body arrives as a Resource transfer per `flows/receive-resource.md`. The Link's `resource_strategy = ACCEPT_APP` triggers `delivery_resource_advertised(resource)` (`LXMF/LXMRouter.py:1867-1874`): ```python def delivery_resource_advertised(self, resource): diff --git a/flows/receive-opportunistic-lxmf.md b/flows/receive-opportunistic-lxmf.md index aa8dcb4..2036d7c 100644 --- a/flows/receive-opportunistic-lxmf.md +++ b/flows/receive-opportunistic-lxmf.md @@ -2,7 +2,7 @@ The inverse of [`send-opportunistic-lxmf.md`](send-opportunistic-lxmf.md). What happens chronologically on the recipient when wire bytes for an opportunistic LXMF DATA packet arrive at one of its interfaces. -Pinned against **RNS 1.2.0 / LXMF 0.9.6**. Line numbers below are from those versions. +Pinned against **RNS 1.2.4 / LXMF 0.9.7**. Line numbers below are from those versions. Out of scope: receiving a packet over an established Reticulum Link (DIRECT method), receiving propagated messages from a propagation node, and receiving an announce / path-request / link-request. Each gets its own flow document. @@ -10,7 +10,7 @@ Out of scope: receiving a packet over an established Reticulum Link (DIRECT meth ## Preconditions -- Recipient has an `RNS.Identity` with the X25519 + Ed25519 private keys, plus a `lxmf.delivery` `RNS.Destination` registered with `LXMRouter.register_delivery_identity` — that registration calls `delivery_destination.set_packet_callback(self.delivery_packet)` at `LXMF/LXMRouter.py:341`, which is the hand-off point in step 7 below. +- Recipient has an `RNS.Identity` with the X25519 + Ed25519 private keys, plus a `lxmf.delivery` `RNS.Destination` registered with `LXMRouter.register_delivery_identity` — that registration calls `delivery_destination.set_packet_callback(self.delivery_packet)` at `LXMF/LXMRouter.py:340`, which is the hand-off point in step 7 below. - Recipient has, at some point, been the target of one or more announces from the sender, so `RNS.Identity.known_destinations` knows the sender's full `public_key` (X25519 || Ed25519) under their `dest_hash`. Without this, signature validation in step 11 will fail with `unverified_reason = SOURCE_UNKNOWN`. --- @@ -25,17 +25,17 @@ For RNode KISS specifically, `CMD_STAT_RSSI = 0x23` and `CMD_STAT_SNR = 0x24` si ### 2. `Transport.inbound(raw, interface)` entry point -`RNS/Transport.py:1327`. The single entry point for any inbound packet on any interface. The function is gated by `Transport.ready` — packets arriving before transport startup are dropped with a warning. +`RNS/Transport.py:1330`. The single entry point for any inbound packet on any interface. The function is gated by `Transport.ready` — packets arriving before transport startup are dropped with a warning. ### 3. IFAC unmask (Interface Authentication Codes) -`RNS/Transport.py:1338-1387`. If the interface has an `ifac_identity` configured, the high bit of `raw[0]` must be set; the IFAC bytes at `raw[2:2+ifac_size]` are then used to derive an HKDF mask, the rest of the packet is unmasked in place, and the IFAC is verified against `ifac_identity.sign(unmasked_raw)[-ifac_size:]`. Mismatch drops the packet silently. +`RNS/Transport.py:1338-1390`. If the interface has an `ifac_identity` configured, the high bit of `raw[0]` must be set; the IFAC bytes at `raw[2:2+ifac_size]` are then used to derive an HKDF mask, the rest of the packet is unmasked in place, and the IFAC is verified against `ifac_identity.sign(unmasked_raw)[-ifac_size:]`. Mismatch drops the packet silently. If the interface has no IFAC and the high bit IS set, the packet is dropped (an unexpected IFAC). ### 4. Packet parse and physical-layer stats -`RNS/Transport.py:1391-1395`: +`RNS/Transport.py:1394-1398`: ```python packet = RNS.Packet(None, raw) @@ -46,20 +46,20 @@ packet.hops += 1 `packet.unpack` reads the header byte fields per SPEC.md §2.1, sets `packet.header_type`, `packet.packet_type`, `packet.destination_type`, `packet.destination_hash`, `packet.context`, and slices `packet.data` from the remainder. Importantly, **hops is incremented by 1 here**, so even on a leaf-endpoint receive the local `packet.hops` is one more than what flew on the wire — flow logic that treats `packet.hops == 0` as "originator on this interface" must use the wire byte before this increment. -RSSI / SNR / Q link-quality stats are attached to `packet` if the interface exposed them (`RNS/Transport.py:1397-1417`). +RSSI / SNR / Q link-quality stats are attached to `packet` if the interface exposed them (`RNS/Transport.py:1400-1420`). ### 5. Hop fix-up for shared-instance and local-client interfaces -`RNS/Transport.py:1419-1422`. If the receiving interface is to a local shared instance or a local-client TCP socket, the +1 increment from step 4 is undone — the shared-instance path doesn't count as a real network hop. +`RNS/Transport.py:1422-1425`. If the receiving interface is to a local shared instance or a local-client TCP socket, the +1 increment from step 4 is undone — the shared-instance path doesn't count as a real network hop. ### 6. Dedup, then dispatch by packet_type / destination_type -`RNS/Transport.py:1424-1444`. `Transport.packet_filter` checks `packet.packet_hash` against `Transport.packet_hashlist` to drop replays. Hashes are added to the dedup list except for two cases that must be deferred: +`RNS/Transport.py:1427-1447`. `Transport.packet_filter` checks `packet.packet_hash` against `Transport.packet_hashlist` to drop replays. Hashes are added to the dedup list except for two cases that must be deferred: - packet whose `destination_hash` is in `Transport.link_table` — the dedup decision is left to the link itself, - LRPROOF packets — these may legitimately arrive on multiple interfaces during routing-fork chaos and the dedup list is updated only after the LRPROOF is validated. -Then the function fans out by `(packet_type, destination_type)`. For an opportunistic LXMF DATA packet — `packet_type == DATA`, `destination_type == SINGLE` — control reaches `RNS/Transport.py:2087-2103`: +Then the function fans out by `(packet_type, destination_type)`. For an opportunistic LXMF DATA packet — `packet_type == DATA`, `destination_type == SINGLE` — control reaches `RNS/Transport.py:2090-2106`: ```python destination = Transport.destinations_map.get(packet.destination_hash) @@ -78,7 +78,7 @@ If `destination_hash` does not match any locally-registered destination, this br ### 7. `Destination.receive(packet)` — decrypt and run packet callback -`RNS/Destination.py:403-418`: +`RNS/Destination.py:403-450`: ```python def receive(self, packet): @@ -93,11 +93,11 @@ def receive(self, packet): return True ``` -For `lxmf.delivery` destinations, `self.callbacks.packet` was set at `LXMF/LXMRouter.py:341` to the router's `delivery_packet` — see step 9. +For `lxmf.delivery` destinations, `self.callbacks.packet` was set at `LXMF/LXMRouter.py:340` to the router's `delivery_packet` — see step 9. ### 8. `Destination.decrypt` → `Identity.decrypt` — Token decode with ratchet ring -`RNS/Destination.py:611-643` → `RNS/Identity.py:818-872`. The packet body is the Token form from SPEC.md §3.1: `ephemeral_pub(32) || iv(16) || aes_ciphertext || hmac_sha256(32)`. +`RNS/Destination.py:611-645` → `RNS/Identity.py:849-905`. The packet body is the Token form from SPEC.md §3.1: `ephemeral_pub(32) || iv(16) || aes_ciphertext || hmac_sha256(32)`. ```python peer_pub_bytes = ciphertext_token[:32] # sender's ephemeral X25519 pub @@ -123,7 +123,7 @@ If the destination has ratchets enabled but on-disk state has gone stale, `Desti ### 9. `LXMRouter.delivery_packet(data, packet)` — proof first, then async parse -`LXMF/LXMRouter.py:1819-1847`: +`LXMF/LXMRouter.py:1822-1850`: ```python def delivery_packet(self, data, packet): @@ -144,7 +144,7 @@ The opportunistic-form re-prepends `packet.destination.hash` because step 6 of t ### 10. `LXMessage.unpack_from_bytes(lxmf_data)` — body parse and signature validation -`LXMF/LXMessage.py:736-807`. Field slicing: +`LXMF/LXMessage.py:736-810`. Field slicing: ``` destination_hash = lxmf_data[ 0:16] @@ -163,11 +163,11 @@ signed_part = hashed_part || message_hash Signature validation calls `source.identity.validate(signature, signed_part)`. The sender's identity is recalled from `RNS.Identity.known_destinations` via `RNS.Identity.recall(source_hash)` at line 765 — keyed by the sender's **destination hash**, not their identity hash, per SPEC.md §5.4 and §9.1. If the sender is unknown locally (no announce ever received), `unverified_reason = SOURCE_UNKNOWN` and `signature_validated = False`; the message is still surfaced to the app callback in step 12, but downstream UI should mark it untrusted. -Note: `unpack_from_bytes` does the stamp-strip-and-reencode variant but does **not** also try the as-received `packed_payload` if validation fails. SPEC.md §5.6 documents both raw and stripped-reencoded as valid receiver behavior; upstream LXMF 0.9.6 only does the stripped form (or the raw form when no stamp was present). The spec's stronger receiver tolerance is a recommendation for non-upstream implementers, not a description of upstream. +Note: `unpack_from_bytes` does the stamp-strip-and-reencode variant but does **not** also try the as-received `packed_payload` if validation fails. SPEC.md §5.6 documents both raw and stripped-reencoded as valid receiver behavior; upstream LXMF 0.9.7 only does the stripped form (or the raw form when no stamp was present). The spec's stronger receiver tolerance is a recommendation for non-upstream implementers, not a description of upstream. ### 11. Stamp / ticket / dedup checks -`LXMF/LXMRouter.py:1741-1803`: +`LXMF/LXMRouter.py:1741-1810`: - **Ticket** (`message.fields[FIELD_TICKET]`): if present and non-expired, cached for outbound use. - **Stamp**: if the local destination has a `stamp_cost`, `message.validate_stamp(required_cost, tickets=...)` runs; a missing/invalid stamp causes the message to be dropped when `_enforce_stamps` is true. @@ -177,7 +177,7 @@ Note: `unpack_from_bytes` does the stamp-strip-and-reencode variant but does **n ### 12. `__delivery_callback` fires — message reaches the app -`LXMF/LXMRouter.py:1805-1812`. The router's caller (Sideband, NomadNet, MeshChat, …) sets `__delivery_callback` via `register_delivery_callback(...)`. It receives the validated `LXMessage` object and decides what to show in the inbox. +`LXMF/LXMRouter.py:1812-1820`. The router's caller (Sideband, NomadNet, MeshChat, …) sets `__delivery_callback` via `register_delivery_callback(...)`. It receives the validated `LXMessage` object and decides what to show in the inbox. Important: the recipient's app should apply the SPEC.md §9.6 clockless-sender heuristic at this point — `if message.timestamp < 1577836800: message.timestamp = local_now()` — to keep clockless LoRa devices out of January 1970 in the inbox. Upstream `LXMessage.unpack_from_bytes` does **not** do this fix-up. diff --git a/flows/receive-propagated-lxmf.md b/flows/receive-propagated-lxmf.md index 56d56db..bd6cfa8 100644 --- a/flows/receive-propagated-lxmf.md +++ b/flows/receive-propagated-lxmf.md @@ -1,6 +1,6 @@ # Flow: receive a propagated LXMF message (recipient pulls via `/get`) -The closing half of [`send-propagated-lxmf.md`](send-propagated-lxmf.md): how a recipient client retrieves messages that were store-and-forwarded for it by a propagation node. Pinned against **RNS 1.2.0 / LXMF 0.9.6**; cross-references [`../SPEC.md`](../SPEC.md) §5.8 (propagation protocol), §11 (REQUEST/RESPONSE). +The closing half of [`send-propagated-lxmf.md`](send-propagated-lxmf.md): how a recipient client retrieves messages that were store-and-forwarded for it by a propagation node. Pinned against **RNS 1.2.4 / LXMF 0.9.7**; cross-references [`../SPEC.md`](../SPEC.md) §5.8 (propagation protocol), §11 (REQUEST/RESPONSE). This is the inverse-side flow that turns "the message was queued at a propagation node" (`send-propagated-lxmf.md` step 9) into "the message arrives in the recipient's inbox". @@ -17,7 +17,7 @@ This is the inverse-side flow that turns "the message was queued at a propagatio ### 1. Recipient initiates retrieval -`LXMRouter.request_messages_from_propagation_node(identity, max_messages)` (`LXMF/LXMRouter.py:485+`). Triggered by: +`LXMRouter.request_messages_from_propagation_node(identity, max_messages)` (`LXMF/LXMRouter.py:484+`). Triggered by: - Manual user action (Sideband "Refresh inbox" button). - Periodic background poll (every few minutes by default in long-running clients). @@ -34,7 +34,7 @@ self.outbound_propagation_link = RNS.Link( ) ``` -(`LXMF/LXMRouter.py:514`). Standard Link establishment per `flows/send-link-lxmf.md` steps 3-4. +(`LXMF/LXMRouter.py:513`). Standard Link establishment per `flows/send-link-lxmf.md` steps 3-4. ### 3. Identify on the link @@ -47,7 +47,7 @@ data = [None, None] # [wanted, have link.request("/get", data, response_callback=on_message_list) ``` -The propagation node's `message_get_request` handler at `LXMF/LXMRouter.py:1427-1450` walks `propagation_entries` for messages keyed to the requester's destination_hash and returns: +The propagation node's `message_get_request` handler at `LXMF/LXMRouter.py:1426-1450` walks `propagation_entries` for messages keyed to the requester's destination_hash and returns: ```python [ [transient_id_1(16), size_1(int)], @@ -86,7 +86,7 @@ Returns this as a §11 RESPONSE. If the bundle fits in `link.mdu` it's a single ### 7. Recipient unpacks the bundle and processes each message -The recipient's `propagation_resource_concluded` handler (or its single-packet equivalent) at `LXMF/LXMRouter.py:2194+` walks the bundle: +The recipient's `propagation_resource_concluded` handler (or its single-packet equivalent) at `LXMF/LXMRouter.py:2200+` walks the bundle: ```python data = msgpack.unpackb(resource.data.read()) diff --git a/flows/receive-resource.md b/flows/receive-resource.md index ccfcb0b..18681da 100644 --- a/flows/receive-resource.md +++ b/flows/receive-resource.md @@ -1,6 +1,6 @@ # Flow: receive a Resource (large body) over a Link -The inverse of [`send-resource.md`](send-resource.md). What happens chronologically on the receiver when an inbound Resource transfer arrives. Pinned against **RNS 1.2.0**; see [`../SPEC.md`](../SPEC.md) §10 for the wire bytes. +The inverse of [`send-resource.md`](send-resource.md). What happens chronologically on the receiver when an inbound Resource transfer arrives. Pinned against **RNS 1.2.4**; see [`../SPEC.md`](../SPEC.md) §10 for the wire bytes. --- @@ -29,7 +29,7 @@ Branch by `resource_strategy`: ### 3. Receiver issues the first RESOURCE_REQ -`Resource.request_next()` (`RNS/Resource.py:934-983`) builds the request body per §10.5: +`Resource.request_next()` (`RNS/Resource.py:931-981`) builds the request body per §10.5: ``` exhausted_flag(1) [|| last_map_hash(4)] || resource_hash(32) || requested_map_hashes(N × 4) diff --git a/flows/send-announce.md b/flows/send-announce.md index afb8a9d..ba73241 100644 --- a/flows/send-announce.md +++ b/flows/send-announce.md @@ -1,6 +1,6 @@ # Flow: send an announce -What happens chronologically when a node emits an announce — the periodic broadcast that lets the rest of the mesh discover or refresh a path to this destination. Pinned against **RNS 1.2.0**. +What happens chronologically when a node emits an announce — the periodic broadcast that lets the rest of the mesh discover or refresh a path to this destination. Pinned against **RNS 1.2.4**. Out of scope: the relay-side rebroadcast (`forward-announce.md` — see [`../SPEC.md`](../SPEC.md) §12.3) and path-response announces (already covered in [`path-discovery.md`](path-discovery.md)). @@ -85,11 +85,11 @@ Wire form per §4.1: - `context = NONE (0x00)` for periodic re-announces, `PATH_RESPONSE (0x0B)` for path-response announces - `context_flag = 1` if ratchet present (signals the optional ratchet_pub slot in the body) -Announce packets are NOT encrypted — `Packet.pack` (`RNS/Packet.py:189-191`) special-cases ANNOUNCE to skip encryption. The body is signed but plaintext, so anyone in earshot can validate the signature and decode the public key. +Announce packets are NOT encrypted — `Packet.pack` (`RNS/Packet.py:188-191`) special-cases ANNOUNCE to skip encryption. The body is signed but plaintext, so anyone in earshot can validate the signature and decode the public key. ### 8. `Transport.outbound` broadcasts on every OUT interface -Same broadcast branch as a path? request (`flows/path-discovery.md` step 2) — the dest_hash isn't in `path_table` (it's our own destination, not a remote one), so the broadcast branch at `RNS/Transport.py:1119+` fires, emitting on every interface where `interface.OUT == True`. Per §7.5 the announce is rate-limited by `ANNOUNCE_CAP = 2.0` (2% airtime) on each interface. +Same broadcast branch as a path? request (`flows/path-discovery.md` step 2) — the dest_hash isn't in `path_table` (it's our own destination, not a remote one), so the broadcast branch at `RNS/Transport.py:1122+` fires, emitting on every interface where `interface.OUT == True`. Per §7.5 the announce is rate-limited by `ANNOUNCE_CAP = 2.0` (2% airtime) on each interface. ### 9. Periodic re-announce loop diff --git a/flows/send-link-lxmf.md b/flows/send-link-lxmf.md index 2de6a27..9ad7e60 100644 --- a/flows/send-link-lxmf.md +++ b/flows/send-link-lxmf.md @@ -2,7 +2,7 @@ What happens chronologically when an app calls `LXMRouter.handle_outbound(lxm)` for an `LXMessage` whose `desired_method == DIRECT` (or whose payload exceeds the opportunistic single-packet content limit and is downgraded from `OPPORTUNISTIC` to `DIRECT` at pack time). The `DIRECT` method runs the LXMF body over an established Reticulum Link rather than a single Reticulum DATA packet. -Pinned against **RNS 1.2.0 / LXMF 0.9.6**. Line numbers below are from those versions. +Pinned against **RNS 1.2.4 / LXMF 0.9.7**. Line numbers below are from those versions. Out of scope: opportunistic delivery (see [`send-opportunistic-lxmf.md`](send-opportunistic-lxmf.md)), propagation-node delivery (`PROPAGATED`), and paper messages (`PAPER`). @@ -30,7 +30,7 @@ Same as steps 1-2 of `send-opportunistic-lxmf.md`. `LXMessage.pack()` builds the ### 2. `process_outbound` enters the DIRECT branch -`LXMF/LXMRouter.py:2599`. Logic for an `LXMessage` whose `method == DIRECT`: +`LXMF/LXMRouter.py:2531-2545`. Logic for an `LXMessage` whose `method == DIRECT`: ```python delivery_destination_hash = lxmessage.get_destination().hash @@ -63,13 +63,13 @@ Setting `process_outbound` itself as the link's established-callback is the tric ### 3. `RNS.Link(destination)` builds and sends a LINKREQUEST -`RNS/Link.py:233-327`. The `Link` constructor on the **initiator** side generates a fresh ephemeral X25519 + Ed25519 keypair (`pub_bytes` and `sig_pub_bytes`) and constructs the LINKREQUEST body (`RNS/Link.py:308-324`): +`RNS/Link.py:233-330`. The `Link` constructor on the **initiator** side generates a fresh ephemeral X25519 + Ed25519 keypair (`pub_bytes` and `sig_pub_bytes`) and constructs the LINKREQUEST body (`RNS/Link.py:308-324`): ``` request_data = initiator_X25519_pub(32) || initiator_Ed25519_pub(32) || [signalling(3)] ``` -Both initiator-side keys are **fresh-ephemeral**, used only for this link. The optional 3-byte `signalling` field, present iff `Reticulum.link_mtu_discovery()` returns true and the next-hop interface advertises an HW MTU, encodes the path-MTU and link-mode hints (SPEC.md §6.1; encode/decode helpers at `RNS/Link.py:148`). +Both initiator-side keys are **fresh-ephemeral**, used only for this link. The optional 3-byte `signalling` field, present iff `Reticulum.link_mtu_discovery()` returns true and the next-hop interface advertises an HW MTU, encodes the path-MTU and link-mode hints (SPEC.md §6.1; encode/decode helpers at `RNS/Link.py:148-152`). ```python self.packet = RNS.Packet(destination, self.request_data, packet_type=RNS.Packet.LINKREQUEST) @@ -88,12 +88,12 @@ The LINKREQUEST then goes through the same `Transport.outbound` path as any othe ### 4. Wait for LRPROOF; verify; complete handshake -The initiator's link sits in `Link.PENDING` while it waits for the responder's LRPROOF. The responder's side of this exchange (entering `Destination.receive` with `packet_type == LINKREQUEST` → `incoming_link_request` → `Link.validate_request` → `Link.handshake` → `Link.prove`, all at `RNS/Link.py:186-227, 353-381`) is its own flow document; this flow describes only what the initiator sees. +The initiator's link sits in `Link.PENDING` while it waits for the responder's LRPROOF. The responder's side of this exchange (entering `Destination.receive` with `packet_type == LINKREQUEST` → `incoming_link_request` → `Link.validate_request` → `Link.handshake` → `Link.prove`, all at `RNS/Link.py:186-230, 353-381`) is its own flow document; this flow describes only what the initiator sees. The LRPROOF arrives back at `Transport.inbound` with `packet_type = PROOF`, `context = LRPROOF (0xff)`, `dest_hash = link_id`. `Transport.inbound` looks up the link by its `link_id` and dispatches to `Link.validate_proof` (`RNS/Link.py:396`). For an initiator with link still in `PENDING`: ``` -proof_data layout (RNS/Link.py:376): +proof_data layout (RNS/Link.py:371): signature(64) || responder_X25519_pub(32) || [signalling(3)] ``` @@ -103,21 +103,21 @@ Validation (`RNS/Link.py:410-422`): 2. Read the responder's long-term Ed25519 public key from the `Destination.identity` we already had cached (from a prior announce — line 412: `self.destination.identity.get_public_key()[ECPUBSIZE//2:ECPUBSIZE]`). The Ed25519 pub is **not** sent in the LRPROOF body; it must be known locally already. 3. Derive `signed_data = link_id || responder_X25519_pub || responder_long_term_Ed25519_pub || [signalling]` (line 417). 4. Verify the Ed25519 signature against `signed_data` using the responder's long-term Ed25519 pub. -5. On success: `link.handshake()` runs (`RNS/Link.py:353-368`) — ECDH with the responder's fresh X25519 pub, then HKDF over the shared secret with `salt = link_id`, derives `signing_key(32) || encrypt_key(32)` per SPEC.md §6.4. Link state transitions to `Link.ACTIVE`. +5. On success: `link.handshake()` runs (`RNS/Link.py:353-370`) — ECDH with the responder's fresh X25519 pub, then HKDF over the shared secret with `salt = link_id`, derives `signing_key(32) || encrypt_key(32)` per SPEC.md §6.4. Link state transitions to `Link.ACTIVE`. 6. The initiator's `established_callback` registered at step 2 — `LXMRouter.process_outbound` — fires, re-entering step 2 with the link now ACTIVE. A confirmed-MTU hint in the LRPROOF (length is `64+32+3 = 99` instead of `64+32 = 96`) updates `link.mtu` to the smaller of the responder's hint and the initiator's view (`RNS/Link.py:404-408`). An RTT report message follows automatically (`RNS/Link.py:441`, packet with `context = LRRTT`). ### 5. Transfer the LXM body over the active link -Back in `process_outbound` (`LXMF/LXMRouter.py:2618-2625`), with `direct_link.status == ACTIVE`: +Back in `process_outbound` (`LXMF/LXMRouter.py:2620-2630`), with `direct_link.status == ACTIVE`: ```python lxmessage.set_delivery_destination(direct_link) # __delivery_destination = link lxmessage.send() # LXMessage.send branches on representation ``` -`LXMessage.send` for DIRECT/PACKET (`LXMF/LXMessage.py:471-484`) calls `self.__as_packet()` which constructs: +`LXMessage.send` for DIRECT/PACKET (`LXMF/LXMessage.py:471-490`) calls `self.__as_packet()` which constructs: ```python RNS.Packet(self.__delivery_destination, self.packed) # full LXMF body, dest_hash included @@ -161,10 +161,10 @@ When the resource concludes successfully, the same `__mark_delivered` path runs ### 9. Backchannel identification (optional, after first successful DIRECT delivery) -`LXMF/LXMRouter.py:2532-2545`. After a DIRECT delivery completes (`lxmessage.state == DELIVERED`), if the link doesn't yet have a backchannel identity associated, the initiator's `LXMRouter` calls `direct_link.identify(backchannel_identity)`: +`LXMF/LXMRouter.py:2531-2545`. After a DIRECT delivery completes (`lxmessage.state == DELIVERED`), if the link doesn't yet have a backchannel identity associated, the initiator's `LXMRouter` calls `direct_link.identify(backchannel_identity)`: - `direct_link.identify` sends an IDENTIFY packet on the link bound to one of the **sender's** local `lxmf.delivery` destinations. -- The receiving side's `delivery_link_established` callback (set at `LXMRouter.py:1849-1856`) installs `delivery_packet` on the link's packet callback, so subsequent inbound DATA on this link reaches the LXMF parser. +- The receiving side's `delivery_link_established` callback (set at `LXMRouter.py:1852-1858`) installs `delivery_packet` on the link's packet callback, so subsequent inbound DATA on this link reaches the LXMF parser. - The link is now usable in **both directions** without each side having to open its own Link to the other. The receiver can now reply over this same link. This enables an interactive conversation over a single link rather than each message opening a new link. diff --git a/flows/send-opportunistic-lxmf.md b/flows/send-opportunistic-lxmf.md index 85a611e..fc04d0f 100644 --- a/flows/send-opportunistic-lxmf.md +++ b/flows/send-opportunistic-lxmf.md @@ -2,7 +2,7 @@ What happens chronologically when an app calls `LXMRouter.handle_outbound(lxm)` for an `LXMessage` whose `desired_method == OPPORTUNISTIC` and whose payload fits in a single Reticulum packet. -Pinned against **RNS 1.2.0 / LXMF 0.9.6**. Line numbers below are from those versions. +Pinned against **RNS 1.2.4 / LXMF 0.9.7**. Line numbers below are from those versions. Out of scope: messages that need a Reticulum Link (`DIRECT` method, larger payloads), propagation-node delivery (`PROPAGATED`), and paper messages (`PAPER`). Each gets its own flow document. @@ -66,7 +66,7 @@ if not RNS.Transport.has_path(destination_hash) and lxmessage.method == LXMessag lxmessage.next_delivery_attempt = time.time() + LXMRouter.PATH_REQUEST_WAIT ``` -Only fires when **no entry exists** in `Transport.path_table`. Stale-but-present entries are not refreshed by this preamble — they are evicted by the periodic `stale_paths` accumulator in `RNS/Transport.py:747+`, after which the next outbound attempt rediscovers the unknown-path branch and triggers `request_path`. The path-request packet itself (well-known dest hash `6b9f66014d9853faab220fba47d02761`, payload `target_dest_hash || [transport_id ||] tag`) is described in SPEC.md §7.1. +Only fires when **no entry exists** in `Transport.path_table`. Stale-but-present entries are not refreshed by this preamble — they are evicted by the periodic `stale_paths` accumulator in `RNS/Transport.py:750+`, after which the next outbound attempt rediscovers the unknown-path branch and triggers `request_path`. The path-request packet itself (well-known dest hash `6b9f66014d9853faab220fba47d02761`, payload `target_dest_hash || [transport_id ||] tag`) is described in SPEC.md §7.1. When this preamble fires the message is queued, not sent — control returns; sending resumes at step 5 after `PATH_REQUEST_WAIT` elapses or an announce response populates the path table. @@ -113,7 +113,7 @@ ephemeral_pub(32) || iv(16) || aes_ciphertext(...) || hmac_sha256(32) ### 8. `Transport.outbound(packet)` — path-table-aware framing -`RNS/Transport.py:1031-1113`. Verified by [`../tools/verify_packet_header.py`](../tools/verify_packet_header.py). +`RNS/Transport.py:1034-1116`. Verified by [`../tools/verify_packet_header.py`](../tools/verify_packet_header.py). - If `path_table[dest][HOPS] > 1`: convert HEADER_1 → HEADER_2 (SPEC.md §2.3). The originator inserts `path_table[dest][NEXT_HOP]` (16-byte transport_id) at offset 2 and flips the flag bits to `HEADER_2 | TRANSPORT | (orig_low_nibble)`. Resulting wire packet is 35 + ciphertext bytes. - If `path_table[dest][HOPS] == 1` AND the local node is connected to a shared instance: same conversion applies (lines 1094-1105). @@ -148,7 +148,7 @@ The receive flow is its own document; see `receive-opportunistic-lxmf.md` (TODO) ### 12. PROOF receipt returns -`RNS/Transport.py:1031-1054`. Because the packet is `DATA` for a non-PLAIN, non-LINK destination with `create_receipt == True`, `Transport.outbound` registered a `PacketReceipt` on the sender side. When the recipient calls `Packet.prove`, a PROOF packet flies back containing `SHA256(packet.hashable_part)`; the sender's `PacketReceipt` matches it, fires the delivery callback registered at step 5, and the LXMessage state advances `SENT → DELIVERED`. +`RNS/Transport.py:1034-1057`. Because the packet is `DATA` for a non-PLAIN, non-LINK destination with `create_receipt == True`, `Transport.outbound` registered a `PacketReceipt` on the sender side. When the recipient calls `Packet.prove`, a PROOF packet flies back containing `SHA256(packet.hashable_part)`; the sender's `PacketReceipt` matches it, fires the delivery callback registered at step 5, and the LXMessage state advances `SENT → DELIVERED`. If no proof arrives within the receipt timeout, `__link_packet_timed_out` runs on the receipt's timeout callback and the LXMessage state can drive a retry (see `LXMRouter.py::process_outbound` retry logic at `:2571+`, which may itself trigger a fresh `request_path` after `MAX_PATHLESS_TRIES`). diff --git a/flows/send-propagated-lxmf.md b/flows/send-propagated-lxmf.md index 64a5305..427814f 100644 --- a/flows/send-propagated-lxmf.md +++ b/flows/send-propagated-lxmf.md @@ -1,6 +1,6 @@ # Flow: send a PROPAGATED LXMF message via a propagation node -What happens when an LXMF client submits a message to a propagation node for store-and-forward delivery — the path used when the recipient is offline, intermittent, or simply somewhere the sender can't reach directly. Pinned against **RNS 1.2.0**; cross-references [`../SPEC.md`](../SPEC.md) §5.8 (propagation protocol), §6 (Link), §10 (Resource), §11 (REQUEST/RESPONSE). +What happens when an LXMF client submits a message to a propagation node for store-and-forward delivery — the path used when the recipient is offline, intermittent, or simply somewhere the sender can't reach directly. Pinned against **RNS 1.2.4 / LXMF 0.9.7**; cross-references [`../SPEC.md`](../SPEC.md) §5.8 (propagation protocol), §6 (Link), §10 (Resource), §11 (REQUEST/RESPONSE). --- @@ -15,7 +15,7 @@ What happens when an LXMF client submits a message to a propagation node for sto ### 1. App constructs an LXMessage with `desired_method = PROPAGATED` -Same as `send-opportunistic-lxmf.md` step 1 except the desired_method differs. The router's `handle_outbound` (`LXMF/LXMRouter.py:1639+`) will eventually route this to the propagation pipeline. +Same as `send-opportunistic-lxmf.md` step 1 except the desired_method differs. The router's `handle_outbound` (`LXMF/LXMRouter.py:1644+`) will eventually route this to the propagation pipeline. ### 2. `LXMessage.pack()` — propagation-specific encryption @@ -30,7 +30,7 @@ Same as `send-opportunistic-lxmf.md` step 1 except the desired_method differs. T ### 3. `LXMRouter.process_outbound` for PROPAGATED method -`LXMF/LXMRouter.py:2547-...` (the `PROPAGATED` branch is structurally similar to the `DIRECT` branch in `send-link-lxmf.md`). High-level state: +`LXMF/LXMRouter.py:2544+` (the `PROPAGATED` branch is structurally similar to the `DIRECT` branch in `send-link-lxmf.md`). High-level state: - If a Link to the propagation node already exists and is `ACTIVE`: reuse it. - Else if the path is known: open a fresh `RNS.Link(propagation_node_destination)` with `LXMRouter.process_outbound` registered as the `link_established_callback` so the LXM is sent as soon as the link establishes. diff --git a/flows/send-resource.md b/flows/send-resource.md index 41ab2da..dc53e0c 100644 --- a/flows/send-resource.md +++ b/flows/send-resource.md @@ -2,7 +2,7 @@ What happens chronologically when an LXMF DIRECT message, NomadNet page, or `rncp` file transfer too big to fit in one Link DATA packet is sent as an `RNS.Resource`. Builds on top of an established Reticulum Link — a Link must already be `ACTIVE` before this flow starts (see [`send-link-lxmf.md`](send-link-lxmf.md) steps 3-4 for how the Link gets there). -Pinned against **RNS 1.2.0**. Wire-level details are in [`../SPEC.md`](../SPEC.md) §10; this document covers chronology and step ordering. +Pinned against **RNS 1.2.4**. Wire-level details are in [`../SPEC.md`](../SPEC.md) §10; this document covers chronology and step ordering. Out of scope: the receive side (`receive-resource.md` — TODO), Resource cancellation paths beyond a brief mention in step 9, and the watchdog / RTT estimation machinery (implementation-private). @@ -125,7 +125,7 @@ body = resource_hash(32) || full_proof(32) as `RNS.Packet(link, proof_data, packet_type=PROOF, context=RESOURCE_PRF)` — a PROOF-type packet, not DATA. -The initiator's `validate_proof(proof_data)` (`RNS/Resource.py:785-829`): +The initiator's `validate_proof(proof_data)` (`RNS/Resource.py:782-826`): 1. Checks `len(proof_data) == 64` and `proof_data[32:] == self.expected_proof`. 2. On match, transitions `status = COMPLETE` and fires the resource callback. @@ -144,7 +144,7 @@ Both transition `status = FAILED` and notify `link.resource_concluded(self)` so ### 10. Watchdog and recovery -`RNS/Resource.py:564-642`. The Resource owns a watchdog thread that runs through the lifecycle and adjusts timeouts based on observed link RTT. Key points for interop: +`RNS/Resource.py:564-670`. The Resource owns a watchdog thread that runs through the lifecycle and adjusts timeouts based on observed link RTT. Key points for interop: - **Per-part timeout:** `PART_TIMEOUT_FACTOR = 4` × (link RTT) before any part has arrived; drops to `PART_TIMEOUT_FACTOR_AFTER_RTT = 2` once RTT is calibrated. - **Proof timeout:** `PROOF_TIMEOUT_FACTOR = 3` × link RTT after all parts have been sent. diff --git a/test-vectors/README.md b/test-vectors/README.md index 4fc01d3..97b1c1e 100644 --- a/test-vectors/README.md +++ b/test-vectors/README.md @@ -4,7 +4,7 @@ Known-good byte sequences that any Reticulum-compatible implementation should be ## Status -Populated against RNS 1.2.0 / LXMF 0.9.6: +Populated against RNS 1.2.4 / LXMF 0.9.7: - ✅ `identities.json` — Alice + Bob identity vectors (regenerator: `../tools/regen_identities.py`, verifier: `../tools/verify_destination_hash.py`). - ✅ `announces.json` — two announce vectors (no-ratchet + with-ratchet) signed by Alice (regenerator: `../tools/regen_announces.py`, verifier: `../tools/verify_announce_roundtrip.py`). diff --git a/todo.md b/todo.md index 3bd0330..177c60f 100644 --- a/todo.md +++ b/todo.md @@ -396,6 +396,42 @@ order: top three save the most debugging hours. implicit in §4.5 / §7.x / §10 / §12 today; a single appendix table would be a quick reference card. +## Upstream distribution shift + +RNS 1.2.4 (2026-05-07) is *"probably the last release that is also +published to GitHub, since everything can now run over Reticulum +itself."* Pip continues *"at least until `rnpkg` is complete, and RNS +is completely self-hosting."* Watch-items so the verifier doesn't +strand when GitHub / PyPI stop being authoritative: + +- [ ] **Stand up a local Reticulum node with internet reach.** Doesn't + need to be 24/7. Needed so `rngit` and (later) `rnpkg` can fetch + from upstream once the GitHub mirror is gone. Capture the node's + identity / config in a private spot (not this repo). +- [ ] **Capture the upstream Reticulum repo node's destination hash + once published** — markqvist will publish a `rngit` address + for the official source repo. When that lands, record it + somewhere durable (suggest `tools/sources.md`, new file) so + anyone bringing up the verifier knows where to fetch from + when GitHub goes dark. +- [ ] **Watch `rnpkg` for install/upgrade commands.** Currently a stub + in 1.2.4 (only `--config` / `--exampleconfig` / `--version` + flags). When `rnpkg install` / `rnpkg upgrade` ship, swap + `pip install -r tools/requirements.txt` instructions to the + `rnpkg` equivalent (or document both during the transition). +- [ ] **`rsg` signature verification.** RNS 1.2.4 introduced a new + `rsg` file signature format for release artifacts. Once releases + stop being GitHub-signed, we'll need to verify `rsg` signatures + on whatever we pull through `rnpkg`/`rngit` to know we got + authentic upstream code. Likely a small helper script in + `tools/`. +- [ ] **Mirror upstream source citations into `references/`.** SPEC.md + cites upstream Python by file + line throughout. Once upstream + moves off GitHub, those citations get harder to follow without a + checkout. Consider extracting the cited functions/lines into a + `references/` tree keyed by RNS version, so the spec stays + navigable even when upstream is Reticulum-only. + ## Spec polishing (lower priority) - [x] **Navigation polish for `SPEC.md`** — at ~3300 lines, splitting diff --git a/tools/README.md b/tools/README.md index 4eeb3c2..9729c23 100644 --- a/tools/README.md +++ b/tools/README.md @@ -19,7 +19,7 @@ The scripts read `RNS.__version__` at startup and print it in their output so a ## Status -Populated against RNS 1.2.0 / LXMF 0.9.6: +Populated against RNS 1.2.4 / LXMF 0.9.7: | Script | Verifies SPEC.md section | Status | |---|---|---| diff --git a/tools/requirements.txt b/tools/requirements.txt new file mode 100644 index 0000000..a6053ed --- /dev/null +++ b/tools/requirements.txt @@ -0,0 +1,15 @@ +# Pinned versions the verifier scripts have been run against. +# +# Update only after the corresponding verifier scripts have been +# re-run cleanly against the new upstream version. The README +# "Spec corrections" section exists to cover the case where the +# spec ships text that turns out to be wrong against a new pin. +# +# Why pin: upstream is migrating distribution off GitHub onto +# Reticulum-native channels (rngit / rnpkg). PyPI continues +# "at least until rnpkg is complete" per the RNS 1.2.4 release +# notes (2026-05-07). A pin keeps the verifier reproducible if +# upstream stops mirroring to PyPI before our migration is ready. + +rns==1.2.4 +lxmf==0.9.7 diff --git a/tools/verify_announce_app_data.py b/tools/verify_announce_app_data.py index 9a324ab..a59736c 100644 --- a/tools/verify_announce_app_data.py +++ b/tools/verify_announce_app_data.py @@ -4,8 +4,8 @@ destinations). Exercises: - Upstream LXMF.LXMRouter.get_announce_app_data emits a 2-element msgpack - array [display_name_bytes, stamp_cost] in LXMF 0.9.6. The dead-code - supported_functionality line at LXMF/LXMRouter.py:999 is computed but + array [display_name_bytes, stamp_cost] in LXMF 0.9.7. The dead-code + supported_functionality line at LXMF/LXMRouter.py:998 is computed but never appended. - The wire-byte form for display_name="Reticulum5", stamp_cost=None matches the hex documented in SPEC.md S4.3: @@ -57,7 +57,7 @@ def verify_two_element_wire_bytes(): def verify_producer_is_two_element_in_this_lxmf(): # Read the LXMRouter source and confirm it appends 2 elements (not 3) in - # the LXMF 0.9.6 producer. We do this by inspecting the function source so + # the LXMF 0.9.7 producer. We do this by inspecting the function source so # the verifier breaks loudly if upstream restores the 3-element variant. import inspect from LXMF.LXMRouter import LXMRouter @@ -72,7 +72,7 @@ def verify_producer_is_two_element_in_this_lxmf(): fail("S4.3 producer NOW appends supported_functionality. " "Update SPEC.md S4.3 to describe the 3-element variant as live.") print(f"PASS S4.3 LXMF {LXMF.__version__} producer emits 2-element form only " - "(supported_functionality is dead code at LXMF/LXMRouter.py:999)") + "(supported_functionality is dead code at LXMF/LXMRouter.py:998)") def verify_parser_tolerance(): diff --git a/tools/verify_packet_header.py b/tools/verify_packet_header.py index e54d574..9d908e7 100644 --- a/tools/verify_packet_header.py +++ b/tools/verify_packet_header.py @@ -12,7 +12,7 @@ Verifies: and HEADER_2 layout flags(1) hops(1) transport_id(16) dest_hash(16) context(1) data. - §2.3: originator HEADER_1 → HEADER_2 conversion when path_table reports - hops > 1. The conversion logic at RNS/Transport.py:1074-1083 is + hops > 1. The conversion logic at RNS/Transport.py:1077-1108 is exercised by stubbing Transport.transmit and seeding the path_table with a synthetic multi-hop entry. The wire bytes captured at transmit-time are compared to the expected HEADER_2 form. @@ -209,7 +209,7 @@ def verify_header_conversion(rns_instance): fail("§2.3 conversion: trailing bytes (orig dest_hash + ctx + payload) mismatch") print("PASS S2.3 HEADER_1 -> HEADER_2 conversion at originator " - "(matches RNS/Transport.py:1074-1083)") + "(matches RNS/Transport.py:1077-1108)") finally: Transport.transmit = real_transmit with Transport.path_table_lock: diff --git a/tools/verify_proof_packet.py b/tools/verify_proof_packet.py index 553cbba..6173434 100644 --- a/tools/verify_proof_packet.py +++ b/tools/verify_proof_packet.py @@ -1,11 +1,11 @@ """ Verifier for SPEC.md S6.5 (regular PROOF packet wire form). -Three scenarios against upstream RNS 1.2.0: +Three scenarios against upstream RNS 1.2.4: 1. Implicit-mode opportunistic DATA proof: when Reticulum is configured with use_implicit_proof = True (the upstream default - per Reticulum.py:259), Identity.prove emits a 64-byte body + per Reticulum.py:256), Identity.prove emits a 64-byte body containing only the Ed25519 signature over packet.packet_hash. 2. Explicit-mode opportunistic DATA proof: when use_implicit_proof diff --git a/tools/verify_token_crypto.py b/tools/verify_token_crypto.py index b8aa6f2..9a6e0ff 100644 --- a/tools/verify_token_crypto.py +++ b/tools/verify_token_crypto.py @@ -2,7 +2,7 @@ Verifier for SPEC.md S3 (Token cryptography). Exercises the modified-Fernet Token construction in two directions -against upstream RNS 1.2.0: +against upstream RNS 1.2.4: 1. Identity-style encrypt (with ephemeral X25519 prefix) per S3.1 opportunistic form. Round-trips a known plaintext through