diff --git a/README.md b/README.md index f3b51e3..8097036 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,9 @@ As content grows, `SPEC.md` will be split into per-layer files (packet header, i Errata that may invalidate code built against an earlier revision of `SPEC.md`. Newest first. Feature additions and ordinary edits live in `git log` — this section is reserved for cases where the spec said one thing, that turned out to be wrong, and an implementer who pulled the bad version needs to fix their code. +- **2026-06-08 — §12.5 ordinary DATA reverse routing and timeout.** + Earlier prose stated that reverse-table entries expire after 30 seconds and the opportunistic receive flow showed HEADER_2 arriving at a relayed endpoint. RNS 1.2.4 defines `REVERSE_TIMEOUT = 8*60`, and the final relay strips HEADER_2 before endpoint delivery. The audit also found that a wrong-interface PROOF consumes the one-shot reverse entry before being dropped. Corrected and runtime-locked by `tools/verify_transport_data.py`. + - **2026-06-08 — §12.5 transport-relayed Link forwarding gates and proof routing.** Earlier prose implied that established-Link forwarding begins only after LRPROOF validation, that relay lookup requires `destination_type == LINK`, and that Link proofs use `reverse_table`. Upstream RNS 1.2.4 forwards matching link-id traffic based on interface and hop count without checking `IDX_LT_VALIDATED` or destination type; endpoints still require LINK destination type for active-Link DATA dispatch. Link-addressed proofs use `link_table`, while `reverse_table` is for ordinary destination-routed DATA proofs. Corrected and runtime-locked by `tools/verify_transport_link.py`. diff --git a/SPEC.md b/SPEC.md index 298d36f..ea19d1a 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3252,7 +3252,7 @@ For any other forwarded DATA (the much-more-common opportunistic LXMF case), the time.time() ] # 2 IDX_RT_TIMESTAMP ``` -The reverse_table is what lets the eventual PROOF receipt (§6.5) trace its way back to the originator without consulting the path_table again — see §12.5. +The reverse_table is what lets the eventual PROOF receipt (§6.5) trace its way back to the originator without consulting the path_table again — see §12.5. The key is invariant across HEADER_1/HEADER_2 conversion, transport-id replacement, and hop increments because those mutable fields are excluded from `Packet.get_hashable_part()`. ### 12.3 ANNOUNCE rebroadcasting @@ -3355,7 +3355,7 @@ new_raw = packet.raw[0:1] + struct.pack("!B", packet.hops) + packet.raw[2:] Transport.transmit(reverse_entry[IDX_RT_RCVD_IF], new_raw) ``` -Reverse_table entries are popped on use (one-shot routing) and aged out by `Transport.jobs` after `Transport.REVERSE_TIMEOUT` (default `30s`). This bounds the relay's memory regardless of whether the proof ever arrives. +The pop occurs **before** the interface check. A PROOF arriving on the wrong interface is dropped but still consumes the reverse route, so a later valid copy cannot be forwarded. Reverse_table entries are otherwise aged out by `Transport.jobs` after `Transport.REVERSE_TIMEOUT = 8*60` (480 seconds in RNS 1.2.4), or when either recorded interface disappears. ### 12.6 Tunnels and shared-instance protocol @@ -3589,6 +3589,7 @@ A client running on a constrained device (less RAM, slower CPU) can scale all of | RNode receives correctly but TX is silent | §8.4.2 — KISS configuration handshake incomplete. CMD_RADIO_STATE = 0x01 must be the LAST step | §8.4.2 | | Received RSSI/SNR values are garbage | §8.4.5 — wrong sidecar decode. `RSSI = byte - 157`, `SNR = signed Q6.2 / 4`. Sidecar frames precede each `CMD_DATA` frame | §8.4.5 | | Multi-hop packets arrive but local-destination packets don't | §2.3 — originator HEADER_1→HEADER_2 conversion not applied for hops > 1. Originators must do this conversion themselves when path table reports `hops > 1` | `tools/verify_packet_header.py` | +| DATA reaches destination and a PROOF is observed, but originator never receives it | §12.5.3 — inspect every reverse-table entry and PROOF ingress interface. A wrong-interface PROOF is dropped after consuming the one-shot reverse route | `tools/verify_transport_data.py` | | Sending to multi-hop peers fails silently after path table populated | §7.6 — `TCPServerInterface.OUT` is True by default in practice (constructor's `False` is overridden at runtime). Don't waste time chasing a stuck OUT flag | §7.6 | ### LXMF specifics @@ -3730,7 +3731,7 @@ Embedded clean-room implementations need to know up front which data structures | `Transport.path_requests` | (unbounded — one entry per recently-issued path? request) | §7.1 | Aged out at `Transport.PATH_REQUEST_GATE_TIMEOUT = 120s`. | | `Transport.discovery_path_requests` | (unbounded) | §7.2.3, §12.6.1 | Aged out at `Transport.PATH_REQUEST_TIMEOUT = 15s`. | | `Transport.link_table` (transit-relay link state) | (unbounded) | §12.2.4, §12.5 | One per Link the relay is forwarding for; cleared on link teardown or stale aging. | -| `Transport.reverse_table` | (unbounded) | §12.5.3 | One entry per in-flight DATA→PROOF round-trip; popped on use, aged at `Transport.REVERSE_TIMEOUT = 30s`. | +| `Transport.reverse_table` | (unbounded) | §12.5.3 | One entry per in-flight DATA→PROOF round-trip; popped before proof-interface validation, aged at `Transport.REVERSE_TIMEOUT = 480s`, or removed when a recorded interface disappears. | | `Transport.tunnels` | (unbounded) | §12.6.2 | One per tunnel-able interface; aged at `Transport.TUNNEL_TIMEOUT`. | | `Transport.packet_hashlist` (dedup ring) | `Transport.hashlist_maxsize = 1,000,000` | §13.4 | Half is purged on next `Transport.jobs` after the cap is hit. | | `Transport.active_links` | (unbounded — one per active Link the node owns or relays) | §6 | | diff --git a/agent.md b/agent.md index 41c99a0..426a38d 100644 --- a/agent.md +++ b/agent.md @@ -170,7 +170,7 @@ Initial confidence assessment (subjective, not authoritative — re-do this audi | §10 Resource fragmentation | Source-audited against RNS 1.2.4 and runtime-verified by `tools/verify_resource.py`, including deterministic vectors, receiver assembly/proof, multi-segment sizing, and negative cases. | | §11 REQUEST/RESPONSE | Source-audited against RNS 1.2.4 and runtime-verified by `tools/verify_request_response.py`, including packet/Resource forms, request-ID domains, correlation, and authorization constants. | | §18 Test vectors | Populated with identities, announces, opportunistic LXMF, Link establishment, link-delivered LXMF, Resource, and REQUEST/RESPONSE. Future work should add broader negative vectors. | -| §12 Transport | Source-audited against RNS 1.2.4. Transport-relayed Link establishment, bidirectional traffic, and failure cases are runtime-verified by `tools/verify_transport_link.py`; broader announce/path/tunnel behavior remains source-cited. | +| §12 Transport | Source-audited against RNS 1.2.4. Relayed Link behavior is runtime-verified by `tools/verify_transport_link.py`; ordinary DATA and reverse-PROOF behavior by `tools/verify_transport_data.py`. Broader announce/path/tunnel behavior remains source-cited. | **Historical bootstrap tasks from the initial audit, now mostly complete:** diff --git a/audits/transport-data-tier1-rns-1.2.4.md b/audits/transport-data-tier1-rns-1.2.4.md new file mode 100644 index 0000000..a345b72 --- /dev/null +++ b/audits/transport-data-tier1-rns-1.2.4.md @@ -0,0 +1,66 @@ +# Tier 1 Audit: Destination-Routed DATA and PROOF Transport + +Question: How does RNS 1.2.4 forward ordinary destination-routed DATA across +multiple transport relays and route the returning PROOF? + +Evidence baseline: + +- RNS package: `rns==1.2.4` +- Primary sources: `RNS/Transport.py`, `RNS/Packet.py` +- Audit date: 2026-06-08 + +Tier 2 evidence is `tools/verify_transport_data.py` and +`test-vectors/transport-data.json`. Confirmed findings are promoted into +`SPEC.md` and `flows/transport-data.md`. + +## Confirmed Model + +1. An intermediate relay receiving HEADER_2 DATA addressed to its transport + identity keeps HEADER_2, increments hops, and replaces `transport_id` with + the next relay's identity hash. + +2. The final relay, whose path entry reports one remaining hop, strips the + transport slot and delivers HEADER_1 toward the destination. Ordinary + routed DATA does not arrive at the endpoint as HEADER_2. + +3. Every relay that forwards non-LINKREQUEST DATA writes a `reverse_table` + entry keyed by `packet.getTruncatedHash()`. This proof-destination hash is + invariant across HEADER_1/HEADER_2 conversion, transport-id replacement, + and hop increments. + +4. A returning ordinary PROOF is HEADER_1 and addressed to that truncated + packet hash. Each relay pops its reverse entry and forwards the proof on + the entry's recorded receive-side interface. + +5. The reverse entry is popped **before** the relay verifies that PROOF arrived + on the recorded outbound interface. A wrong-interface PROOF is dropped but + consumes the route; a later correct PROOF cannot be forwarded. + +6. DATA addressed to another transport identity is rejected by + `packet_filter`. DATA addressed to the local relay but lacking a path is + dropped without creating reverse state. + +7. `Transport.REVERSE_TIMEOUT` is 480 seconds in RNS 1.2.4, not 30 seconds. + Entries are also culled when either recorded interface disappears. + +## Corrections to Bootstrap Prose + +- The opportunistic receive flow incorrectly showed HEADER_2 arriving at the + endpoint after relay transport. The last relay strips the transport header. +- §12.5.3 and §16.1 stated `REVERSE_TIMEOUT = 30s`; upstream defines + `8*60`. +- §12.5.3 did not warn that a wrong-interface PROOF consumes reverse state. + +## Tier 2 Scope + +`tools/verify_transport_data.py` drives upstream `Transport.inbound` and +verifies: + +1. Intermediate HEADER_2 retargeting and final HEADER_1 delivery. +2. Stable proof-destination hash across all DATA wire forms. +3. Per-relay reverse-table entry shape. +4. Two-relay PROOF return and one-shot entry consumption. +5. Wrong-interface PROOF drop plus reverse-entry consumption. +6. Unknown-path and wrong-transport-identity DATA drops. +7. The RNS 1.2.4 reverse timeout constant. + diff --git a/flows/README.md b/flows/README.md index 1e16717..f855cbc 100644 --- a/flows/README.md +++ b/flows/README.md @@ -22,6 +22,8 @@ The two views are complementary: SPEC.md tells you what each piece looks like; t | [`receive-propagated-lxmf.md`](receive-propagated-lxmf.md) (recipient pulling messages via `/get`) | ✅ | | [`propagation-peer-sync.md`](propagation-peer-sync.md) (propagation-node announce, `/offer`, and peer Resource sync) | ✅ | | [`lxmf-outbound-retry.md`](lxmf-outbound-retry.md) (process_outbound retry loop, per-message state machine, fail_message) | ✅ | +| [`transport-link.md`](transport-link.md) (relayed Link establishment and established-Link traffic) | ✅ | +| [`transport-data.md`](transport-data.md) (ordinary DATA and returning PROOF through relays) | ✅ | ## Conventions diff --git a/flows/receive-opportunistic-lxmf.md b/flows/receive-opportunistic-lxmf.md index 2036d7c..642765f 100644 --- a/flows/receive-opportunistic-lxmf.md +++ b/flows/receive-opportunistic-lxmf.md @@ -195,20 +195,18 @@ SPEC.md §9.5: if the operator runs both an originator and a transport node on t ## Wire-byte summary (mirror of the send-flow summary) -What arrives at the recipient before deframing — assumes a 0-hop direct send (HEADER_1) or a >1-hop relayed packet that arrives as HEADER_2 with a transport_id at offset 2: +What arrives at the recipient before deframing is HEADER_1 whether sent directly or delivered by a last-hop transport relay. Intermediate relays use HEADER_2, but the final relay strips the transport-id slot before transmission: ``` HEADER_1: [ 1B flags ][ 1B hops ][ 16B dest_hash ][ 1B context=0x00 ] [ 32B sender_ephemeral_X25519_pub ][ 16B iv ] [ N×16B aes_ciphertext ][ 32B hmac_sha256 ] - -HEADER_2 (after relay): -[ 1B flags ][ 1B hops ][ 16B transport_id ][ 16B dest_hash ][ 1B context=0x00 ] -[ 32B sender_ephemeral_X25519_pub ][ 16B iv ] -[ N×16B aes_ciphertext ][ 32B hmac_sha256 ] ``` +See [`transport-data.md`](transport-data.md) for the intermediate HEADER_2 +forms and reverse-PROOF path. + After AES-CBC decryption with the matching ratchet (or long-term) key, the plaintext is the opportunistic LXMF body **without** the recipient's dest_hash (SPEC.md §5.1): ``` diff --git a/flows/send-opportunistic-lxmf.md b/flows/send-opportunistic-lxmf.md index fc04d0f..0f3c041 100644 --- a/flows/send-opportunistic-lxmf.md +++ b/flows/send-opportunistic-lxmf.md @@ -142,7 +142,7 @@ Strictly speaking, the flow above ends at step 10. Steps 11-13 are about **what ### 11. Recipient processes the inbound DATA packet -Inverse of steps 7-9, in the order: deframe → optional HEADER_2 strip / hop-table lookup → packet enters `Transport.inbound` → handed to the destination → `RNS.Destination.decrypt` reverses the Token (HMAC verified **before** AES decrypt per SPEC.md §3.3) → LXMF body parsed → Ed25519 signature verified, with the dual-msgpack-variant tolerance described in SPEC.md §5.6 → message surfaced to the recipient's app. +Inverse of steps 7-9, in the order: intermediate relays process HEADER_2 via their path tables → the last relay strips the transport slot and emits HEADER_1 → recipient deframes → packet enters `Transport.inbound` → handed to the destination → `RNS.Destination.decrypt` reverses the Token (HMAC verified **before** AES decrypt per SPEC.md §3.3) → LXMF body parsed → Ed25519 signature verified, with the dual-msgpack-variant tolerance described in SPEC.md §5.6 → message surfaced to the recipient's app. See `transport-data.md` for the relay sequence. The receive flow is its own document; see `receive-opportunistic-lxmf.md` (TODO) for the detailed step list. diff --git a/flows/transport-data.md b/flows/transport-data.md new file mode 100644 index 0000000..531007e --- /dev/null +++ b/flows/transport-data.md @@ -0,0 +1,46 @@ +# Flow: Destination-Routed DATA Through Transport Relays + +This flow follows ordinary DATA and its returning PROOF across two transport +relays, pinned to RNS 1.2.4. Opportunistic LXMF is the common example. + +## Forward DATA + +1. The originator emits HEADER_2 because its path has more than one hop. The + `transport_id` names relay 1; the final destination hash remains in the + second address slot. +2. Relay 1 verifies that `transport_id` names itself and finds the final + destination in `path_table`. +3. With more than one remaining hop, relay 1 increments hops, replaces + `transport_id` with relay 2's identity hash, and preserves HEADER_2. +4. Relay 1 records `reverse_table[packet.getTruncatedHash()]` with the ingress + and egress interfaces. +5. Relay 2 receives the retargeted HEADER_2 packet. With one remaining hop, it + increments hops, strips the transport slot, and emits HEADER_1 toward the + destination. +6. Relay 2 records its own reverse entry under the same truncated packet hash. + +The hash is stable because `Packet.get_hashable_part()` excludes hops, +transport type/header type, and the HEADER_2 transport-id slot. + +## Returning PROOF + +1. The destination emits a HEADER_1 PROOF addressed to the original DATA + packet's truncated hash. +2. Relay 2 pops that hash from `reverse_table`. If the PROOF arrived on the + recorded outbound interface, it increments hops and sends toward relay 1. +3. Relay 1 repeats the operation and sends toward the originator. +4. Each entry is one-shot. A duplicate PROOF has no route. + +## Failure Diagnostics + +| Observation | Meaning | +|---|---| +| First relay receives DATA but does not forward it | Check HEADER_2 `transport_id` and relay `path_table`. | +| Intermediate relay forwards HEADER_2 but final endpoint parser expects HEADER_2 | Endpoint receives HEADER_1; the last relay strips the transport slot. | +| DATA reaches endpoint but no PROOF returns | Inspect every relay's reverse entry and recorded interfaces. | +| PROOF appears first on the wrong interface | Upstream drops it after popping the reverse entry; a later correct copy cannot return. | +| Reverse entry remains indefinitely in a clean-room relay | RNS 1.2.4 expires it after 480 seconds or when a recorded interface disappears. | + +Executable evidence: `tools/verify_transport_data.py` and +`test-vectors/transport-data.json`. + diff --git a/playbook.md b/playbook.md index c490422..30c9ef3 100644 --- a/playbook.md +++ b/playbook.md @@ -247,6 +247,13 @@ Each entry: date, one-line symptom, spec section that governs it, one-line fix, - **Fix:** `useHeader2 = dest.hopCount > 1 && dest.nextHop != null`. Build the packet with `headerType = HEADER_2` and `transportId = dest.nextHop`. - **Lesson:** "It works at one hop" is not "it works." Test multi-hop early. +### 2026-06 — Wrong-interface PROOF consumes the reverse route + +- **Symptom:** DATA reaches the destination, a PROOF is observed, but the originator never receives it; later valid PROOF copies also fail. +- **Spec section:** §12.5.3. Upstream pops `reverse_table[proof_dest_hash]` before checking the PROOF's receiving interface. +- **Fix:** Prevent wrong-interface proof injection and log reverse-table creation, ingress, pop, and forwarding as one diagnostic sequence. +- **Lesson:** A dropped PROOF can mutate relay state; observing a later correct packet is not enough. + (Older entries: see `agent.md` §5 audit table and `reticulum-mobile-app/CLAUDE.md` "Key bugs we found" for additional history.) --- diff --git a/test-vectors/README.md b/test-vectors/README.md index 613c33f..0fc53e2 100644 --- a/test-vectors/README.md +++ b/test-vectors/README.md @@ -16,8 +16,9 @@ Populated against RNS 1.2.4 / LXMF 0.9.7: - ✅ `propagated-lxmf.json` — PROPAGATED LXMF submission packet/Resource boundary, full transient IDs, and `/get` framing (regenerator: `../tools/regen_propagated_lxmf.py`, verifier: `../tools/verify_propagated_lxmf.py`). - ✅ `propagation-peer.json` — propagation-node announce, directional peering key, `/offer`, and peer-sync Resource plaintext (regenerator: `../tools/regen_propagation_peer.py`, verifier: `../tools/verify_propagation_peer.py`). - ✅ `transport-link.json` — one-relay LINKREQUEST, LRPROOF, established-Link DATA, and invalid HEADER_2 fixtures (regenerator: `../tools/regen_transport_link.py`, verifier: `../tools/verify_transport_link.py`). +- ✅ `transport-data.json` — two-relay ordinary DATA and returning reverse-table PROOF fixtures (regenerator: `../tools/regen_transport_data.py`, verifier: `../tools/verify_transport_data.py`). -All ten files are byte-deterministic across runs: regenerators pin every random source (ephemeral keys, IVs, `random_hash` values, timestamps) so the output is reproducible against a fixed upstream RNS / LXMF version. +All eleven files are byte-deterministic across runs: regenerators pin every random source (ephemeral keys, IVs, `random_hash` values, timestamps) so the output is reproducible against a fixed upstream RNS / LXMF version. See [`../agent.md`](../agent.md) §3 and [`../todo.md`](../todo.md) for the evidence model and remaining task list. @@ -35,6 +36,7 @@ Each vector lives in a per-domain JSON file, e.g.: - `propagated-lxmf.json` — PROPAGATED submission and `/get` response forms - `propagation-peer.json` — propagation-node announce and peer-sync forms - `transport-link.json` — transport-relayed Link establishment and traffic forms +- `transport-data.json` — destination-routed DATA and reverse-PROOF forms Each entry should include: @@ -65,5 +67,6 @@ For the spec to claim "an implementation that passes all test vectors interopera 9. **Propagated LXMF** — recipient-encrypted submission, full transient-ID derivation, packet/Resource selection, and `/get` framing. 10. **Propagation peer sync** — node announce, directional peering key, `/offer` selection, and stamped Resource transfer. 11. **Transport-relayed Link** — LINKREQUEST/LRPROOF relay state, bidirectional established-Link forwarding, and invalid-header failure behavior. +12. **Transport-relayed DATA** — multi-relay HEADER_2/HEADER_1 transitions, reverse-table state, and returning proofs. A separate vector set for FAILURE cases is also useful: malformed announces, expired ratchets, mismatched signatures. An implementation should reject those as a regression-prevention measure. diff --git a/test-vectors/transport-data.json b/test-vectors/transport-data.json new file mode 100644 index 0000000..252df0f --- /dev/null +++ b/test-vectors/transport-data.json @@ -0,0 +1,30 @@ +{ + "_about": "Two-relay ordinary DATA and reverse-table PROOF fixtures derived from the deterministic opportunistic LXMF vector.", + "inputs": { + "lxmf_vector_label": "alice_to_bob_simple", + "relay_one_identity_label": "alice", + "relay_one_identity_hash_hex": "28d43a11abc1094301a59ed3b44f127b", + "relay_two_identity_label": "bob", + "relay_two_identity_hash_hex": "c090410e5b5bf8956194c1872dccec3b" + }, + "expected": { + "destination_hash_hex": "9695d17f22fa6e45d2b0cd3439a7ca7e", + "proof_destination_hash_hex": "15042202b08d6529fcb2c222e9dff137", + "origin_header1_data_raw_hex": "00009695d17f22fa6e45d2b0cd3439a7ca7e0021c3332b61be6a7b6ab8461e155651b17501b6e07532ecf9ab6661bd5a2ca57511223344556677889900aabbccddeeffefa9d24b76e1adf393cfd588214e236e219743697af96912b8eae84f3a1f28a3d68abd62f3e42c6944015c3d00e5e7aa8af732123d079ab10353597669c8cd3ba57cfae3a28ea1a99a44e0b492ba5deedd23232d2edab78fa037967757808c8578496aee7b21c70ce2476c54540d96d928e8ddf35c6bfb5d76261c07f1bb48af9d7bec8261cd30f3b03986614ba93173", + "relay_one_inbound_header2_data_raw_hex": "500028d43a11abc1094301a59ed3b44f127b9695d17f22fa6e45d2b0cd3439a7ca7e0021c3332b61be6a7b6ab8461e155651b17501b6e07532ecf9ab6661bd5a2ca57511223344556677889900aabbccddeeffefa9d24b76e1adf393cfd588214e236e219743697af96912b8eae84f3a1f28a3d68abd62f3e42c6944015c3d00e5e7aa8af732123d079ab10353597669c8cd3ba57cfae3a28ea1a99a44e0b492ba5deedd23232d2edab78fa037967757808c8578496aee7b21c70ce2476c54540d96d928e8ddf35c6bfb5d76261c07f1bb48af9d7bec8261cd30f3b03986614ba93173", + "relay_one_forwarded_header2_data_raw_hex": "5001c090410e5b5bf8956194c1872dccec3b9695d17f22fa6e45d2b0cd3439a7ca7e0021c3332b61be6a7b6ab8461e155651b17501b6e07532ecf9ab6661bd5a2ca57511223344556677889900aabbccddeeffefa9d24b76e1adf393cfd588214e236e219743697af96912b8eae84f3a1f28a3d68abd62f3e42c6944015c3d00e5e7aa8af732123d079ab10353597669c8cd3ba57cfae3a28ea1a99a44e0b492ba5deedd23232d2edab78fa037967757808c8578496aee7b21c70ce2476c54540d96d928e8ddf35c6bfb5d76261c07f1bb48af9d7bec8261cd30f3b03986614ba93173", + "relay_two_delivered_header1_data_raw_hex": "00029695d17f22fa6e45d2b0cd3439a7ca7e0021c3332b61be6a7b6ab8461e155651b17501b6e07532ecf9ab6661bd5a2ca57511223344556677889900aabbccddeeffefa9d24b76e1adf393cfd588214e236e219743697af96912b8eae84f3a1f28a3d68abd62f3e42c6944015c3d00e5e7aa8af732123d079ab10353597669c8cd3ba57cfae3a28ea1a99a44e0b492ba5deedd23232d2edab78fa037967757808c8578496aee7b21c70ce2476c54540d96d928e8ddf35c6bfb5d76261c07f1bb48af9d7bec8261cd30f3b03986614ba93173", + "destination_proof_raw_hex": "030015042202b08d6529fcb2c222e9dff13700abababababababababababababababababababababababababababababababababababababababababababababababababababababababababababababababab", + "relay_two_forwarded_proof_raw_hex": "030115042202b08d6529fcb2c222e9dff13700abababababababababababababababababababababababababababababababababababababababababababababababababababababababababababababababab", + "relay_one_forwarded_proof_raw_hex": "030215042202b08d6529fcb2c222e9dff13700abababababababababababababababababababababababababababababababababababababababababababababababababababababababababababababababab" + }, + "rns_version_at_generation": "1.2.4", + "lxmf_version_at_generation": "0.9.7", + "generator_script": "tools/regen_transport_data.py", + "verifies_spec_sections": [ + "2.3", + "6.5", + "12.2", + "12.5.3" + ] +} diff --git a/todo.md b/todo.md index 741e2c3..fae1183 100644 --- a/todo.md +++ b/todo.md @@ -80,6 +80,13 @@ Outstanding work for the spec repo. admission, and throttle timing, and adds `flows/propagation-peer-sync.md`. +- [x] **Transport DATA and reverse-PROOF three-tier work unit.** Tier 1 is + recorded in `audits/transport-data-tier1-rns-1.2.4.md`; Tier 2 adds + `test-vectors/transport-data.json`, `tools/regen_transport_data.py`, and + `tools/verify_transport_data.py`; Tier 3 corrects final-hop HEADER_1 + delivery, the 480-second reverse timeout, and wrong-interface PROOF + route consumption, and adds `flows/transport-data.md`. + ## Open `⚠️ UNVERIFIED` items in SPEC.md These need either a runtime test or a stronger upstream source citation @@ -316,10 +323,10 @@ discovery_path_requests): that survives interface flap, and the shared-instance wire protocol (just regular Reticulum packets over a TCP loopback; what's "shared" is the Transport state, not the wire format). -- [x] **Reverse-table link transport** — §12.5 covers LRPROOF - forwarding via link_table, Link DATA forwarding in both - directions once the link_table entry is validated, and PROOF - receipt forwarding via reverse_table (one-shot pop on use). +- [x] **Link-table and reverse-table transport** — §12.5 covers LRPROOF + forwarding and established-Link traffic via `link_table`, plus ordinary + DATA proof forwarding via `reverse_table`. Focused runtime verifiers are + `tools/verify_transport_link.py` and `tools/verify_transport_data.py`. ## Developer-experience gaps (would save real implementers real time) diff --git a/tools/regen_transport_data.py b/tools/regen_transport_data.py new file mode 100644 index 0000000..9b2b455 --- /dev/null +++ b/tools/regen_transport_data.py @@ -0,0 +1,111 @@ +""" +Regenerator for test-vectors/transport-data.json. + +Builds a deterministic two-relay opportunistic DATA and returning PROOF path +from the existing LXMF and identity vectors. +""" + +from __future__ import annotations + +import json +import os + +import RNS + + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +LXMF_PATH = os.path.join(REPO_ROOT, "test-vectors", "lxmf.json") +IDS_PATH = os.path.join(REPO_ROOT, "test-vectors", "identities.json") +OUT_PATH = os.path.join(REPO_ROOT, "test-vectors", "transport-data.json") + + +def load_json(path: str): + with open(path, "r", encoding="utf-8") as input_file: + return json.load(input_file) + + +def header_two(raw: bytes, transport_id: bytes) -> bytes: + flags = ( + (RNS.Packet.HEADER_2 << 6) + | (RNS.Transport.TRANSPORT << 4) + | (raw[0] & 0x0F) + ) + return bytes([flags, raw[1]]) + transport_id + raw[2:] + + +def packet(raw: bytes) -> RNS.Packet: + parsed = RNS.Packet(None, raw) + if not parsed.unpack(): + raise RuntimeError("generated invalid packet") + return parsed + + +def main() -> None: + lxmf = load_json(LXMF_PATH)["vectors"][0] + identities = load_json(IDS_PATH)["vectors"] + alice = next(item for item in identities if item["label"] == "alice") + bob = next(item for item in identities if item["label"] == "bob") + relay_one = bytes.fromhex(alice["expected"]["identity_hash_hex"]) + relay_two = bytes.fromhex(bob["expected"]["identity_hash_hex"]) + destination_hash = bytes.fromhex( + lxmf["expected"]["fields_layout"]["destination_hash_hex"] + ) + ciphertext = bytes.fromhex(lxmf["expected"]["token_ciphertext_hex"]) + + header_one_data = ( + bytes([(RNS.Destination.SINGLE << 2) | RNS.Packet.DATA, 0]) + + destination_hash + + bytes([RNS.Packet.NONE]) + + ciphertext + ) + inbound_relay_one = header_two(header_one_data, relay_one) + forwarded_relay_one = ( + inbound_relay_one[:1] + b"\x01" + relay_two + inbound_relay_one[18:] + ) + delivered_header_one = header_one_data[:1] + b"\x02" + header_one_data[2:] + proof_destination = packet(header_one_data).getTruncatedHash() + implicit_proof_body = bytes.fromhex("ab" * 64) + proof = ( + bytes([(RNS.Destination.SINGLE << 2) | RNS.Packet.PROOF, 0]) + + proof_destination + + bytes([RNS.Packet.NONE]) + + implicit_proof_body + ) + + vector = { + "_about": ( + "Two-relay ordinary DATA and reverse-table PROOF fixtures derived " + "from the deterministic opportunistic LXMF vector." + ), + "inputs": { + "lxmf_vector_label": lxmf["label"], + "relay_one_identity_label": "alice", + "relay_one_identity_hash_hex": relay_one.hex(), + "relay_two_identity_label": "bob", + "relay_two_identity_hash_hex": relay_two.hex(), + }, + "expected": { + "destination_hash_hex": destination_hash.hex(), + "proof_destination_hash_hex": proof_destination.hex(), + "origin_header1_data_raw_hex": header_one_data.hex(), + "relay_one_inbound_header2_data_raw_hex": inbound_relay_one.hex(), + "relay_one_forwarded_header2_data_raw_hex": forwarded_relay_one.hex(), + "relay_two_delivered_header1_data_raw_hex": delivered_header_one.hex(), + "destination_proof_raw_hex": proof.hex(), + "relay_two_forwarded_proof_raw_hex": (proof[:1] + b"\x01" + proof[2:]).hex(), + "relay_one_forwarded_proof_raw_hex": (proof[:1] + b"\x02" + proof[2:]).hex(), + }, + "rns_version_at_generation": RNS.__version__, + "lxmf_version_at_generation": "0.9.7", + "generator_script": "tools/regen_transport_data.py", + "verifies_spec_sections": ["2.3", "6.5", "12.2", "12.5.3"], + } + + with open(OUT_PATH, "w", encoding="utf-8") as output_file: + json.dump(vector, output_file, indent=2) + output_file.write("\n") + print(f"Wrote {OUT_PATH}") + + +if __name__ == "__main__": + main() diff --git a/tools/verify_transport_data.py b/tools/verify_transport_data.py new file mode 100644 index 0000000..34ad7b0 --- /dev/null +++ b/tools/verify_transport_data.py @@ -0,0 +1,288 @@ +""" +Verifier for ordinary DATA and PROOF routing through transport relays. + +Exercises a deterministic two-relay path against upstream RNS 1.2.4 +Transport.inbound. This distinguishes path_table forwarding from link_table +forwarding and verifies the one-shot reverse_table return path. +""" + +from __future__ import annotations + +import json +import os +import sys +import tempfile +import time + +import RNS +from RNS import Transport +from RNS.Transport import IDX_RT_OUTB_IF, IDX_RT_RCVD_IF, IDX_RT_TIMESTAMP + + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +VECTORS_PATH = os.path.join(REPO_ROOT, "test-vectors", "transport-data.json") +IDS_PATH = os.path.join(REPO_ROOT, "test-vectors", "identities.json") + + +def fail(message: str) -> None: + print(f"FAIL: {message}") + sys.exit(1) + + +def load_json(path: str): + with open(path, "r", encoding="utf-8") as input_file: + return json.load(input_file) + + +def init_minimal_rns(): + config_dir = tempfile.mkdtemp(prefix="rns-verify-transport-data-") + config_path = os.path.join(config_dir, "config") + with open(config_path, "w", encoding="utf-8") as config: + config.write("[reticulum]\nenable_transport = No\nshare_instance = No\n") + return RNS.Reticulum(configdir=config_dir, loglevel=0) + + +class FakeInterface: + OUT = True + IN = True + HW_MTU = RNS.Reticulum.MTU + AUTOCONFIGURE_MTU = True + FIXED_MTU = True + bitrate = 1_000_000 + mode = RNS.Interfaces.Interface.Interface.MODE_FULL + + def __init__(self, name: str): + self.name = name + + def __str__(self): + return self.name + + +def clear_state() -> None: + Transport.packet_hashlist = set() + Transport.packet_hashlist_prev = set() + Transport.path_table.clear() + Transport.reverse_table.clear() + Transport.link_table.clear() + Transport.destinations_map.clear() + Transport.receipts.clear() + Transport.local_client_interfaces.clear() + + +def seed_path(destination_hash: bytes, next_hop: bytes, hops: int, outbound_if) -> None: + Transport.path_table[destination_hash] = [ + time.time(), next_hop, hops, time.time() + 60, [], outbound_if, None, + ] + + +def expect_forward(captured: list, interface, raw: bytes, label: str) -> None: + if len(captured) != 1: + fail(f"{label}: expected one forwarded packet, got {len(captured)}") + if captured[0][0] is not interface: + fail(f"{label}: wrong outbound interface") + if captured[0][1] != raw: + fail(f"{label}: forwarded bytes mismatch") + + +def packet(raw: bytes) -> RNS.Packet: + parsed = RNS.Packet(None, raw) + if not parsed.unpack(): + fail("could not unpack fixture") + return parsed + + +def check_reverse_entry(entry: list, ingress, egress, label: str) -> None: + if entry[IDX_RT_RCVD_IF] is not ingress or entry[IDX_RT_OUTB_IF] is not egress: + fail(f"{label}: reverse-table interfaces mismatch") + if entry[IDX_RT_TIMESTAMP] > time.time(): + fail(f"{label}: reverse-table timestamp is in the future") + + +def verify_data_path(vector: dict, relay_one_identity, relay_two_identity, + origin_if, between_if, destination_if, captured: list) -> tuple[list, list]: + expected = vector["expected"] + destination_hash = bytes.fromhex(expected["destination_hash_hex"]) + proof_hash = bytes.fromhex(expected["proof_destination_hash_hex"]) + + Transport.identity = relay_one_identity + seed_path(destination_hash, relay_two_identity.hash, 2, between_if) + Transport.inbound( + bytes.fromhex(expected["relay_one_inbound_header2_data_raw_hex"]), origin_if + ) + expect_forward( + captured, between_if, + bytes.fromhex(expected["relay_one_forwarded_header2_data_raw_hex"]), + "intermediate relay DATA", + ) + if proof_hash not in Transport.reverse_table: + fail("intermediate relay did not create reverse-table entry") + relay_one_reverse = Transport.reverse_table[proof_hash] + check_reverse_entry(relay_one_reverse, origin_if, between_if, "intermediate relay") + + captured.clear() + Transport.packet_hashlist.clear() + Transport.packet_hashlist_prev.clear() + Transport.path_table.clear() + Transport.reverse_table.clear() + Transport.identity = relay_two_identity + seed_path(destination_hash, destination_hash, 1, destination_if) + Transport.inbound( + bytes.fromhex(expected["relay_one_forwarded_header2_data_raw_hex"]), between_if + ) + expect_forward( + captured, destination_if, + bytes.fromhex(expected["relay_two_delivered_header1_data_raw_hex"]), + "last-hop relay DATA", + ) + if proof_hash not in Transport.reverse_table: + fail("last-hop relay did not create reverse-table entry") + relay_two_reverse = Transport.reverse_table[proof_hash] + check_reverse_entry(relay_two_reverse, between_if, destination_if, "last-hop relay") + + forms = [ + bytes.fromhex(expected["origin_header1_data_raw_hex"]), + bytes.fromhex(expected["relay_one_inbound_header2_data_raw_hex"]), + bytes.fromhex(expected["relay_one_forwarded_header2_data_raw_hex"]), + bytes.fromhex(expected["relay_two_delivered_header1_data_raw_hex"]), + ] + hashes = {packet(raw).getTruncatedHash() for raw in forms} + if hashes != {proof_hash}: + fail("DATA proof destination changed across header/hop transformations") + + print("PASS S12.2 two-relay DATA path retargets HEADER_2 then delivers HEADER_1") + print("PASS S6.5/S12.2.5 proof destination is invariant and each relay records reverse state") + return relay_one_reverse, relay_two_reverse + + +def verify_proof_path(vector: dict, relay_one_identity, relay_two_identity, + origin_if, between_if, destination_if, captured: list, + relay_one_reverse: list, relay_two_reverse: list) -> None: + expected = vector["expected"] + proof_hash = bytes.fromhex(expected["proof_destination_hash_hex"]) + proof = bytes.fromhex(expected["destination_proof_raw_hex"]) + + captured.clear() + Transport.packet_hashlist.clear() + Transport.packet_hashlist_prev.clear() + Transport.reverse_table = {proof_hash: relay_two_reverse} + Transport.identity = relay_two_identity + Transport.inbound(proof, destination_if) + expect_forward( + captured, between_if, + bytes.fromhex(expected["relay_two_forwarded_proof_raw_hex"]), + "last-hop relay PROOF", + ) + if proof_hash in Transport.reverse_table: + fail("last-hop relay did not consume reverse-table entry") + + captured.clear() + Transport.packet_hashlist.clear() + Transport.packet_hashlist_prev.clear() + Transport.reverse_table = {proof_hash: relay_one_reverse} + Transport.identity = relay_one_identity + Transport.inbound( + bytes.fromhex(expected["relay_two_forwarded_proof_raw_hex"]), between_if + ) + expect_forward( + captured, origin_if, + bytes.fromhex(expected["relay_one_forwarded_proof_raw_hex"]), + "intermediate relay PROOF", + ) + if proof_hash in Transport.reverse_table: + fail("intermediate relay did not consume reverse-table entry") + + wrong_if = FakeInterface("wrong-proof-interface") + captured.clear() + Transport.packet_hashlist.clear() + Transport.packet_hashlist_prev.clear() + Transport.reverse_table = {proof_hash: relay_two_reverse} + Transport.identity = relay_two_identity + Transport.inbound(proof, wrong_if) + if captured: + fail("wrong-interface PROOF was forwarded") + if proof_hash in Transport.reverse_table: + fail("wrong-interface PROOF did not consume reverse-table entry") + + captured.clear() + Transport.packet_hashlist.clear() + Transport.packet_hashlist_prev.clear() + Transport.inbound(proof, destination_if) + if captured: + fail("PROOF routed after wrong-interface packet consumed reverse state") + + print("PASS S12.5.3 PROOF returns through both relays and consumes one-shot reverse entries") + print("PASS S12.5.3 wrong-interface PROOF is dropped after consuming its reverse entry") + + +def verify_guards(vector: dict, relay_one_identity, origin_if, between_if, + captured: list) -> None: + expected = vector["expected"] + destination_hash = bytes.fromhex(expected["destination_hash_hex"]) + incoming = bytes.fromhex(expected["relay_one_inbound_header2_data_raw_hex"]) + + captured.clear() + clear_state() + Transport.identity = relay_one_identity + Transport.inbound(incoming, origin_if) + if captured or Transport.reverse_table: + fail("DATA with no path was forwarded or recorded") + + captured.clear() + clear_state() + Transport.identity = RNS.Identity() + seed_path(destination_hash, bytes(16), 1, between_if) + Transport.inbound(incoming, origin_if) + if captured or Transport.reverse_table: + fail("DATA addressed to another transport identity was forwarded or recorded") + + if Transport.REVERSE_TIMEOUT != 8 * 60: + fail(f"REVERSE_TIMEOUT is {Transport.REVERSE_TIMEOUT}, expected 480 seconds") + print("PASS S12.2 unknown-path/wrong-transport DATA drops without reverse state") + print("PASS S12.5.3 RNS 1.2.4 REVERSE_TIMEOUT is 480 seconds") + + +def main() -> None: + print(f"verify_transport_data.py against RNS {RNS.__version__}") + init_minimal_rns() + vector = load_json(VECTORS_PATH) + identities = load_json(IDS_PATH)["vectors"] + alice = next(item for item in identities if item["label"] == "alice") + bob = next(item for item in identities if item["label"] == "bob") + relay_one_identity = RNS.Identity.from_bytes(bytes.fromhex(alice["inputs"]["private_key_hex"])) + relay_two_identity = RNS.Identity.from_bytes(bytes.fromhex(bob["inputs"]["private_key_hex"])) + origin_if = FakeInterface("origin-side") + between_if = FakeInterface("between-relays") + destination_if = FakeInterface("destination-side") + captured: list[tuple[object, bytes]] = [] + original_transmit = Transport.transmit + original_transport_enabled = RNS.Reticulum.transport_enabled + original_identity = Transport.identity + + try: + clear_state() + RNS.Reticulum.transport_enabled = staticmethod(lambda: True) + Transport.transmit = staticmethod(lambda interface, raw: captured.append((interface, raw))) + relay_one_reverse, relay_two_reverse = verify_data_path( + vector, relay_one_identity, relay_two_identity, + origin_if, between_if, destination_if, captured, + ) + verify_proof_path( + vector, relay_one_identity, relay_two_identity, + origin_if, between_if, destination_if, captured, + relay_one_reverse, relay_two_reverse, + ) + verify_guards(vector, relay_one_identity, origin_if, between_if, captured) + print("ALL PASS") + finally: + Transport.transmit = original_transmit + RNS.Reticulum.transport_enabled = original_transport_enabled + Transport.identity = original_identity + clear_state() + try: + RNS.Reticulum.exit_handler() + except Exception: + pass + + +if __name__ == "__main__": + main()