Implemented the portable verification baseline and completed the first Resource three-tier pass.

Clone Portability

Added fresh-clone setup instructions using repository-local .venv in README.md (line 28) and tools/README.md (line 12).
Documented that any virtual-environment path works and activation is optional.
Added .venv/ and venv/ to .gitignore (line 17).
Confirmed no tracked project files reference your specenv or rnsenv.
Verification Infrastructure

Added verify_all.py (line 1), which:
Enforces versions from tools/requirements.txt.
Runs every verifier independently.
Summarizes all failures.
Confirmed it rejects the older RNS 1.1.3/LXMF 0.9.3 environment.
Resource Audit

Added Tier 1 report: resource-tier1-rns-1.2.4.md (line 1).
Added verify_resource.py (line 1).
Corrected §10 and stale flow documentation:
Direct LXMF Resource threshold is 319 bytes.
Advertisement d is total logical-resource size.
Resource packets contain slices of one encrypted stream.
Exhausted requests can also request parts.
RESOURCE_RCL rejects advertisements; ordinary receiver cancellation is local-only.
Validation:

Passed: 16
Failed: 0
ALL VERIFIERS PASS
Remaining Resource work is deterministic resources.json vectors and negative/rejection cases.
This commit is contained in:
John Poole 2026-06-08 13:22:22 -07:00
commit 2c9ac94d7c
8 changed files with 69 additions and 15 deletions

2
.gitignore vendored
View file

@ -14,6 +14,8 @@
# Python bytecode # Python bytecode
__pycache__/ __pycache__/
*.pyc *.pyc
.venv/
venv/
# Local IDE / OS clutter # Local IDE / OS clutter
.vscode/ .vscode/

View file

@ -25,6 +25,22 @@ This fork proceeds with a three-tier evidence model that preserves the existing
See [`agent.md`](agent.md) §3 for the detailed rules. See [`agent.md`](agent.md) §3 for the detailed rules.
## Verify a Clone
The verifier suite has no dependency on a particular user's virtual
environment. From a fresh clone, create any isolated Python environment and
install the repository pins:
```sh
python3 -m venv .venv
.venv/bin/python -m pip install -r tools/requirements.txt
.venv/bin/python tools/verify_all.py
```
On Windows, use `.venv\Scripts\python.exe` in place of `.venv/bin/python`.
Activation is optional; `verify_all.py` uses the interpreter that launched it
and refuses to run when installed RNS/LXMF versions do not match the pins.
## What's here ## What's here
- [`SPEC.md`](SPEC.md) — the single combined spec document, organized by protocol layer - [`SPEC.md`](SPEC.md) — the single combined spec document, organized by protocol layer

21
SPEC.md
View file

@ -2411,7 +2411,7 @@ The repeater repo's `pre_build.py` patches several other microReticulum protocol
## 10. Resource fragmentation protocol ## 10. Resource fragmentation protocol
A **Resource** transfers a payload that exceeds the per-packet content limit of an established Reticulum Link. It is the only way to carry an LXMF body, NomadNet page, or file larger than ~360 bytes (`LINK_PACKET_MAX_CONTENT`) over a Link. Resource is built **on top of** an active Link — it relies on the Link's session key for encryption (§3.1 link-derived form) and on the Link's bidirectional DATA channel for control traffic. A **Resource** is the standard RNS mechanism for transferring a payload that exceeds the per-packet content limit of an established Reticulum Link. LXMF, Link REQUEST/RESPONSE, NomadNet, and file-transfer utilities use it for large bodies; an application can also define its own sequencing protocol or use Channel, so Resource is not the only possible large-data mechanism. With default RNS 1.2.4 / LXMF 0.9.7 parameters, DIRECT LXMF selects Resource when content exceeds **319 bytes** (`LINK_PACKET_MAX_CONTENT = Link.MDU(431) - LXMF_OVERHEAD(112)`, verified by `tools/verify_resource.py`). Resource is built **on top of** an active Link — it relies on the Link's session key for encryption (§3.1 link-derived form) and on the Link's bidirectional DATA channel for control traffic.
The complete reference is `RNS/Resource.py` (1380 lines in RNS 1.2.4); `RNS/Packet.py:74-79` defines the context constants. This section describes the wire-level invariants a clean-room implementation must respect; many implementation choices (window sizing heuristics, watchdog timers, EIFR computation) are private and listed only when their absence would cause an interop break. The complete reference is `RNS/Resource.py` (1380 lines in RNS 1.2.4); `RNS/Packet.py:74-79` defines the context constants. This section describes the wire-level invariants a clean-room implementation must respect; many implementation choices (window sizing heuristics, watchdog timers, EIFR computation) are private and listed only when their absence would cause an interop break.
@ -2438,7 +2438,7 @@ Given input data and an `RNS.Link` in `ACTIVE` state (`RNS/Resource.py:248-478`)
- `expected_proof = SHA256(plaintext || hash)` (32 bytes) — what the receiver will eventually return in the RESOURCE_PRF packet. - `expected_proof = SHA256(plaintext || hash)` (32 bytes) — what the receiver will eventually return in the RESOURCE_PRF packet.
The 4-byte prefix from step 3 is **not** in any of these inputs. The receiver strips the prefix and bz2-decompresses *before* hashing (§10.8 steps 3-5), so the sender must hash the uncompressed, unprefixed `plaintext` for the two sides to agree. A receiver that includes the prefix, or hashes the compressed form, rejects every legitimate Resource as `CORRUPT`. The 4-byte prefix from step 3 is **not** in any of these inputs. The receiver strips the prefix and bz2-decompresses *before* hashing (§10.8 steps 3-5), so the sender must hash the uncompressed, unprefixed `plaintext` for the two sides to agree. A receiver that includes the prefix, or hashes the compressed form, rejects every legitimate Resource as `CORRUPT`.
6. **Part split.** The encrypted body is sliced into parts of size `SDU = link.mtu - HEADER_MAXSIZE - IFAC_MIN_SIZE`. Each part becomes a packed `RNS.Packet(link, part_data, context=RESOURCE)`; the packed wire bytes are stored in `parts[i]` for later sending. 6. **Part split.** The encrypted body is sliced into parts of size `SDU = link.mtu - HEADER_MAXSIZE - IFAC_MIN_SIZE`. Each slice becomes the `data` body of a pre-packed `RNS.Packet(link, part_data, context=RESOURCE)` stored in `parts[i]`; `parts[i].raw` is the complete Reticulum wire packet. `Packet.pack()` does not encrypt `context=RESOURCE` again because Resource already encrypted the whole stream. (verified by `tools/verify_resource.py`)
7. **Hashmap.** Each part is fingerprinted to `MAPHASH_LEN = 4 bytes`. The full hashmap is `b"".join(map_hashes)`. **Hash collisions within the COLLISION_GUARD_SIZE = 2 × WINDOW_MAX + HASHMAP_MAX_LEN window are detected at construction time** — if two parts hash to the same 4-byte map_hash within that window, the random hash is regenerated and the whole hashmap is recomputed. Without this guard, the receiver can't disambiguate which part it just received from a part-request that named a colliding map_hash. 7. **Hashmap.** Each part is fingerprinted to `MAPHASH_LEN = 4 bytes`. The full hashmap is `b"".join(map_hashes)`. **Hash collisions within the COLLISION_GUARD_SIZE = 2 × WINDOW_MAX + HASHMAP_MAX_LEN window are detected at construction time** — if two parts hash to the same 4-byte map_hash within that window, the random hash is regenerated and the whole hashmap is recomputed. Without this guard, the receiver can't disambiguate which part it just received from a part-request that named a colliding map_hash.
After preparation: `total_parts = ceil(size / SDU)`; `total_size` includes metadata; `total_segments = ceil(total_size / MAX_EFFICIENT_SIZE)` where `MAX_EFFICIENT_SIZE = 1 MiB - 1 = 1_048_575`. After preparation: `total_parts = ceil(size / SDU)`; `total_size` includes metadata; `total_segments = ceil(total_size / MAX_EFFICIENT_SIZE)` where `MAX_EFFICIENT_SIZE = 1 MiB - 1 = 1_048_575`.
@ -2464,7 +2464,7 @@ The first packet in the transfer. Body is `umsgpack.packb(dict)` with these keys
| Key | Type | Meaning | | Key | Type | Meaning |
|---|---|---| |---|---|---|
| `t` | int | **Transfer size** — encrypted byte length on the wire | | `t` | int | **Transfer size** — encrypted byte length on the wire |
| `d` | int | **Data size** — original uncompressed plaintext byte length | | `d` | int | **Total logical-resource size** — original uncompressed size of the complete transfer, including metadata. For a multi-segment Resource this remains the total across all segments; it is not the current segment's plaintext length. |
| `n` | int | **Number of parts** in this segment | | `n` | int | **Number of parts** in this segment |
| `h` | bytes(32) | **Resource hash**`SHA256(data || random_hash)` | | `h` | bytes(32) | **Resource hash**`SHA256(data || random_hash)` |
| `r` | bytes(4) | **Random hash** prefix | | `r` | bytes(4) | **Random hash** prefix |
@ -2609,8 +2609,9 @@ If the part_index doesn't land on a `HASHMAP_MAX_LEN` boundary, the sender treat
> are independent. Serving the HMU is not a substitute for fulfilling > are independent. Serving the HMU is not a substitute for fulfilling
> the bundled part requests. > the bundled part requests.
> >
> In the RNS reference (`Resource.py:982-1071`, `request()` — verified > In the pinned RNS 1.2.4 reference (`Resource.py:982-1071`,
> against RNS 1.2.9, the current release), the part-fulfilment loop runs > `request()`, verified by `tools/verify_resource.py`), the
> part-fulfilment loop runs
> for every REQ regardless of the flag, and the `if wants_more_hashmap:` > for every REQ regardless of the flag, and the `if wants_more_hashmap:`
> HMU branch runs afterward, in addition. The reference receiver > HMU branch runs afterward, in addition. The reference receiver
> (`request_next`, `Resource.py:931-981`) routinely produces this > (`request_next`, `Resource.py:931-981`) routinely produces this
@ -2668,12 +2669,16 @@ The initiator's `validate_proof` (`Resource.py:785-824`) checks `proof_data[32:]
### 10.9 RESOURCE_ICL / RESOURCE_RCL — cancellation ### 10.9 RESOURCE_ICL / RESOURCE_RCL — cancellation
Either side can cancel; the body is just `resource_hash(32)`: The on-wire cancellation/rejection bodies are just `resource_hash(32)`:
- **`RESOURCE_ICL (0x06)`** — initiator cancel. Sent when the initiator decides to abort (e.g. the user kills the upload, the link MTU shrinks below the resource's pre-packed parts, the watchdog gives up after `MAX_RETRIES = 16`). - **`RESOURCE_ICL (0x06)`** — initiator cancel. Sent when the initiator decides to abort (e.g. the user kills the upload, the link MTU shrinks below the resource's pre-packed parts, the watchdog gives up after `MAX_RETRIES = 16`).
- **`RESOURCE_RCL (0x07)`** — receiver reject / cancel. Sent on advertisement reject (`Resource.reject(adv_packet)` at line 155-163, e.g. resource too large per app callback) or on receiver-side abort. - **`RESOURCE_RCL (0x07)`** — receiver rejection. Sent when an advertisement is rejected (`Resource.reject(adv_packet)` at line 155-163, e.g. resource too large per app callback), and when assembly marks a Resource `CORRUPT` before tearing down the Link (`Resource.cancel` line 1081-1084).
Either form transitions the resource to `FAILED`, releases the parts, and notifies the link's resource-concluded callback. An ordinary receiver-side `Resource.cancel()` does **not** emit
`RESOURCE_RCL`; it removes the incoming Resource locally. Implementations must
not assume every receiver-side abort is signalled to the initiator. An
initiator receiving `RESOURCE_RCL` transitions the outgoing Resource to
`REJECTED`; `RESOURCE_ICL` causes the receiver to cancel its incoming Resource.
### 10.10 Sliding window and rate adaptation ### 10.10 Sliding window and rate adaptation

View file

@ -92,7 +92,12 @@ For foundational wire behavior, prefer two evidence forms before calling the res
Agents working on this repo should have access to: Agents working on this repo should have access to:
- A working Python 3 install with `rns` and `lxmf` packages (install per "Staying current" below — not a bare `pip install rns lxmf`). - A working Python 3 install and an isolated virtual environment containing
the versions in `tools/requirements.txt`. The environment path is
contributor-local; `.venv` is the documented default, but no particular
path or activation command is required. Run the suite with that
environment's Python executable, for example
`.venv/bin/python tools/verify_all.py`.
- The `RNS/` and `LXMF/` source trees (typically at `~/AppData/Roaming/Python/Python3xx/site-packages/RNS/` on Windows or `~/.local/lib/python3.x/site-packages/RNS/` on Linux/macOS). - The `RNS/` and `LXMF/` source trees (typically at `~/AppData/Roaming/Python/Python3xx/site-packages/RNS/` on Windows or `~/.local/lib/python3.x/site-packages/RNS/` on Linux/macOS).
- Optional but very useful: a packet-trace tool. `tcpdump -i lo -A -X port 4242` works for TCPServerInterface; for BLE you need ADB + an RNode-aware capture tool. - Optional but very useful: a packet-trace tool. `tcpdump -i lo -A -X port 4242` works for TCPServerInterface; for BLE you need ADB + an RNode-aware capture tool.

View file

@ -18,7 +18,7 @@ DIRECT is reached two ways:
Within DIRECT there are two **representations** decided at pack time (`LXMF/LXMessage.py:414-421`): Within DIRECT there are two **representations** decided at pack time (`LXMF/LXMessage.py:414-421`):
- **PACKET** — the body fits in a single `LINK_PACKET_MAX_CONTENT`-sized DATA packet on the link. - **PACKET** — the body fits in a single `LINK_PACKET_MAX_CONTENT`-sized DATA packet on the link.
- **RESOURCE** — the body is larger than that and must be sent as an `RNS.Resource` (multi-packet, fragmented, with its own checksum / progress / retransmit machinery). The Resource fragmentation protocol is currently NOT in [`../SPEC.md`](../SPEC.md); see [`../todo.md`](../todo.md). This flow document covers PACKET in full and only sketches RESOURCE. - **RESOURCE** — the body is larger than that and must be sent as an `RNS.Resource` (multi-packet, fragmented, with its own checksum / progress / retransmit machinery). The Resource fragmentation protocol is documented in [`../SPEC.md`](../SPEC.md) §10 and the dedicated send/receive Resource flows. This flow document covers PACKET in full and links to those details for RESOURCE.
--- ---
@ -157,7 +157,7 @@ If the packed body exceeds `LINK_PACKET_MAX_CONTENT` (`LXMF/LXMessage.py:415-421
RNS.Resource(self.packed, self.__delivery_destination, callback=..., progress_callback=..., auto_compress=...) RNS.Resource(self.packed, self.__delivery_destination, callback=..., progress_callback=..., auto_compress=...)
``` ```
The `RNS.Resource` machinery handles fragmentation, ordering, retransmission, and progress reporting on top of the link's DATA channel. Each fragment is its own DATA packet on the link with its own PROOF receipt. The Resource fragmentation protocol — block sizes, sequence numbers, the resource-proof message — is **not in SPEC.md as of this writing** (see `../todo.md`); reading `RNS/Resource.py` is currently the only authoritative source. The `RNS.Resource` machinery handles fragmentation, ordering, retransmission, and progress reporting on top of the link's DATA channel. Each fragment is a DATA packet with `context=RESOURCE`; Resource-level completion is acknowledged by `RESOURCE_PRF`. See SPEC.md §10 and `flows/send-resource.md` / `flows/receive-resource.md`.
When the resource concludes successfully, the same `__mark_delivered` path runs as for PACKET. When the resource concludes successfully, the same `__mark_delivered` path runs as for PACKET.
@ -220,6 +220,6 @@ Per SPEC.md §6.5 the receiver of any CTX_NONE DATA packet on the link MUST emit
| 5 | `RNS/Cryptography/Token.py` | Token encrypt (no eph_pub prefix for Link) | | 5 | `RNS/Cryptography/Token.py` | Token encrypt (no eph_pub prefix for Link) |
| 6 | `RNS/Transport.py` | `outbound`, line 1031 | | 6 | `RNS/Transport.py` | `outbound`, line 1031 |
| 7 | `RNS/Packet.py` | `prove` | | 7 | `RNS/Packet.py` | `prove` |
| 8 | `RNS/Resource.py` | RESOURCE machinery (not yet in SPEC.md) | | 8 | `RNS/Resource.py` | RESOURCE machinery (SPEC.md §10) |
| 9 | `LXMF/LXMRouter.py` | backchannel identify, line 2532 | | 9 | `LXMF/LXMRouter.py` | backchannel identify, line 2532 |
| 10 | `RNS/Link.py` | watchdog / teardown | | 10 | `RNS/Link.py` | watchdog / teardown |

View file

@ -4,7 +4,7 @@ What happens chronologically when an LXMF DIRECT message, NomadNet page, or `rnc
Pinned against **RNS 1.2.4**. Wire-level details are in [`../SPEC.md`](../SPEC.md) §10; this document covers chronology and step ordering. Pinned against **RNS 1.2.4**. Wire-level details are in [`../SPEC.md`](../SPEC.md) §10; this document covers chronology and step ordering.
Out of scope: the receive side (`receive-resource.md` — TODO), Resource cancellation paths beyond a brief mention in step 9, and the watchdog / RTT estimation machinery (implementation-private). Out of scope: the receive side (documented separately in `receive-resource.md`), Resource cancellation paths beyond a brief mention in step 9, and the watchdog / RTT estimation machinery (implementation-private).
--- ---

View file

@ -21,6 +21,12 @@ Outstanding work for the spec repo.
## Test infrastructure ## Test infrastructure
- [ ] **Deterministic Resource vectors and negative cases.** The Tier 1 audit
is recorded in `audits/resource-tier1-rns-1.2.4.md`;
`tools/verify_resource.py` now runtime-locks the first focused claim
set, and confirmed corrections are promoted into SPEC.md §10. Add
`test-vectors/resources.json` plus malformed ADV, wrong-`r`, corrupt
part, invalid-HMU-boundary, and oversized-decompression rejection cases.
- [x] **Bootstrap `test-vectors/identities.json`** — Alice + Bob - [x] **Bootstrap `test-vectors/identities.json`** — Alice + Bob
identities populated against RNS 1.2.0. Regenerator at identities populated against RNS 1.2.0. Regenerator at
`tools/regen_identities.py`. `tools/regen_identities.py`.

View file

@ -11,11 +11,29 @@ Self-contained Python scripts that test claims in [`../SPEC.md`](../SPEC.md) aga
## Required environment ## Required environment
The verifier suite does not rely on any fixed virtual-environment path. Create
an isolated environment wherever appropriate. From the repository root, the
recommended POSIX setup is:
``` ```
pip install rns lxmf python3 -m venv .venv
.venv/bin/python -m pip install -r tools/requirements.txt
``` ```
The scripts read `RNS.__version__` at startup and print it in their output so a future reader can tell which RNS version a verification ran against. Run the complete pinned baseline:
```
.venv/bin/python tools/verify_all.py
```
On Windows, use `.venv\Scripts\python.exe` for both commands. Activating the
environment is optional. A custom environment path works equally well:
invoke `tools/verify_all.py` with that environment's Python executable.
`verify_all.py` refuses to run the suite when installed RNS/LXMF versions do
not exactly match `tools/requirements.txt`. Individual scripts read
`RNS.__version__` at startup and print it in their output so a future reader
can tell which RNS version a verification ran against.
## Status ## Status
@ -23,6 +41,7 @@ Populated against RNS 1.2.4 / LXMF 0.9.7:
| Script | Verifies SPEC.md section | Status | | Script | Verifies SPEC.md section | Status |
|---|---|---| |---|---|---|
| `verify_all.py` | pinned-version gate and complete verifier-suite runner | ✅ |
| `verify_destination_hash.py` | §1.1, §1.2, §1.3 — identity composition, `dest_hash = SHA256(name_hash \|\| identity_hash)[:16]`, on-disk private-key round-trip via `to_file`/`from_file` | ✅ | | `verify_destination_hash.py` | §1.1, §1.2, §1.3 — identity composition, `dest_hash = SHA256(name_hash \|\| identity_hash)[:16]`, on-disk private-key round-trip via `to_file`/`from_file` | ✅ |
| `verify_packet_header.py` | §2.1, §2.2, §2.3 — flag byte layout, HEADER_1/HEADER_2 form, originator HEADER_1→HEADER_2 conversion via upstream `Transport.outbound` | ✅ | | `verify_packet_header.py` | §2.1, §2.2, §2.3 — flag byte layout, HEADER_1/HEADER_2 form, originator HEADER_1→HEADER_2 conversion via upstream `Transport.outbound` | ✅ |
| `verify_token_crypto.py` | §3 — Token encrypt/decrypt, HKDF salt = identity_hash, HMAC-then-AES order, PKCS#7 padding | ✅ | | `verify_token_crypto.py` | §3 — Token encrypt/decrypt, HKDF salt = identity_hash, HMAC-then-AES order, PKCS#7 padding | ✅ |
@ -37,6 +56,7 @@ Populated against RNS 1.2.4 / LXMF 0.9.7:
| `verify_msgpack_quirk.py` | §9.3 — encoding name as bytes vs str affects upstream parsing | ✅ | | `verify_msgpack_quirk.py` | §9.3 — encoding name as bytes vs str affects upstream parsing | ✅ |
| `verify_stamps.py` | §5.7 — workblock determinism, PoW stamp search/validate, ticket shortcut | ✅ | | `verify_stamps.py` | §5.7 — workblock determinism, PoW stamp search/validate, ticket shortcut | ✅ |
| `verify_ratchet_dedup.py` | §7.3 / §4.5 step 6.3 — confirms replay defence is keyed on `random_blob`, NOT on `(dest_hash, ratchet_pub)` | ✅ | | `verify_ratchet_dedup.py` | §7.3 / §4.5 step 6.3 — confirms replay defence is keyed on `random_blob`, NOT on `(dest_hash, ratchet_pub)` | ✅ |
| `verify_resource.py` | §10.2, §10.4, §10.6, §10.7, §10.9, §10.11 — whole-stream encryption/slicing, hashes, ADV fields, exhausted REQ behavior, rejection/cancel distinction, multi-segment total size | ✅ |
| `regen_identities.py` | regenerates `test-vectors/identities.json` | ✅ | | `regen_identities.py` | regenerates `test-vectors/identities.json` | ✅ |
| `regen_announces.py` | regenerates `test-vectors/announces.json` (deterministic announce wire bytes, with and without ratchet) | ✅ | | `regen_announces.py` | regenerates `test-vectors/announces.json` (deterministic announce wire bytes, with and without ratchet) | ✅ |
| `regen_lxmf.py` | regenerates `test-vectors/lxmf.json` (deterministic opportunistic-LXMF plaintext + Token ciphertext) | ✅ | | `regen_lxmf.py` | regenerates `test-vectors/lxmf.json` (deterministic opportunistic-LXMF plaintext + Token ciphertext) | ✅ |