Compare commits
4 commits
ae5738ea2f
...
3659ba0324
| Author | SHA1 | Date | |
|---|---|---|---|
| 3659ba0324 | |||
| 1fb8b8ec10 | |||
| 2c9ac94d7c | |||
| e15e8fd62e |
14 changed files with 765 additions and 41 deletions
2
.gitignore
vendored
2
.gitignore
vendored
|
|
@ -14,6 +14,8 @@
|
|||
# Python bytecode
|
||||
__pycache__/
|
||||
*.pyc
|
||||
.venv/
|
||||
venv/
|
||||
|
||||
# Local IDE / OS clutter
|
||||
.vscode/
|
||||
|
|
|
|||
3
Codex_prompt_1.md
Normal file
3
Codex_prompt_1.md
Normal file
|
|
@ -0,0 +1,3 @@
|
|||
I have staged at /usr/local/src/reticulum/reticulum-specifications a fork from https://github.com/thatSFguy/reticulum-specifications which will be developed on https://salemdata.net/repo/jlpoole/reticiulum-specification
|
||||
|
||||
You previously performed a specification extraction of the LXMF protocol in three task steps for the project under /usr/local/src/reticulum/LXMF. Here in reticulum-specification, an extraction has already been performed by GitHub user thatSFguy using Claude AI. What I would like you to do is compare the analytical method you used for LXMF and then assess what Claude AI did (possibly preserved under /usr/local/src/reticulum/reticulum-specifications/.claude) and then report your findings. What I want to do is create a specification for Reticulum and I don't want to jetticson Claude's work, rather I'd like to build on it. I requested thatSFguy to identify himself to me confidentially and he ignored request; I do not want to develop something with a ghost, so I want to proceed in a way that will align with your approach and/or determine if you can adopt Claude's approach so we can move forward without a lot duplication.
|
||||
109
Codex_response_1.md
Normal file
109
Codex_response_1.md
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
**Assessment**
|
||||
|
||||
Do not jettison Claude’s work. It is a valuable and unusually extensive implementation-derived specification. However, it should be treated as a strong candidate specification undergoing verification, not yet as an authoritative specification.
|
||||
|
||||
I can adopt its approach. The best path is a hybrid: retain Claude’s corpus, workflows, vectors, and verifiers, while applying the narrower evidence discipline used for the LXMF extraction to each section before declaring it stable.
|
||||
|
||||
**Methods Compared**
|
||||
|
||||
The prior LXMF extraction followed three controlled stages:
|
||||
|
||||
1. **Source analysis**
|
||||
- Answered one narrow question: exact serialized `LXMessage` structure.
|
||||
- Recorded the exact upstream commit and source locations.
|
||||
- Explicitly distinguished confirmed and unresolved behavior.
|
||||
|
||||
2. **Independent structural verification**
|
||||
- Created deterministic Perl vectors and a separate MessagePack implementation.
|
||||
- Verified field positions, payload types, message ID, and signature input.
|
||||
- Explicitly did not claim signature validation because signatures were placeholders.
|
||||
|
||||
3. **Specification writing**
|
||||
- Wrote only the evidence-supported section.
|
||||
- Preserved unresolved questions.
|
||||
- Avoided assigning normative requirements beyond the evidence.
|
||||
|
||||
See [LXMF/SPEC.md](/usr/local/src/reticulum/LXMF/SPEC.md:1) and [verify_examples.pl](/usr/local/src/reticulum/LXMF/tools/verify_examples.pl:1).
|
||||
|
||||
Claude’s Reticulum approach began with a broad 545-line bootstrap specification derived from working implementation notes, then iteratively:
|
||||
|
||||
- inspected upstream source;
|
||||
- created upstream-runtime verifiers;
|
||||
- generated deterministic wire vectors;
|
||||
- added chronological protocol flows;
|
||||
- corrected claims based on issues and interoperability incidents;
|
||||
- documented maintenance and evidence rules.
|
||||
|
||||
The repository now contains approximately 11,000 lines of specification, verifier, vector, and flow material. All 59 commits identify Claude Opus as co-author.
|
||||
|
||||
**Where Claude’s Approach Is Stronger**
|
||||
|
||||
- The verifier scripts frequently exercise the actual upstream Python implementation rather than only an independent reconstruction.
|
||||
- Deterministic vectors include real signatures, encryption, handshakes, and derived session keys.
|
||||
- The repository records corrections instead of hiding mistakes.
|
||||
- The flows explain complete operations, not merely individual byte layouts.
|
||||
- The playbook captures important lessons about false-positive self-round-trip tests.
|
||||
- The evidence policy in [agent.md](/usr/local/src/reticulum/reticulum-specifications/agent.md:1) is fundamentally sound.
|
||||
|
||||
Representative strong artifact: [verify_lxmf_opportunistic.py](/usr/local/src/reticulum/reticulum-specifications/tools/verify_lxmf_opportunistic.py:1) performs real identity, signing, encryption, decryption, parsing, and signature-validation operations.
|
||||
|
||||
**Where Claude’s Approach Is Weaker**
|
||||
|
||||
The broad initial extraction moved faster than its evidence could support. Several material claims were initially wrong:
|
||||
|
||||
- private-key on-disk order;
|
||||
- packet flag bit layout;
|
||||
- LRPROOF body order and link ID offsets;
|
||||
- ratchet deduplication mechanism;
|
||||
- Resource integrity hash;
|
||||
- KEEPALIVE encryption and proof behavior;
|
||||
- Link DATA proof signing key;
|
||||
- `rrc.hub` encoding.
|
||||
|
||||
This is not a reason to discard the work. It demonstrates why broad source-derived prose must remain provisional until executable verification exists.
|
||||
|
||||
There are also maintenance inconsistencies:
|
||||
|
||||
- [SPEC.md](/usr/local/src/reticulum/reticulum-specifications/SPEC.md:3765) says announce, LXMF, and Link vectors are not populated, but all are present.
|
||||
- [agent.md](/usr/local/src/reticulum/reticulum-specifications/agent.md:105) still describes completed verifier/vector tasks as outstanding.
|
||||
- The repository is pinned to RNS 1.2.4/LXMF 0.9.7, but the available `rnsenv` contains RNS 1.1.3/LXMF 0.9.3. The verifier suite failed immediately on that mismatch.
|
||||
- Some broad behavioral claims rely on source citations or implementation experience rather than direct upstream-runtime or cross-implementation tests.
|
||||
|
||||
**Recommended Direction**
|
||||
|
||||
Use the existing repository as the working foundation, with these governance rules:
|
||||
|
||||
1. **Preserve Claude’s specification, flows, vectors, and history.**
|
||||
2. **Classify each section by evidence level:**
|
||||
- source-derived;
|
||||
- upstream-runtime verified;
|
||||
- independently reconstructed;
|
||||
- cross-implementation verified;
|
||||
- unresolved or application extension.
|
||||
3. **Require two forms of evidence for foundational wire claims.**
|
||||
- Prefer upstream-runtime verification plus an independent decoder/reconstruction or interoperability test.
|
||||
4. **Work section-by-section in narrow extraction tasks.**
|
||||
- Question/report.
|
||||
- Verifier and deterministic vector.
|
||||
- Specification correction or promotion.
|
||||
5. **Treat flows, failure guides, and implementation advice as informative rather than wire-normative.**
|
||||
6. **Make verification status machine-checkable**, rather than relying only on prose markers.
|
||||
7. **Require the exact pinned RNS/LXMF environment in CI** and run all verifiers there.
|
||||
8. **Add negative vectors** proving malformed packets are rejected.
|
||||
9. **Audit stale documentation automatically.**
|
||||
|
||||
**Regarding the Unidentified Contributor**
|
||||
|
||||
The Git history identifies the author as `Rob <rob@woodhousellc.com>` and Claude as co-author, but that does not authenticate the person behind the account. You do not need to depend on that identity to continue safely.
|
||||
|
||||
The sustainable trust boundary should be:
|
||||
|
||||
- not “trust the contributor”;
|
||||
- not “trust Claude” or “trust Codex”;
|
||||
- trust reproducible evidence, reviewed changes, exact upstream pins, and independently executable tests.
|
||||
|
||||
**Conclusion**
|
||||
|
||||
Claude’s work and the prior LXMF approach are compatible. Claude supplied breadth, operational experience, upstream-runtime tooling, and a substantial protocol map. The LXMF method supplies the stricter progression needed to turn that map into a defensible specification.
|
||||
|
||||
I recommend continuing from this repository, preserving its history, while treating each existing section as a candidate claim to be audited and promoted through the narrower three-stage LXMF process. No files were modified during this assessment.
|
||||
28
README.md
28
README.md
|
|
@ -15,6 +15,32 @@ This repo collects those findings in one place. The hope is that future client a
|
|||
|
||||
Each finding is grounded in upstream source citations (file + line) so it can be re-verified as RNS evolves.
|
||||
|
||||
## Method
|
||||
|
||||
This fork proceeds with a three-tier evidence model that preserves the existing Claude-assisted work while promoting claims only after verification:
|
||||
|
||||
1. **Source analysis report** — answer one narrow protocol question from pinned upstream source, with exact citations and unresolved cases.
|
||||
2. **Executable verification** — add runnable evidence under `tools/` and deterministic vectors under `test-vectors/` where bytes are involved.
|
||||
3. **Specification promotion** — update `SPEC.md` as normative prose only after the verifier/vector exists; add correction or incident notes when earlier text would mislead implementers.
|
||||
|
||||
See [`agent.md`](agent.md) §3 for the detailed rules.
|
||||
|
||||
## Verify a Clone
|
||||
|
||||
The verifier suite has no dependency on a particular user's virtual
|
||||
environment. From a fresh clone, create any isolated Python environment and
|
||||
install the repository pins:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r tools/requirements.txt
|
||||
.venv/bin/python tools/verify_all.py
|
||||
```
|
||||
|
||||
On Windows, use `.venv\Scripts\python.exe` in place of `.venv/bin/python`.
|
||||
Activation is optional; `verify_all.py` uses the interpreter that launched it
|
||||
and refuses to run when installed RNS/LXMF versions do not match the pins.
|
||||
|
||||
## What's here
|
||||
|
||||
- [`SPEC.md`](SPEC.md) — the single combined spec document, organized by protocol layer
|
||||
|
|
@ -75,6 +101,8 @@ If you've debugged a Reticulum interop problem and the answer wasn't in the upst
|
|||
|
||||
Add a worked test vector to `test-vectors/` if the finding is byte-level.
|
||||
|
||||
## Provenance
|
||||
This project is a fork (June 8, 2026) of https://github.com/thatSFguy/reticulum-specifications. thatSFguy's approach was to use Claude, I use Codex and have applied the approach I used for the LXMF-specification project here.
|
||||
## License
|
||||
|
||||
[CC BY 4.0](LICENSE) — use freely, attribution appreciated.
|
||||
|
|
|
|||
28
SPEC.md
28
SPEC.md
|
|
@ -1916,7 +1916,7 @@ The 32-byte `ratchet_pub` field in announces is meant to rotate periodically. Th
|
|||
|
||||
The actual replay-and-loop defence in upstream is keyed on **`random_hash`**, not on `ratchet_pub` — see §4.5 step 6.3 (path-table replacement check `not random_blob in random_blobs` at `RNS/Transport.py:1710, 1735, 1748`). Verified by `tools/verify_ratchet_dedup.py`: two announces sharing a `ratchet_pub` but differing in `random_hash[:5]` are both accepted by upstream's replay machinery.
|
||||
|
||||
> ⚠️ **Spec correction:** Earlier revisions of this section claimed transit nodes dedup announces on `(destination_hash, ratchet_pub)` tuples and that a non-rotating client becomes invisible to the mesh after one announce. That was wrong on the mechanism: upstream's `RATCHET_INTERVAL = 30 min` × `ANNOUNCE_INTERVAL = 5–15 min` means most upstream announces share a ratchet across 2–6 emissions, so if relays really dropped on `ratchet_pub` equality, upstream wouldn't function. The actual win observed in the bootstrap test (per `agent.md` §5) was incidental — the fix that rotated ratchets per announce also rotated `random_hash`, and it was the latter that mattered.
|
||||
> ⚠️ **Spec correction:** Earlier revisions of this section claimed transit nodes dedup announces on `(destination_hash, ratchet_pub)` tuples and that a non-rotating client becomes invisible to the mesh after one announce. That was wrong on the mechanism: upstream's `RATCHET_INTERVAL = 30 min` × `ANNOUNCE_INTERVAL = 5–15 min` means most upstream announces share a ratchet across 2–6 emissions, so if relays really dropped on `ratchet_pub` equality, upstream wouldn't function. The actual win observed in the bootstrap test (per `agent.md` §6) was incidental — the fix that rotated ratchets per announce also rotated `random_hash`, and it was the latter that mattered.
|
||||
|
||||
#### 7.3.1 Rotation cadence
|
||||
|
||||
|
|
@ -2411,7 +2411,7 @@ The repeater repo's `pre_build.py` patches several other microReticulum protocol
|
|||
|
||||
## 10. Resource fragmentation protocol
|
||||
|
||||
A **Resource** transfers a payload that exceeds the per-packet content limit of an established Reticulum Link. It is the only way to carry an LXMF body, NomadNet page, or file larger than ~360 bytes (`LINK_PACKET_MAX_CONTENT`) over a Link. Resource is built **on top of** an active Link — it relies on the Link's session key for encryption (§3.1 link-derived form) and on the Link's bidirectional DATA channel for control traffic.
|
||||
A **Resource** is the standard RNS mechanism for transferring a payload that exceeds the per-packet content limit of an established Reticulum Link. LXMF, Link REQUEST/RESPONSE, NomadNet, and file-transfer utilities use it for large bodies; an application can also define its own sequencing protocol or use Channel, so Resource is not the only possible large-data mechanism. With default RNS 1.2.4 / LXMF 0.9.7 parameters, DIRECT LXMF selects Resource when content exceeds **319 bytes** (`LINK_PACKET_MAX_CONTENT = Link.MDU(431) - LXMF_OVERHEAD(112)`, verified by `tools/verify_resource.py`). Resource is built **on top of** an active Link — it relies on the Link's session key for encryption (§3.1 link-derived form) and on the Link's bidirectional DATA channel for control traffic.
|
||||
|
||||
The complete reference is `RNS/Resource.py` (1380 lines in RNS 1.2.4); `RNS/Packet.py:74-79` defines the context constants. This section describes the wire-level invariants a clean-room implementation must respect; many implementation choices (window sizing heuristics, watchdog timers, EIFR computation) are private and listed only when their absence would cause an interop break.
|
||||
|
||||
|
|
@ -2438,7 +2438,7 @@ Given input data and an `RNS.Link` in `ACTIVE` state (`RNS/Resource.py:248-478`)
|
|||
- `expected_proof = SHA256(plaintext || hash)` (32 bytes) — what the receiver will eventually return in the RESOURCE_PRF packet.
|
||||
|
||||
The 4-byte prefix from step 3 is **not** in any of these inputs. The receiver strips the prefix and bz2-decompresses *before* hashing (§10.8 steps 3-5), so the sender must hash the uncompressed, unprefixed `plaintext` for the two sides to agree. A receiver that includes the prefix, or hashes the compressed form, rejects every legitimate Resource as `CORRUPT`.
|
||||
6. **Part split.** The encrypted body is sliced into parts of size `SDU = link.mtu - HEADER_MAXSIZE - IFAC_MIN_SIZE`. Each part becomes a packed `RNS.Packet(link, part_data, context=RESOURCE)`; the packed wire bytes are stored in `parts[i]` for later sending.
|
||||
6. **Part split.** The encrypted body is sliced into parts of size `SDU = link.mtu - HEADER_MAXSIZE - IFAC_MIN_SIZE`. Each slice becomes the `data` body of a pre-packed `RNS.Packet(link, part_data, context=RESOURCE)` stored in `parts[i]`; `parts[i].raw` is the complete Reticulum wire packet. `Packet.pack()` does not encrypt `context=RESOURCE` again because Resource already encrypted the whole stream. (verified by `tools/verify_resource.py`)
|
||||
7. **Hashmap.** Each part is fingerprinted to `MAPHASH_LEN = 4 bytes`. The full hashmap is `b"".join(map_hashes)`. **Hash collisions within the COLLISION_GUARD_SIZE = 2 × WINDOW_MAX + HASHMAP_MAX_LEN window are detected at construction time** — if two parts hash to the same 4-byte map_hash within that window, the random hash is regenerated and the whole hashmap is recomputed. Without this guard, the receiver can't disambiguate which part it just received from a part-request that named a colliding map_hash.
|
||||
|
||||
After preparation: `total_parts = ceil(size / SDU)`; `total_size` includes metadata; `total_segments = ceil(total_size / MAX_EFFICIENT_SIZE)` where `MAX_EFFICIENT_SIZE = 1 MiB - 1 = 1_048_575`.
|
||||
|
|
@ -2464,7 +2464,7 @@ The first packet in the transfer. Body is `umsgpack.packb(dict)` with these keys
|
|||
| Key | Type | Meaning |
|
||||
|---|---|---|
|
||||
| `t` | int | **Transfer size** — encrypted byte length on the wire |
|
||||
| `d` | int | **Data size** — original uncompressed plaintext byte length |
|
||||
| `d` | int | **Total logical-resource size** — original uncompressed size of the complete transfer, including metadata. For a multi-segment Resource this remains the total across all segments; it is not the current segment's plaintext length. |
|
||||
| `n` | int | **Number of parts** in this segment |
|
||||
| `h` | bytes(32) | **Resource hash** — `SHA256(data || random_hash)` |
|
||||
| `r` | bytes(4) | **Random hash** prefix |
|
||||
|
|
@ -2609,8 +2609,9 @@ If the part_index doesn't land on a `HASHMAP_MAX_LEN` boundary, the sender treat
|
|||
> are independent. Serving the HMU is not a substitute for fulfilling
|
||||
> the bundled part requests.
|
||||
>
|
||||
> In the RNS reference (`Resource.py:982-1071`, `request()` — verified
|
||||
> against RNS 1.2.9, the current release), the part-fulfilment loop runs
|
||||
> In the pinned RNS 1.2.4 reference (`Resource.py:982-1071`,
|
||||
> `request()`, verified by `tools/verify_resource.py`), the
|
||||
> part-fulfilment loop runs
|
||||
> for every REQ regardless of the flag, and the `if wants_more_hashmap:`
|
||||
> HMU branch runs afterward, in addition. The reference receiver
|
||||
> (`request_next`, `Resource.py:931-981`) routinely produces this
|
||||
|
|
@ -2668,12 +2669,16 @@ The initiator's `validate_proof` (`Resource.py:785-824`) checks `proof_data[32:]
|
|||
|
||||
### 10.9 RESOURCE_ICL / RESOURCE_RCL — cancellation
|
||||
|
||||
Either side can cancel; the body is just `resource_hash(32)`:
|
||||
The on-wire cancellation/rejection bodies are just `resource_hash(32)`:
|
||||
|
||||
- **`RESOURCE_ICL (0x06)`** — initiator cancel. Sent when the initiator decides to abort (e.g. the user kills the upload, the link MTU shrinks below the resource's pre-packed parts, the watchdog gives up after `MAX_RETRIES = 16`).
|
||||
- **`RESOURCE_RCL (0x07)`** — receiver reject / cancel. Sent on advertisement reject (`Resource.reject(adv_packet)` at line 155-163, e.g. resource too large per app callback) or on receiver-side abort.
|
||||
- **`RESOURCE_RCL (0x07)`** — receiver rejection. Sent when an advertisement is rejected (`Resource.reject(adv_packet)` at line 155-163, e.g. resource too large per app callback), and when assembly marks a Resource `CORRUPT` before tearing down the Link (`Resource.cancel` line 1081-1084).
|
||||
|
||||
Either form transitions the resource to `FAILED`, releases the parts, and notifies the link's resource-concluded callback.
|
||||
An ordinary receiver-side `Resource.cancel()` does **not** emit
|
||||
`RESOURCE_RCL`; it removes the incoming Resource locally. Implementations must
|
||||
not assume every receiver-side abort is signalled to the initiator. An
|
||||
initiator receiving `RESOURCE_RCL` transitions the outgoing Resource to
|
||||
`REJECTED`; `RESOURCE_ICL` causes the receiver to cancel its incoming Resource.
|
||||
|
||||
### 10.10 Sliding window and rate adaptation
|
||||
|
||||
|
|
@ -3761,8 +3766,11 @@ RRC does **not** use opportunistic packets, Resource transfer (§10), REQUEST/RE
|
|||
See [`test-vectors/`](test-vectors/). Currently populated:
|
||||
|
||||
- **`identities.json`** — Alice and Bob private-key inputs plus their derived `public_key`, `identity_hash`, and `lxmf.delivery` `destination_hash`. Verified by `tools/verify_destination_hash.py`; regenerated by `tools/regen_identities.py`. Covers SPEC.md §1.1 and §1.2.
|
||||
- **`announces.json`** — signed announce packets, with and without ratchet material. Verified by `tools/verify_announce_roundtrip.py`; regenerated by `tools/regen_announces.py`. Covers SPEC.md §4.1, §4.2, and §4.5.
|
||||
- **`lxmf.json`** — deterministic opportunistic LXMF plaintext and Token ciphertext vectors. Verified by `tools/verify_lxmf_opportunistic.py`; regenerated by `tools/regen_lxmf.py`. Covers SPEC.md §3 and §5.
|
||||
- **`links.json`** — Link handshake and LRRTT vectors, including LINKREQUEST, LRPROOF, derived keys, and the activation packet. Verified by `tools/verify_link_handshake.py` and `tools/verify_link_lrrtt.py`; regenerated by `tools/regen_links.py`. Covers SPEC.md §6.1-§6.4 and §6.6.
|
||||
|
||||
> ⚠️ **UNVERIFIED:** The remaining vector categories — signed announce packets, encrypted opportunistic LXMF DATA, and Link handshake (LINKREQUEST + LRPROOF + derived session keys) — are not yet populated. See [`agent.md`](agent.md) §5 and [`todo.md`](todo.md) for the remaining bootstrap work.
|
||||
Remaining vector work should focus on negative/rejection cases and link-delivered LXMF bodies rather than the original bootstrap categories.
|
||||
|
||||
An implementation that round-trips every test vector — both directions — should be wire-compatible with upstream Reticulum and LXMF for the covered operations.
|
||||
|
||||
|
|
|
|||
90
agent.md
90
agent.md
|
|
@ -51,11 +51,53 @@ PRs must include the verifier scripts. Don't commit a "verified" claim without t
|
|||
|
||||
---
|
||||
|
||||
## 3. Required tools
|
||||
## 3. Three-tier evidence model
|
||||
|
||||
This repository keeps the breadth of the existing Claude-assisted extraction, but promotes claims through the narrower three-tier process used for the LXMF extraction. Treat existing prose as useful candidate material until the relevant tier is satisfied.
|
||||
|
||||
### Tier 1 — Source analysis report
|
||||
|
||||
Use this tier to answer one narrow protocol question, not to rewrite a whole subsystem. The output is an analysis note, issue comment, or draft SPEC.md text that includes:
|
||||
|
||||
- The exact upstream RNS/LXMF version or commit inspected.
|
||||
- The files, functions, constants, and line numbers that define the behavior.
|
||||
- A byte layout or state-machine description precise enough to test.
|
||||
- A clear list of unresolved cases and assumptions.
|
||||
|
||||
Tier 1 is enough to add an `⚠️ UNVERIFIED` claim with citations. It is not enough to remove a marker for a foundational wire-format claim.
|
||||
|
||||
### Tier 2 — Executable verification
|
||||
|
||||
Convert the Tier 1 claim into runnable evidence under `tools/` and, where byte-level data is involved, deterministic vectors under `test-vectors/`.
|
||||
|
||||
Prefer upstream-runtime verification: a Python script should exercise installed upstream RNS/LXMF and compare exact bytes, parsed fields, signatures, keys, or state transitions. For high-risk layouts, add an independent reconstruction or decoder as a cross-check. The LXMF extraction's Perl MessagePack verifier is a good example of this second form.
|
||||
|
||||
Tier 2 is enough to remove an `⚠️ UNVERIFIED` marker when the verifier is committed beside the claim and passes against the pinned upstream versions.
|
||||
|
||||
### Tier 3 — Specification promotion and maintenance
|
||||
|
||||
Only after Tier 2, update `SPEC.md` as normative prose:
|
||||
|
||||
- Add or correct the section text.
|
||||
- Link the verifier or vector that backs the claim.
|
||||
- Preserve version-specific behavior if upstream changed.
|
||||
- Add a `README.md` spec-correction entry when earlier published text would mislead implementers.
|
||||
- Add a `playbook.md` incident entry when the finding explains a non-obvious interoperability failure.
|
||||
|
||||
For foundational wire behavior, prefer two evidence forms before calling the result stable: upstream-runtime verification plus either an independent decoder/reconstruction or a live interop test. Flows and troubleshooting prose may be source-cited without this second form, but must not claim byte-level finality they do not have.
|
||||
|
||||
---
|
||||
|
||||
## 4. Required tools
|
||||
|
||||
Agents working on this repo should have access to:
|
||||
|
||||
- A working Python 3 install with `rns` and `lxmf` packages (install per "Staying current" below — not a bare `pip install rns lxmf`).
|
||||
- A working Python 3 install and an isolated virtual environment containing
|
||||
the versions in `tools/requirements.txt`. The environment path is
|
||||
contributor-local; `.venv` is the documented default, but no particular
|
||||
path or activation command is required. Run the suite with that
|
||||
environment's Python executable, for example
|
||||
`.venv/bin/python tools/verify_all.py`.
|
||||
- The `RNS/` and `LXMF/` source trees (typically at `~/AppData/Roaming/Python/Python3xx/site-packages/RNS/` on Windows or `~/.local/lib/python3.x/site-packages/RNS/` on Linux/macOS).
|
||||
- Optional but very useful: a packet-trace tool. `tcpdump -i lo -A -X port 4242` works for TCPServerInterface; for BLE you need ADB + an RNode-aware capture tool.
|
||||
|
||||
|
|
@ -86,7 +128,7 @@ This spec is only as good as the upstream version it was checked against. **Befo
|
|||
|
||||
---
|
||||
|
||||
## 4. Marking convention
|
||||
## 5. Marking convention
|
||||
|
||||
Use these GitHub-flavored Markdown blockquote forms so they render distinctly:
|
||||
|
||||
|
|
@ -100,7 +142,7 @@ When you verify a previously-marked claim, **delete the entire blockquote** and
|
|||
|
||||
---
|
||||
|
||||
## 5. Audit pass — initial state of `SPEC.md`
|
||||
## 6. Audit pass — initial state of `SPEC.md`
|
||||
|
||||
The bootstrap `SPEC.md` was assembled from the working notes of two reverse-engineering efforts ([webclient](https://github.com/thatSFguy/reticulum-lora-webclient) and [mobile-app](https://github.com/thatSFguy/reticulum-mobile-app)). Some sections are already strongly verified by working code in those repos; some are source-cited but not directly tested in this spec repo's tools; some are observational claims that need formal verification.
|
||||
|
||||
|
|
@ -126,20 +168,20 @@ Initial confidence assessment (subjective, not authoritative — re-do this audi
|
|||
| §8 KISS / HDLC framing | High — both work in production on the reference clients |
|
||||
| §9.1–§9.8 Implementation gotchas | Each was a real bug that bit a real implementation. High confidence each is real; some lack formal test scripts. |
|
||||
| §10 Resource fragmentation | Source-cited from `RNS/Resource.py` against RNS 1.2.4; not yet runtime-verified in this repo's `tools/`. |
|
||||
| §11 Test vectors | The vectors themselves are verified; the test-vectors/ directory needs to be populated in this repo (currently partially populated). |
|
||||
| §11 Test vectors | Historical bootstrap item. `test-vectors/` is now populated with identities, announces, opportunistic LXMF, and Link vectors. Future work should add negative vectors and link-delivered LXMF coverage. |
|
||||
| §12 Source map | High |
|
||||
|
||||
**Concrete next-task list** for the agent picking this up:
|
||||
**Historical bootstrap tasks from the initial audit, now mostly complete:**
|
||||
|
||||
1. Run the audit — re-evaluate the table above with your own reasoning, don't just trust it.
|
||||
2. Populate `test-vectors/` with at least: identity material, a signed announce, an opportunistic-LXMF round-trip, a Link handshake.
|
||||
3. Write `tools/verify_announce.py` that loads a test vector and verifies bytes against upstream RNS.
|
||||
4. For each `> ⚠️` or `> 🔮` callout you add, write the verifier alongside.
|
||||
5. Open issues for any claim you can't verify and tag them `needs-verification`.
|
||||
1. Re-evaluate the table above with your own reasoning, not by copying it.
|
||||
2. Keep completed vectors current: identities, signed announces, opportunistic LXMF, and Link handshake.
|
||||
3. Maintain the verifier suite listed in `tools/README.md`; add focused scripts for new claims instead of broad catch-all verifiers.
|
||||
4. For each new `> ⚠️` or `> 🔮` callout, track the Tier 2 verifier or issue needed to resolve it.
|
||||
5. Open issues for claims that cannot be verified yet and tag them `needs-verification`.
|
||||
|
||||
---
|
||||
|
||||
## 6. PR rules
|
||||
## 7. PR rules
|
||||
|
||||
For any PR touching `SPEC.md`:
|
||||
|
||||
|
|
@ -156,7 +198,7 @@ For PRs adding to `test-vectors/`:
|
|||
|
||||
---
|
||||
|
||||
## 7. What to do when upstream changes
|
||||
## 8. What to do when upstream changes
|
||||
|
||||
RNS evolves. When a future RNS version changes the wire format (or the in-source behavior cited in this spec):
|
||||
|
||||
|
|
@ -169,7 +211,7 @@ The goal is for this spec to be useful even when run against an RNS version a ye
|
|||
|
||||
---
|
||||
|
||||
## 8. Don't
|
||||
## 9. Don't
|
||||
|
||||
- Don't paste large blocks of upstream code into this repo (license & churn). Cite by file + line + small inline snippet only.
|
||||
- Don't add claims based purely on what some other client does. Other clients have bugs too.
|
||||
|
|
@ -179,7 +221,7 @@ The goal is for this spec to be useful even when run against an RNS version a ye
|
|||
|
||||
---
|
||||
|
||||
## 9. Repo layout (current and aspirational)
|
||||
## 10. Repo layout
|
||||
|
||||
```
|
||||
reticulum-specifications/
|
||||
|
|
@ -188,15 +230,17 @@ reticulum-specifications/
|
|||
├── agent.md This file
|
||||
├── SPEC.md Combined spec (will be split into per-layer files as it grows)
|
||||
├── tools/ Verifier scripts (Python, callable against upstream RNS)
|
||||
│ ├── verify_announce.py (TODO)
|
||||
│ ├── verify_packet_header.py (TODO)
|
||||
│ ├── verify_lxmf_roundtrip.py (TODO)
|
||||
│ └── verify_link_handshake.py (TODO)
|
||||
│ ├── verify_announce_roundtrip.py
|
||||
│ ├── verify_packet_header.py
|
||||
│ ├── verify_lxmf_opportunistic.py
|
||||
│ ├── verify_link_handshake.py
|
||||
│ └── ...
|
||||
└── test-vectors/ Known-good byte sequences
|
||||
├── README.md (TODO — describe vector format)
|
||||
├── identities.json (TODO)
|
||||
├── announces.json (TODO)
|
||||
└── ...
|
||||
├── README.md
|
||||
├── identities.json
|
||||
├── announces.json
|
||||
├── lxmf.json
|
||||
└── links.json
|
||||
```
|
||||
|
||||
Tools are Python because Python RNS is the reference. Verifiers should be self-contained scripts that print PASS/FAIL plus a diagnostic on mismatch. Exit code 0 on PASS, non-zero on FAIL.
|
||||
|
|
|
|||
141
audits/resource-tier1-rns-1.2.4.md
Normal file
141
audits/resource-tier1-rns-1.2.4.md
Normal file
|
|
@ -0,0 +1,141 @@
|
|||
# Tier 1 Audit: Resource Fragmentation
|
||||
|
||||
Question: Does `SPEC.md` §10 accurately describe the wire-visible Resource
|
||||
fragmentation behavior of upstream RNS 1.2.4?
|
||||
|
||||
Evidence baseline:
|
||||
|
||||
- RNS package: `rns==1.2.4`
|
||||
- Source: installed package paths `RNS/Resource.py`, `RNS/Packet.py`,
|
||||
`RNS/Link.py`, and `LXMF/LXMessage.py`
|
||||
- Audit date: 2026-06-08
|
||||
|
||||
This began as a Tier 1 source-analysis report. The focused Tier 2 checks now
|
||||
live in `tools/verify_resource.py`; the confirmed F1-F6 corrections were
|
||||
promoted into `SPEC.md` §10 on 2026-06-08.
|
||||
|
||||
## Confirmed Core Model
|
||||
|
||||
The central §10 model matches RNS 1.2.4:
|
||||
|
||||
- Resource prepends a throwaway 4-byte prefix, encrypts the complete stream
|
||||
once with the Link, then slices the ciphertext into parts
|
||||
(`Resource.py:404-428`, `453-472`; `Packet.py:201-204`).
|
||||
- Advertisement `r` is a distinct 4-byte salt used for the resource hash and
|
||||
per-part map hashes (`Resource.py:440-443`, `505-506`).
|
||||
- Resource integrity is `SHA256(uncompressed_segment_plaintext || r)`, and the
|
||||
expected proof is `SHA256(uncompressed_segment_plaintext || resource_hash)`
|
||||
(`Resource.py:440-443`, `681-695`, `752-758`).
|
||||
- RESOURCE_REQ may carry requested part hashes while also requesting a hashmap
|
||||
continuation; the sender fulfils parts before emitting RESOURCE_HMU
|
||||
(`Resource.py:994-1027`, `1027-1064`).
|
||||
- RESOURCE parts are matched only within the receiver's current window, while
|
||||
sender lookup is bounded by `COLLISION_GUARD_SIZE`
|
||||
(`Resource.py:863-890`, `999-1010`).
|
||||
|
||||
## Findings Requiring Correction or Clarification
|
||||
|
||||
### F1 — Direct LXMF threshold is 319 bytes, not approximately 360
|
||||
|
||||
`SPEC.md` §10 introduction says Resource carries an LXMF body larger than
|
||||
approximately 360 bytes. With default RNS 1.2.4 / LXMF 0.9.7 parameters:
|
||||
|
||||
```text
|
||||
RNS.Link.MDU = 431
|
||||
LXMessage.LXMF_OVERHEAD = 112
|
||||
LXMessage.LINK_PACKET_MAX_CONTENT = 319
|
||||
```
|
||||
|
||||
`LXMessage.pack()` selects Resource representation for DIRECT delivery when
|
||||
`content_size > 319` (`LXMF/LXMessage.py:80-89`, `414-421`).
|
||||
|
||||
Recommended Tier 3 correction: say 319 bytes with default parameters, while
|
||||
noting that the threshold derives from Link MDU and may vary with protocol
|
||||
parameters.
|
||||
|
||||
### F2 — Resource is not the only possible way to carry larger application data
|
||||
|
||||
The §10 introduction calls Resource "the only way" to carry payloads exceeding
|
||||
one Link packet. Resource is the standard RNS mechanism used by LXMF, Link
|
||||
REQUEST/RESPONSE, and `rncp`, but applications can also stream or sequence data
|
||||
over Channel or their own Link DATA protocol.
|
||||
|
||||
Recommended Tier 3 correction: replace "the only way" with "the standard RNS
|
||||
mechanism used by LXMF, REQUEST/RESPONSE, and file-transfer utilities".
|
||||
|
||||
### F3 — Advertisement `d` is total logical-resource size
|
||||
|
||||
For multi-segment resources, advertisement field `d` is `resource.total_size`,
|
||||
the total uncompressed logical transfer size including metadata, not the
|
||||
plaintext size of the advertised segment (`Resource.py:281-314`,
|
||||
`Resource.py:1281-1283`).
|
||||
|
||||
Each segment still has its own:
|
||||
|
||||
- `t`: encrypted transfer size for this segment
|
||||
- `n`: part count for this segment
|
||||
- `h`: integrity hash for this segment
|
||||
- `r`: salt for this segment
|
||||
|
||||
Recommended Tier 3 correction: define `d` as total logical-resource size and
|
||||
explicitly distinguish it from the current segment's uncompressed plaintext
|
||||
length, which is not directly advertised.
|
||||
|
||||
### F4 — `RESOURCE_RCL` is not a general receiver-side cancel notification
|
||||
|
||||
`Resource.reject(advertisement_packet)` sends `RESOURCE_RCL`
|
||||
(`Resource.py:154-163`). A corrupt receiver also calls `reject()` and tears
|
||||
down the Link (`Resource.py:1081-1084`).
|
||||
|
||||
However, ordinary receiver-side `Resource.cancel()` only removes the incoming
|
||||
resource locally; it does not send `RESOURCE_RCL`
|
||||
(`Resource.py:1086-1097`). The current §10.9 wording implies either side can
|
||||
always notify cancellation on the wire.
|
||||
|
||||
Recommended Tier 3 correction: describe `RESOURCE_RCL` as advertisement
|
||||
rejection / corrupt-resource rejection. Do not claim ordinary receiver cancel
|
||||
always emits it.
|
||||
|
||||
### F5 — Resource-part packet storage wording is imprecise
|
||||
|
||||
`SPEC.md` §10.2 says packed wire bytes are stored in `parts[i]`. Upstream stores
|
||||
pre-packed `RNS.Packet` objects in `parts`; each packet's `data` is the raw
|
||||
ciphertext slice and its `raw` field is the packed Reticulum packet
|
||||
(`Resource.py:450-472`).
|
||||
|
||||
Recommended Tier 3 clarification: distinguish the stored Packet object, its
|
||||
Resource body (`part.data`), and complete Reticulum wire packet (`part.raw`).
|
||||
|
||||
### F6 — Pinned-source mismatch in §10.7 citation
|
||||
|
||||
The exhausted-REQ callout cites RNS 1.2.9 while this repository is pinned to
|
||||
RNS 1.2.4. The behavior is already present in RNS 1.2.4
|
||||
(`Resource.py:994-1064`).
|
||||
|
||||
Recommended Tier 3 correction: cite the pinned 1.2.4 behavior first; retain a
|
||||
later-version note only when documenting an actual version change.
|
||||
|
||||
## Tier 2 Verifier Scope
|
||||
|
||||
The first focused verifier is implemented in `tools/verify_resource.py` and
|
||||
avoids a live threaded transfer. It currently covers items 1-6 below:
|
||||
|
||||
1. Construct a Resource with a deterministic fake Link encryption key, fixed
|
||||
throwaway prefix, and fixed advertisement `r`.
|
||||
2. Verify whole-stream encryption occurs before slicing.
|
||||
3. Verify Resource-part Packet bodies are raw slices and are not packet-level
|
||||
re-encrypted.
|
||||
4. Verify advertisement dictionary fields, flags, and hashmap first segment.
|
||||
5. Verify `hash`, `truncated_hash`, `expected_proof`, and map-hash formulas.
|
||||
6. Verify RESOURCE_REQ parsing for both normal and exhausted-with-parts forms,
|
||||
including simultaneous part fulfilment and RESOURCE_HMU generation.
|
||||
7. Verify receiver assembly strips the throwaway prefix, decrypts once,
|
||||
validates the hash, and emits the expected RESOURCE_PRF bytes.
|
||||
8. Add a multi-segment fixture proving `d` remains total logical size while
|
||||
`t`, `n`, `h`, and `r` describe the current segment.
|
||||
9. Add negative cases: malformed ADV, wrong `r`, corrupt part, invalid HMU
|
||||
boundary, and oversized decompression.
|
||||
|
||||
Remaining Tier 2 work is a deterministic `test-vectors/resources.json` plus
|
||||
items 7-9. The confirmed first claim set has already been promoted into
|
||||
`SPEC.md` and the Resource flow documents.
|
||||
|
|
@ -18,7 +18,7 @@ DIRECT is reached two ways:
|
|||
Within DIRECT there are two **representations** decided at pack time (`LXMF/LXMessage.py:414-421`):
|
||||
|
||||
- **PACKET** — the body fits in a single `LINK_PACKET_MAX_CONTENT`-sized DATA packet on the link.
|
||||
- **RESOURCE** — the body is larger than that and must be sent as an `RNS.Resource` (multi-packet, fragmented, with its own checksum / progress / retransmit machinery). The Resource fragmentation protocol is currently NOT in [`../SPEC.md`](../SPEC.md); see [`../todo.md`](../todo.md). This flow document covers PACKET in full and only sketches RESOURCE.
|
||||
- **RESOURCE** — the body is larger than that and must be sent as an `RNS.Resource` (multi-packet, fragmented, with its own checksum / progress / retransmit machinery). The Resource fragmentation protocol is documented in [`../SPEC.md`](../SPEC.md) §10 and the dedicated send/receive Resource flows. This flow document covers PACKET in full and links to those details for RESOURCE.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -157,7 +157,7 @@ If the packed body exceeds `LINK_PACKET_MAX_CONTENT` (`LXMF/LXMessage.py:415-421
|
|||
RNS.Resource(self.packed, self.__delivery_destination, callback=..., progress_callback=..., auto_compress=...)
|
||||
```
|
||||
|
||||
The `RNS.Resource` machinery handles fragmentation, ordering, retransmission, and progress reporting on top of the link's DATA channel. Each fragment is its own DATA packet on the link with its own PROOF receipt. The Resource fragmentation protocol — block sizes, sequence numbers, the resource-proof message — is **not in SPEC.md as of this writing** (see `../todo.md`); reading `RNS/Resource.py` is currently the only authoritative source.
|
||||
The `RNS.Resource` machinery handles fragmentation, ordering, retransmission, and progress reporting on top of the link's DATA channel. Each fragment is a DATA packet with `context=RESOURCE`; Resource-level completion is acknowledged by `RESOURCE_PRF`. See SPEC.md §10 and `flows/send-resource.md` / `flows/receive-resource.md`.
|
||||
|
||||
When the resource concludes successfully, the same `__mark_delivered` path runs as for PACKET.
|
||||
|
||||
|
|
@ -220,6 +220,6 @@ Per SPEC.md §6.5 the receiver of any CTX_NONE DATA packet on the link MUST emit
|
|||
| 5 | `RNS/Cryptography/Token.py` | Token encrypt (no eph_pub prefix for Link) |
|
||||
| 6 | `RNS/Transport.py` | `outbound`, line 1031 |
|
||||
| 7 | `RNS/Packet.py` | `prove` |
|
||||
| 8 | `RNS/Resource.py` | RESOURCE machinery (not yet in SPEC.md) |
|
||||
| 8 | `RNS/Resource.py` | RESOURCE machinery (SPEC.md §10) |
|
||||
| 9 | `LXMF/LXMRouter.py` | backchannel identify, line 2532 |
|
||||
| 10 | `RNS/Link.py` | watchdog / teardown |
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ What happens chronologically when an LXMF DIRECT message, NomadNet page, or `rnc
|
|||
|
||||
Pinned against **RNS 1.2.4**. Wire-level details are in [`../SPEC.md`](../SPEC.md) §10; this document covers chronology and step ordering.
|
||||
|
||||
Out of scope: the receive side (`receive-resource.md` — TODO), Resource cancellation paths beyond a brief mention in step 9, and the watchdog / RTT estimation machinery (implementation-private).
|
||||
Out of scope: the receive side (documented separately in `receive-resource.md`), Resource cancellation paths beyond a brief mention in step 9, and the watchdog / RTT estimation machinery (implementation-private).
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ Populated against RNS 1.2.4 / LXMF 0.9.7:
|
|||
|
||||
All four files are byte-deterministic across runs: regenerators pin every random source (ephemeral keys, IVs, `random_hash` prefix + timestamp, LXMF timestamp) so the output is reproducible against a fixed upstream RNS / LXMF version.
|
||||
|
||||
See [`../agent.md`](../agent.md) §5 and [`../todo.md`](../todo.md) for the remaining bootstrap task list.
|
||||
See [`../agent.md`](../agent.md) §3 and [`../todo.md`](../todo.md) for the evidence model and remaining task list.
|
||||
|
||||
## Format (proposed)
|
||||
|
||||
|
|
|
|||
6
todo.md
6
todo.md
|
|
@ -21,6 +21,12 @@ Outstanding work for the spec repo.
|
|||
|
||||
## Test infrastructure
|
||||
|
||||
- [ ] **Deterministic Resource vectors and negative cases.** The Tier 1 audit
|
||||
is recorded in `audits/resource-tier1-rns-1.2.4.md`;
|
||||
`tools/verify_resource.py` now runtime-locks the first focused claim
|
||||
set, and confirmed corrections are promoted into SPEC.md §10. Add
|
||||
`test-vectors/resources.json` plus malformed ADV, wrong-`r`, corrupt
|
||||
part, invalid-HMU-boundary, and oversized-decompression rejection cases.
|
||||
- [x] **Bootstrap `test-vectors/identities.json`** — Alice + Bob
|
||||
identities populated against RNS 1.2.0. Regenerator at
|
||||
`tools/regen_identities.py`.
|
||||
|
|
|
|||
|
|
@ -11,11 +11,29 @@ Self-contained Python scripts that test claims in [`../SPEC.md`](../SPEC.md) aga
|
|||
|
||||
## Required environment
|
||||
|
||||
The verifier suite does not rely on any fixed virtual-environment path. Create
|
||||
an isolated environment wherever appropriate. From the repository root, the
|
||||
recommended POSIX setup is:
|
||||
|
||||
```
|
||||
pip install rns lxmf
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r tools/requirements.txt
|
||||
```
|
||||
|
||||
The scripts read `RNS.__version__` at startup and print it in their output so a future reader can tell which RNS version a verification ran against.
|
||||
Run the complete pinned baseline:
|
||||
|
||||
```
|
||||
.venv/bin/python tools/verify_all.py
|
||||
```
|
||||
|
||||
On Windows, use `.venv\Scripts\python.exe` for both commands. Activating the
|
||||
environment is optional. A custom environment path works equally well:
|
||||
invoke `tools/verify_all.py` with that environment's Python executable.
|
||||
|
||||
`verify_all.py` refuses to run the suite when installed RNS/LXMF versions do
|
||||
not exactly match `tools/requirements.txt`. Individual scripts read
|
||||
`RNS.__version__` at startup and print it in their output so a future reader
|
||||
can tell which RNS version a verification ran against.
|
||||
|
||||
## Status
|
||||
|
||||
|
|
@ -23,6 +41,7 @@ Populated against RNS 1.2.4 / LXMF 0.9.7:
|
|||
|
||||
| Script | Verifies SPEC.md section | Status |
|
||||
|---|---|---|
|
||||
| `verify_all.py` | pinned-version gate and complete verifier-suite runner | ✅ |
|
||||
| `verify_destination_hash.py` | §1.1, §1.2, §1.3 — identity composition, `dest_hash = SHA256(name_hash \|\| identity_hash)[:16]`, on-disk private-key round-trip via `to_file`/`from_file` | ✅ |
|
||||
| `verify_packet_header.py` | §2.1, §2.2, §2.3 — flag byte layout, HEADER_1/HEADER_2 form, originator HEADER_1→HEADER_2 conversion via upstream `Transport.outbound` | ✅ |
|
||||
| `verify_token_crypto.py` | §3 — Token encrypt/decrypt, HKDF salt = identity_hash, HMAC-then-AES order, PKCS#7 padding | ✅ |
|
||||
|
|
@ -37,9 +56,10 @@ Populated against RNS 1.2.4 / LXMF 0.9.7:
|
|||
| `verify_msgpack_quirk.py` | §9.3 — encoding name as bytes vs str affects upstream parsing | ✅ |
|
||||
| `verify_stamps.py` | §5.7 — workblock determinism, PoW stamp search/validate, ticket shortcut | ✅ |
|
||||
| `verify_ratchet_dedup.py` | §7.3 / §4.5 step 6.3 — confirms replay defence is keyed on `random_blob`, NOT on `(dest_hash, ratchet_pub)` | ✅ |
|
||||
| `verify_resource.py` | §10.2, §10.4, §10.6, §10.7, §10.9, §10.11 — whole-stream encryption/slicing, hashes, ADV fields, exhausted REQ behavior, rejection/cancel distinction, multi-segment total size | ✅ |
|
||||
| `regen_identities.py` | regenerates `test-vectors/identities.json` | ✅ |
|
||||
| `regen_announces.py` | regenerates `test-vectors/announces.json` (deterministic announce wire bytes, with and without ratchet) | ✅ |
|
||||
| `regen_lxmf.py` | regenerates `test-vectors/lxmf.json` (deterministic opportunistic-LXMF plaintext + Token ciphertext) | ✅ |
|
||||
| `regen_links.py` | regenerates `test-vectors/links.json` (deterministic LINKREQUEST + LRPROOF + derived session key) | ✅ |
|
||||
|
||||
See [`../agent.md`](../agent.md) §5 and [`../todo.md`](../todo.md) for the remaining priority order.
|
||||
See [`../agent.md`](../agent.md) §3 and [`../todo.md`](../todo.md) for the evidence model and remaining priority order.
|
||||
|
|
|
|||
110
tools/verify_all.py
Normal file
110
tools/verify_all.py
Normal file
|
|
@ -0,0 +1,110 @@
|
|||
#!/usr/bin/env python3
|
||||
"""
|
||||
Run the complete verifier suite against the versions pinned in requirements.txt.
|
||||
|
||||
This is the repository's Tier 2 baseline gate. It refuses to run individual
|
||||
verifiers when the active Python environment does not contain the exact pinned
|
||||
RNS and LXMF versions, then runs every tools/verify_*.py script in isolation.
|
||||
|
||||
Exit code 0 means every verifier passed. Exit code 1 means the environment
|
||||
does not match the pins or at least one verifier failed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import importlib.metadata
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
TOOLS_DIR = pathlib.Path(__file__).resolve().parent
|
||||
REQUIREMENTS_PATH = TOOLS_DIR / "requirements.txt"
|
||||
PIN_PATTERN = re.compile(r"^(rns|lxmf)==([A-Za-z0-9_.+-]+)$", re.IGNORECASE)
|
||||
|
||||
|
||||
def load_pins() -> dict[str, str]:
|
||||
pins: dict[str, str] = {}
|
||||
for raw_line in REQUIREMENTS_PATH.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
match = PIN_PATTERN.match(line)
|
||||
if match:
|
||||
pins[match.group(1).lower()] = match.group(2)
|
||||
|
||||
missing = {"rns", "lxmf"} - pins.keys()
|
||||
if missing:
|
||||
names = ", ".join(sorted(missing))
|
||||
raise RuntimeError(f"missing required pins in {REQUIREMENTS_PATH}: {names}")
|
||||
return pins
|
||||
|
||||
|
||||
def installed_versions(names: list[str]) -> dict[str, str]:
|
||||
versions: dict[str, str] = {}
|
||||
for name in names:
|
||||
try:
|
||||
versions[name] = importlib.metadata.version(name)
|
||||
except importlib.metadata.PackageNotFoundError:
|
||||
versions[name] = "<not installed>"
|
||||
return versions
|
||||
|
||||
|
||||
def verifier_paths() -> list[pathlib.Path]:
|
||||
return sorted(
|
||||
path
|
||||
for path in TOOLS_DIR.glob("verify_*.py")
|
||||
if path.name != pathlib.Path(__file__).name
|
||||
)
|
||||
|
||||
|
||||
def main() -> int:
|
||||
try:
|
||||
pins = load_pins()
|
||||
except (OSError, RuntimeError) as exc:
|
||||
print(f"FAIL baseline configuration: {exc}")
|
||||
return 1
|
||||
|
||||
installed = installed_versions(sorted(pins))
|
||||
mismatches = [
|
||||
f"{name}: installed {installed[name]}, pinned {pins[name]}"
|
||||
for name in sorted(pins)
|
||||
if installed[name] != pins[name]
|
||||
]
|
||||
|
||||
print(f"Python: {sys.executable}")
|
||||
print("Pinned environment: " + ", ".join(
|
||||
f"{name}=={pins[name]}" for name in sorted(pins)
|
||||
))
|
||||
|
||||
if mismatches:
|
||||
print("FAIL environment does not match tools/requirements.txt:")
|
||||
for mismatch in mismatches:
|
||||
print(f" - {mismatch}")
|
||||
return 1
|
||||
|
||||
verifiers = verifier_paths()
|
||||
if not verifiers:
|
||||
print("FAIL no verifier scripts found")
|
||||
return 1
|
||||
|
||||
failures: list[tuple[str, int]] = []
|
||||
for index, path in enumerate(verifiers, start=1):
|
||||
print(f"\n=== [{index}/{len(verifiers)}] {path.name} ===", flush=True)
|
||||
result = subprocess.run([sys.executable, str(path)], cwd=TOOLS_DIR.parent)
|
||||
if result.returncode != 0:
|
||||
failures.append((path.name, result.returncode))
|
||||
|
||||
print("\n=== Verification summary ===")
|
||||
print(f"Passed: {len(verifiers) - len(failures)}")
|
||||
print(f"Failed: {len(failures)}")
|
||||
if failures:
|
||||
for name, returncode in failures:
|
||||
print(f" - {name}: exit {returncode}")
|
||||
return 1
|
||||
|
||||
print("ALL VERIFIERS PASS")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
253
tools/verify_resource.py
Normal file
253
tools/verify_resource.py
Normal file
|
|
@ -0,0 +1,253 @@
|
|||
"""
|
||||
Verifier for SPEC.md S10 (Resource fragmentation protocol).
|
||||
|
||||
Exercises upstream RNS 1.2.4 Resource construction with a deterministic fake
|
||||
Link and verifies:
|
||||
|
||||
1. Default direct-LXMF single-packet content limit is 319 bytes.
|
||||
2. Resource encrypts one complete stream, then slices that ciphertext into
|
||||
RESOURCE packet bodies without packet-level re-encryption.
|
||||
3. Resource hash, expected proof, and map-hash formulas.
|
||||
4. RESOURCE_ADV fields and flags.
|
||||
5. Multi-segment advertisement `d` is total logical-resource size.
|
||||
6. An exhausted RESOURCE_REQ may carry part requests; upstream fulfils the
|
||||
requested parts and emits RESOURCE_HMU.
|
||||
7. RESOURCE_RCL is emitted for advertisement rejection, while an ordinary
|
||||
receiver-side cancel is local-only.
|
||||
|
||||
Exit code 0 on PASS, non-zero on FAIL.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import LXMF
|
||||
import RNS
|
||||
from LXMF.LXMessage import LXMessage
|
||||
from RNS.Cryptography.Token import Token
|
||||
from RNS.Resource import Resource, ResourceAdvertisement
|
||||
|
||||
|
||||
FIXED_TOKEN_KEY = bytes(range(64))
|
||||
|
||||
|
||||
def fail(msg: str) -> None:
|
||||
print(f"FAIL: {msg}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
class FakeLink:
|
||||
"""Minimum Link surface needed by Resource and Packet construction."""
|
||||
|
||||
def __init__(self, mtu: int = RNS.Reticulum.MTU):
|
||||
self.type = RNS.Destination.LINK
|
||||
self.status = RNS.Link.ACTIVE
|
||||
self.mtu = mtu
|
||||
self.mdu = RNS.Link.MDU
|
||||
self.hash = bytes.fromhex("00112233445566778899aabbccddeeff")
|
||||
self.link_id = self.hash
|
||||
self.rtt = 0.1
|
||||
self.traffic_timeout_factor = 1
|
||||
self.last_outbound = 0
|
||||
self.tx = 0
|
||||
self.txbytes = 0
|
||||
self._token = Token(FIXED_TOKEN_KEY)
|
||||
self.cancelled_incoming = []
|
||||
self.cancelled_outgoing = []
|
||||
|
||||
def encrypt(self, data: bytes) -> bytes:
|
||||
return self._token.encrypt(data)
|
||||
|
||||
def decrypt(self, data: bytes) -> bytes:
|
||||
return self._token.decrypt(data)
|
||||
|
||||
def cancel_incoming_resource(self, resource) -> None:
|
||||
self.cancelled_incoming.append(resource)
|
||||
|
||||
def cancel_outgoing_resource(self, resource) -> None:
|
||||
self.cancelled_outgoing.append(resource)
|
||||
|
||||
def resource_concluded(self, resource) -> None:
|
||||
pass
|
||||
|
||||
|
||||
def verify_default_lxmf_threshold() -> None:
|
||||
expected = RNS.Link.MDU - LXMessage.LXMF_OVERHEAD
|
||||
if LXMessage.LINK_PACKET_MAX_CONTENT != expected:
|
||||
fail(
|
||||
"S10 direct-LXMF threshold formula mismatch: "
|
||||
f"{LXMessage.LINK_PACKET_MAX_CONTENT} != {expected}"
|
||||
)
|
||||
if expected != 319:
|
||||
fail(f"S10 default direct-LXMF threshold changed: got {expected}, want 319")
|
||||
print("PASS S10 default direct-LXMF Resource threshold: content > 319 bytes")
|
||||
|
||||
|
||||
def verify_preparation_and_advertisement() -> None:
|
||||
link = FakeLink()
|
||||
plaintext = (b"resource-verifier-" * 70) + bytes(range(64))
|
||||
resource = Resource(plaintext, link, advertise=False, auto_compress=False)
|
||||
|
||||
stream = b"".join(part.data for part in resource.parts)
|
||||
decrypted = link.decrypt(stream)
|
||||
prefix = decrypted[:Resource.RANDOM_HASH_SIZE]
|
||||
recovered = decrypted[Resource.RANDOM_HASH_SIZE:]
|
||||
|
||||
if recovered != plaintext:
|
||||
fail("S10.2 whole-stream decrypt did not recover original plaintext")
|
||||
if len(prefix) != Resource.RANDOM_HASH_SIZE:
|
||||
fail("S10.2 throwaway prefix has wrong length")
|
||||
|
||||
for index, part in enumerate(resource.parts):
|
||||
if part.context != RNS.Packet.RESOURCE:
|
||||
fail(f"S10.3 part {index} has context {part.context:#x}, want RESOURCE")
|
||||
if part.raw[19:] != part.data:
|
||||
fail(f"S10.6 part {index} was packet-level encrypted or altered")
|
||||
|
||||
expected_hash = RNS.Identity.full_hash(plaintext + resource.random_hash)
|
||||
expected_proof = RNS.Identity.full_hash(plaintext + expected_hash)
|
||||
if resource.hash != expected_hash:
|
||||
fail("S10.2 resource hash formula mismatch")
|
||||
if resource.expected_proof != expected_proof:
|
||||
fail("S10.2 expected proof formula mismatch")
|
||||
|
||||
for index, part in enumerate(resource.parts):
|
||||
expected_map_hash = RNS.Identity.full_hash(part.data + resource.random_hash)[:Resource.MAPHASH_LEN]
|
||||
if part.map_hash != expected_map_hash:
|
||||
fail(f"S10.2 map-hash formula mismatch for part {index}")
|
||||
|
||||
adv = ResourceAdvertisement.unpack(ResourceAdvertisement(resource).pack())
|
||||
if adv.t != len(stream):
|
||||
fail(f"S10.4 ADV t mismatch: {adv.t} != {len(stream)}")
|
||||
if adv.d != len(plaintext):
|
||||
fail(f"S10.4 ADV d mismatch for single segment: {adv.d} != {len(plaintext)}")
|
||||
if adv.n != len(resource.parts):
|
||||
fail(f"S10.4 ADV n mismatch: {adv.n} != {len(resource.parts)}")
|
||||
if adv.h != resource.hash or adv.r != resource.random_hash:
|
||||
fail("S10.4 ADV hash or random-hash field mismatch")
|
||||
if not adv.e or adv.c or adv.s or adv.u or adv.p or adv.x:
|
||||
fail(f"S10.4 ADV flags unexpected: {adv.f:#x}")
|
||||
|
||||
print(
|
||||
"PASS S10.2/S10.4/S10.6 whole-stream encryption, ciphertext slicing, "
|
||||
"hash formulas, and ADV fields"
|
||||
)
|
||||
|
||||
|
||||
def verify_multisegment_total_size() -> None:
|
||||
link = FakeLink()
|
||||
logical_size = Resource.MAX_EFFICIENT_SIZE + 257
|
||||
resource = Resource(b"M" * logical_size, link, advertise=False, auto_compress=False)
|
||||
adv = ResourceAdvertisement.unpack(ResourceAdvertisement(resource).pack())
|
||||
|
||||
if resource.total_segments != 2 or not resource.split:
|
||||
fail("S10.11 expected a two-segment Resource")
|
||||
if adv.d != logical_size:
|
||||
fail(f"S10.4 multi-segment ADV d is {adv.d}, want total logical size {logical_size}")
|
||||
if resource.uncompressed_size >= logical_size:
|
||||
fail("S10.4 first-segment plaintext unexpectedly equals total logical size")
|
||||
|
||||
print(
|
||||
"PASS S10.4/S10.11 multi-segment ADV d is total logical-resource size, "
|
||||
"not current-segment plaintext size"
|
||||
)
|
||||
|
||||
|
||||
def verify_exhausted_request_with_parts() -> None:
|
||||
link = FakeLink()
|
||||
payload_size = ResourceAdvertisement.HASHMAP_MAX_LEN * (link.mtu - RNS.Reticulum.HEADER_MAXSIZE - RNS.Reticulum.IFAC_MIN_SIZE)
|
||||
resource = Resource(b"R" * payload_size, link, advertise=False, auto_compress=False)
|
||||
if len(resource.parts) <= ResourceAdvertisement.HASHMAP_MAX_LEN:
|
||||
fail("S10.7 fixture did not produce a multi-segment hashmap")
|
||||
|
||||
resource.status = Resource.TRANSFERRING
|
||||
resource.adv_sent = time.time()
|
||||
resource.rtt = 0.1
|
||||
|
||||
requested_part = resource.parts[0]
|
||||
last_known_index = ResourceAdvertisement.HASHMAP_MAX_LEN - 1
|
||||
last_known_hash = resource.parts[last_known_index].map_hash
|
||||
request_data = (
|
||||
bytes([Resource.HASHMAP_IS_EXHAUSTED])
|
||||
+ last_known_hash
|
||||
+ resource.hash
|
||||
+ requested_part.map_hash
|
||||
)
|
||||
|
||||
captured: list[RNS.Packet] = []
|
||||
real_outbound = RNS.Transport.outbound
|
||||
|
||||
def fake_outbound(packet):
|
||||
captured.append(packet)
|
||||
return True
|
||||
|
||||
RNS.Transport.outbound = staticmethod(fake_outbound)
|
||||
try:
|
||||
resource.request(request_data)
|
||||
finally:
|
||||
RNS.Transport.outbound = real_outbound
|
||||
|
||||
contexts = [packet.context for packet in captured]
|
||||
if RNS.Packet.RESOURCE not in contexts:
|
||||
fail("S10.7 exhausted RESOURCE_REQ did not fulfil bundled part request")
|
||||
if RNS.Packet.RESOURCE_HMU not in contexts:
|
||||
fail("S10.7 exhausted RESOURCE_REQ did not emit RESOURCE_HMU")
|
||||
|
||||
print("PASS S10.7 exhausted RESOURCE_REQ fulfils bundled parts and emits RESOURCE_HMU")
|
||||
|
||||
|
||||
def verify_receiver_reject_vs_cancel() -> None:
|
||||
link = FakeLink()
|
||||
outgoing = Resource(b"reject-me", link, advertise=False, auto_compress=False)
|
||||
advertisement_plaintext = ResourceAdvertisement(outgoing).pack()
|
||||
|
||||
class AdvertisementPacket:
|
||||
plaintext = advertisement_plaintext
|
||||
link = None
|
||||
|
||||
AdvertisementPacket.link = link
|
||||
|
||||
captured: list[RNS.Packet] = []
|
||||
real_outbound = RNS.Transport.outbound
|
||||
|
||||
def fake_outbound(packet):
|
||||
captured.append(packet)
|
||||
return True
|
||||
|
||||
RNS.Transport.outbound = staticmethod(fake_outbound)
|
||||
try:
|
||||
Resource.reject(AdvertisementPacket)
|
||||
if [packet.context for packet in captured] != [RNS.Packet.RESOURCE_RCL]:
|
||||
fail("S10.9 advertisement rejection did not emit exactly one RESOURCE_RCL")
|
||||
|
||||
captured.clear()
|
||||
incoming = Resource(None, link)
|
||||
incoming.status = Resource.TRANSFERRING
|
||||
incoming.initiator = False
|
||||
incoming.callback = None
|
||||
incoming.cancel()
|
||||
if captured:
|
||||
fail("S10.9 ordinary receiver-side cancel unexpectedly emitted a packet")
|
||||
if incoming not in link.cancelled_incoming:
|
||||
fail("S10.9 ordinary receiver-side cancel did not remove incoming Resource")
|
||||
finally:
|
||||
RNS.Transport.outbound = real_outbound
|
||||
|
||||
print("PASS S10.9 RESOURCE_RCL rejects advertisements; ordinary receiver cancel is local-only")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
print(f"verify_resource.py against RNS {RNS.__version__} / LXMF {LXMF.__version__}")
|
||||
verify_default_lxmf_threshold()
|
||||
verify_preparation_and_advertisement()
|
||||
verify_multisegment_total_size()
|
||||
verify_exhausted_request_with_parts()
|
||||
verify_receiver_reject_vs_cancel()
|
||||
print("ALL PASS")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Loading…
Add table
Add a link
Reference in a new issue