diff --git a/SPEC.md b/SPEC.md index 6d206e9..e6e4eb0 100644 --- a/SPEC.md +++ b/SPEC.md @@ -3769,8 +3769,9 @@ See [`test-vectors/`](test-vectors/). Currently populated: - **`announces.json`** — signed announce packets, with and without ratchet material. Verified by `tools/verify_announce_roundtrip.py`; regenerated by `tools/regen_announces.py`. Covers SPEC.md §4.1, §4.2, and §4.5. - **`lxmf.json`** — deterministic opportunistic LXMF plaintext and Token ciphertext vectors. Verified by `tools/verify_lxmf_opportunistic.py`; regenerated by `tools/regen_lxmf.py`. Covers SPEC.md §3 and §5. - **`links.json`** — Link handshake and LRRTT vectors, including LINKREQUEST, LRPROOF, derived keys, and the activation packet. Verified by `tools/verify_link_handshake.py` and `tools/verify_link_lrrtt.py`; regenerated by `tools/regen_links.py`. Covers SPEC.md §6.1-§6.4 and §6.6. +- **`resources.json`** — deterministic multi-part Resource ciphertext, part packets, hashmap, advertisement, and proof body. Verified by `tools/verify_resource.py`; regenerated by `tools/regen_resources.py`. Covers SPEC.md §10.2, §10.4, §10.8, and §10.12. -Remaining vector work should focus on negative/rejection cases and link-delivered LXMF bodies rather than the original bootstrap categories. +Remaining vector work should focus on link-delivered LXMF bodies and broader negative/rejection cases rather than the original bootstrap categories. An implementation that round-trips every test vector — both directions — should be wire-compatible with upstream Reticulum and LXMF for the covered operations. diff --git a/agent.md b/agent.md index 693d647..aff8b1b 100644 --- a/agent.md +++ b/agent.md @@ -167,7 +167,7 @@ Initial confidence assessment (subjective, not authoritative — re-do this audi | §7.6 `TCPServerInterface.OUT` override | Source-cited; matches behavior observed in the mobile-app's local-transport experiments. | | §8 KISS / HDLC framing | High — both work in production on the reference clients | | §9.1–§9.8 Implementation gotchas | Each was a real bug that bit a real implementation. High confidence each is real; some lack formal test scripts. | -| §10 Resource fragmentation | Source-cited from `RNS/Resource.py` against RNS 1.2.4; not yet runtime-verified in this repo's `tools/`. | +| §10 Resource fragmentation | Source-audited against RNS 1.2.4 and runtime-verified by `tools/verify_resource.py`, including deterministic vectors, receiver assembly/proof, multi-segment sizing, and negative cases. | | §11 Test vectors | Historical bootstrap item. `test-vectors/` is now populated with identities, announces, opportunistic LXMF, and Link vectors. Future work should add negative vectors and link-delivered LXMF coverage. | | §12 Source map | High | diff --git a/audits/resource-tier1-rns-1.2.4.md b/audits/resource-tier1-rns-1.2.4.md index 61924b4..90a2e2d 100644 --- a/audits/resource-tier1-rns-1.2.4.md +++ b/audits/resource-tier1-rns-1.2.4.md @@ -10,9 +10,9 @@ Evidence baseline: `RNS/Link.py`, and `LXMF/LXMessage.py` - Audit date: 2026-06-08 -This began as a Tier 1 source-analysis report. The focused Tier 2 checks now -live in `tools/verify_resource.py`; the confirmed F1-F6 corrections were -promoted into `SPEC.md` §10 on 2026-06-08. +This began as a Tier 1 source-analysis report. The completed Tier 2 checks +live in `tools/verify_resource.py` and `test-vectors/resources.json`; the +confirmed F1-F6 corrections were promoted into `SPEC.md` §10 on 2026-06-08. ## Confirmed Core Model @@ -117,8 +117,8 @@ later-version note only when documenting an actual version change. ## Tier 2 Verifier Scope -The first focused verifier is implemented in `tools/verify_resource.py` and -avoids a live threaded transfer. It currently covers items 1-6 below: +The focused verifier is implemented in `tools/verify_resource.py` and avoids +a live threaded transfer. It covers: 1. Construct a Resource with a deterministic fake Link encryption key, fixed throwaway prefix, and fixed advertisement `r`. @@ -131,11 +131,11 @@ avoids a live threaded transfer. It currently covers items 1-6 below: including simultaneous part fulfilment and RESOURCE_HMU generation. 7. Verify receiver assembly strips the throwaway prefix, decrypts once, validates the hash, and emits the expected RESOURCE_PRF bytes. -8. Add a multi-segment fixture proving `d` remains total logical size while +8. Verify a multi-segment fixture proving `d` remains total logical size while `t`, `n`, `h`, and `r` describe the current segment. -9. Add negative cases: malformed ADV, wrong `r`, corrupt part, invalid HMU +9. Verify negative cases: malformed ADV, wrong `r`, corrupt part, invalid HMU boundary, and oversized decompression. -Remaining Tier 2 work is a deterministic `test-vectors/resources.json` plus -items 7-9. The confirmed first claim set has already been promoted into -`SPEC.md` and the Resource flow documents. +The deterministic fixture is regenerated by `tools/regen_resources.py`. +Tier 2 Resource work is complete for this audit scope; confirmed claims have +been promoted into `SPEC.md` and the Resource flow documents. diff --git a/test-vectors/README.md b/test-vectors/README.md index 1309bd2..d1202f6 100644 --- a/test-vectors/README.md +++ b/test-vectors/README.md @@ -10,8 +10,9 @@ Populated against RNS 1.2.4 / LXMF 0.9.7: - ✅ `announces.json` — two announce vectors (no-ratchet + with-ratchet) signed by Alice (regenerator: `../tools/regen_announces.py`, verifier: `../tools/verify_announce_roundtrip.py`). - ✅ `lxmf.json` — two opportunistic-LXMF vectors Alice → Bob (regenerator: `../tools/regen_lxmf.py`, verifier: `../tools/verify_lxmf_opportunistic.py`). - ✅ `links.json` — full Link handshake vector (LINKREQUEST + LRPROOF + derived session key) Alice → Bob, plus an LRRTT packet (§6.4.2) emitted from the initiator with pinned IV and `rtt_seconds = 0.05` (regenerator: `../tools/regen_links.py`, verifiers: `../tools/verify_link_handshake.py`, `../tools/verify_link_lrrtt.py`). +- ✅ `resources.json` — deterministic multi-part Resource ciphertext, part packets, hashmap, advertisement, and proof body (regenerator: `../tools/regen_resources.py`, verifier: `../tools/verify_resource.py`). -All four files are byte-deterministic across runs: regenerators pin every random source (ephemeral keys, IVs, `random_hash` prefix + timestamp, LXMF timestamp) so the output is reproducible against a fixed upstream RNS / LXMF version. +All five files are byte-deterministic across runs: regenerators pin every random source (ephemeral keys, IVs, `random_hash` values, timestamps) so the output is reproducible against a fixed upstream RNS / LXMF version. See [`../agent.md`](../agent.md) §3 and [`../todo.md`](../todo.md) for the evidence model and remaining task list. @@ -23,6 +24,7 @@ Each vector lives in a per-domain JSON file, e.g.: - `announces.json` — full hex of a signed announce packet, plus the inputs that produced it (display_name, ratchetPub, etc.) - `lxmf.json` — sender + recipient identity, plaintext, expected ciphertext bytes - `links.json` — LINKREQUEST + LRPROOF + derived session keys +- `resources.json` — Resource plaintext, encrypted stream, parts, hashmap, advertisement, and proof Each entry should include: @@ -48,5 +50,6 @@ For the spec to claim "an implementation that passes all test vectors interopera 4. **Opportunistic LXMF** — full plaintext → ciphertext → plaintext round-trip, signature valid both ways. 5. **Link handshake** — LINKREQUEST built by client A, LRPROOF computed by upstream as B, both arrive at the same `link_id` and session keys. 6. **Link-delivered LXMF** — body packed by client, decrypted + parsed by upstream. +7. **Resource transfer** — encrypt once, split into parts, validate ADV/hashmap, assemble, and emit the expected proof. A separate vector set for FAILURE cases is also useful: malformed announces, expired ratchets, mismatched signatures. An implementation should reject those as a regression-prevention measure. diff --git a/test-vectors/resources.json b/test-vectors/resources.json new file mode 100644 index 0000000..dea0512 --- /dev/null +++ b/test-vectors/resources.json @@ -0,0 +1,52 @@ +{ + "_about": "Deterministic Resource transfer vector over a fixed Link Token key. The Resource is uncompressed and spans multiple RESOURCE packets. A clean-room implementation should encrypt `throwaway_prefix || plaintext` once with the recorded Token key and IV, slice the ciphertext at the Resource SDU, reproduce every part and map hash, pack the advertisement, then assemble and emit the recorded PRF body.", + "vectors": [ + { + "label": "fixed_link_uncompressed_multi_part", + "inputs": { + "plaintext_hex": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f606162636465666768696a6b6c6d6e6f707172737475767778797a7b7c7d7e7f808182838485868788898a8b8c8d8e8f909192939495969798999a9b9c9d9e9fa0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebfc0c1c2c3c4c5c6c7c8c9cacbcccdcecfd0d1d2d3d4d5d6d7d8d9dadbdcdddedfe0e1e2e3e4e5e6e7e8e9eaebecedeeeff0f1f2f3f4f5f6f7f8f9fafbfcfdfeff", + "token_key_hex": "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f", + "token_iv_hex": "404142434445464748494a4b4c4d4e4f", + "throwaway_prefix_hex": "10111213", + "random_hash_hex": "20212223", + "link_hash_hex": "00112233445566778899aabbccddeeff", + "mtu": 500, + "auto_compress": false + }, + "expected": { + "encrypted_stream_hex": "404142434445464748494a4b4c4d4e4f2e9d382a37451544df61eee81f38350c4883c306763ae9491e392fc51bb2579c295964e92e36cd787d899e959ae7c3bd83607c4a45ebabd30a4de196a2f7ba9d77aabd00ca89fc711c072fbef55dc346c51754955bf4616427b2917de04247985af831059d14a52654ed50a0b01e99daa097f369609cfd31957dc8d2737e603f954e7e0ea70a65f7fee60c9b8b8f2482cb00469a155a9e0601111de46004ae4cde1fc14ac9f3be5b37a8308f47a7a7524e9638af44100573cb35a3d55a436c6080ccdcac5bcce07fae741f6049b4f06c011abba11f4647917ecbd7da7071a7936606bfbfd69a4c264060e7e1a2e093b704c76e0b09dc6200e761fd711489fda28679a375fad63df1e732f66ac47fc2650a6db3ee818b201e93b8b7325d375c0dfd850db44215c5fdc4189d199a2758e067c8d2ce29737acc4b81e9d22604b1a3f6d6315b3003b11dcbad4b761fbc4b43a9e3446c6083a4a32293458c843aa57fd829b21afaa600fc438d292a00c1bc651f95291586414bc4e67fa2740da4def4ed2caaac74f39d8cecf00e957f848c1adea9bf78556e55d17c94fd550a99d328bacf6704d7db090a9759aabac010322f539019b28cea0238992638a358582386065fa79a46947f5028d9e884d9fc98304a43f318b13b2a9ac14ddbf7d68d898be03d7881bf347e5cdc209ab155d23cbe922430c71ae14902642310c50d7d9617f462b4674517abf1430b6bc7763299c6556e52187d6950f0ff71ca6a1dcffbce2415f074492dc511c6c489b1ea8ed5d34c46d8ce32cffb71567411b7f8da3a84b53b16bc4384bfbe3238e5487b18f4faf3b09ee83e53f537c699be3f9167fc27a7df92bea5ec4a20646100016cb6728a7a94d51184acbcfffab0b1ccf7f6ee61fc34df1dbfc98cc801e141111a88c5713748d929cbbd14d69613bacf12d60df4f4da0799f7b9b6a1200ad84732789ff629f63aef6e5ffd79741e6d9160a1350882de9a1c76aae13a2c70ed972200af3f4078837882a06501b514081cb465c6b69661c9a82459a59086d6bbda247d4eac9656d413fe88558ae6a4e6d94ed1d2bb87bfc75b68256291d7208bb5e890d30f4c53226e222288de73106b1d4ba3fb01addfb7b0b1b9fed386cfa5afdb8789f22e7cfe7f0b39c8c797d0b70fd33a89293df048820c86b80cfd6f8ed829483d9fadbb05b8b6b4c8680931d74ffc263a23ab7845fe92ab850638fae9c267a623e78afc26055864e0504a951451933f60a357e17d8471653dca6cc97b3ef92c1715c959c4df40bdacf257b106d1fc6e635c82208f4235621458afd2ad23ce1e799f596df9be514f25cfff1f65bebf46ff52c2f43cf7c2edae2873e70a938cb417aea136c2a3b9521863eb0ae97e7093b99ad9059b00ba08463ad3b68bf2a91b053c6119117aee992020cc288f0f1ab9ae3ee3618f83d03378a62282bc9fa93a47dfa5426114551bbf610b5549ba57d548001e8b67bf09a40878799321832ef266b588a4226854366f22822fd36a79443c17f52bfc88341405eef4b6f821ef255270e14a21ae11570f205b550268c69e9ea071f21cd599774ef2e0fe20ec7ab2146d36237b922b7e59b8b78d621872ff682d703829e627b04f23315c93d5304e991e584d29facf6704c4d444418b3af4c6514cc9aa63e396e976713077755f721ebb022013bb75b071308a71ac578dd5c124a0af521c7404f166c26a63116b377928cc5d971d230f1bdf445202869d5af5304fd5e743165c1d65f86cf46247898b43d6cbf65f2be6217421c6a6ba95df7fd0fda278855fcea67916d710511da70eaa8cfb0d630b21e725f5cb304d13d5ca39440edefecc3f394482c08399a6d62cf8", + "resource_hash_hex": "a60281a93eccd1a9473ee0e72e3224bcf76792857bc003f1374facb56b0f3991", + "expected_proof_hex": "1c0e68442e09a54939291962b45ff7a10c9871aee1c7a769c338ba483bd1a256", + "resource_prf_body_hex": "a60281a93eccd1a9473ee0e72e3224bcf76792857bc003f1374facb56b0f39911c0e68442e09a54939291962b45ff7a10c9871aee1c7a769c338ba483bd1a256", + "hashmap_hex": "94abb0d50fcf34cdd9769774", + "advertisement_plaintext_hex": "8ba174cd0540a164cd0500a16e03a168c420a60281a93eccd1a9473ee0e72e3224bcf76792857bc003f1374facb56b0f3991a172c40420212223a16fc420a60281a93eccd1a9473ee0e72e3224bcf76792857bc003f1374facb56b0f3991a16901a16c01a171c0a16601a16dc40c94abb0d50fcf34cdd9769774", + "parts": [ + { + "body_hex": "404142434445464748494a4b4c4d4e4f2e9d382a37451544df61eee81f38350c4883c306763ae9491e392fc51bb2579c295964e92e36cd787d899e959ae7c3bd83607c4a45ebabd30a4de196a2f7ba9d77aabd00ca89fc711c072fbef55dc346c51754955bf4616427b2917de04247985af831059d14a52654ed50a0b01e99daa097f369609cfd31957dc8d2737e603f954e7e0ea70a65f7fee60c9b8b8f2482cb00469a155a9e0601111de46004ae4cde1fc14ac9f3be5b37a8308f47a7a7524e9638af44100573cb35a3d55a436c6080ccdcac5bcce07fae741f6049b4f06c011abba11f4647917ecbd7da7071a7936606bfbfd69a4c264060e7e1a2e093b704c76e0b09dc6200e761fd711489fda28679a375fad63df1e732f66ac47fc2650a6db3ee818b201e93b8b7325d375c0dfd850db44215c5fdc4189d199a2758e067c8d2ce29737acc4b81e9d22604b1a3f6d6315b3003b11dcbad4b761fbc4b43a9e3446c6083a4a32293458c843aa57fd829b21afaa600fc438d292a00c1bc651f95291586414bc4e67fa2740da4def4ed2caaac74f39d8cecf00e957f848c1adea9bf78556e55d17c94fd550a99d328bacf6704d7db090a9759aabac010322f539019b28cea0238992638a358582386", + "map_hash_hex": "94abb0d5", + "raw_hex": "0c0000112233445566778899aabbccddeeff01404142434445464748494a4b4c4d4e4f2e9d382a37451544df61eee81f38350c4883c306763ae9491e392fc51bb2579c295964e92e36cd787d899e959ae7c3bd83607c4a45ebabd30a4de196a2f7ba9d77aabd00ca89fc711c072fbef55dc346c51754955bf4616427b2917de04247985af831059d14a52654ed50a0b01e99daa097f369609cfd31957dc8d2737e603f954e7e0ea70a65f7fee60c9b8b8f2482cb00469a155a9e0601111de46004ae4cde1fc14ac9f3be5b37a8308f47a7a7524e9638af44100573cb35a3d55a436c6080ccdcac5bcce07fae741f6049b4f06c011abba11f4647917ecbd7da7071a7936606bfbfd69a4c264060e7e1a2e093b704c76e0b09dc6200e761fd711489fda28679a375fad63df1e732f66ac47fc2650a6db3ee818b201e93b8b7325d375c0dfd850db44215c5fdc4189d199a2758e067c8d2ce29737acc4b81e9d22604b1a3f6d6315b3003b11dcbad4b761fbc4b43a9e3446c6083a4a32293458c843aa57fd829b21afaa600fc438d292a00c1bc651f95291586414bc4e67fa2740da4def4ed2caaac74f39d8cecf00e957f848c1adea9bf78556e55d17c94fd550a99d328bacf6704d7db090a9759aabac010322f539019b28cea0238992638a358582386" + }, + { + "body_hex": "065fa79a46947f5028d9e884d9fc98304a43f318b13b2a9ac14ddbf7d68d898be03d7881bf347e5cdc209ab155d23cbe922430c71ae14902642310c50d7d9617f462b4674517abf1430b6bc7763299c6556e52187d6950f0ff71ca6a1dcffbce2415f074492dc511c6c489b1ea8ed5d34c46d8ce32cffb71567411b7f8da3a84b53b16bc4384bfbe3238e5487b18f4faf3b09ee83e53f537c699be3f9167fc27a7df92bea5ec4a20646100016cb6728a7a94d51184acbcfffab0b1ccf7f6ee61fc34df1dbfc98cc801e141111a88c5713748d929cbbd14d69613bacf12d60df4f4da0799f7b9b6a1200ad84732789ff629f63aef6e5ffd79741e6d9160a1350882de9a1c76aae13a2c70ed972200af3f4078837882a06501b514081cb465c6b69661c9a82459a59086d6bbda247d4eac9656d413fe88558ae6a4e6d94ed1d2bb87bfc75b68256291d7208bb5e890d30f4c53226e222288de73106b1d4ba3fb01addfb7b0b1b9fed386cfa5afdb8789f22e7cfe7f0b39c8c797d0b70fd33a89293df048820c86b80cfd6f8ed829483d9fadbb05b8b6b4c8680931d74ffc263a23ab7845fe92ab850638fae9c267a623e78afc26055864e0504a951451933f60a357e17d8471653dca6cc97b3ef92c1715", + "map_hash_hex": "0fcf34cd", + "raw_hex": "0c0000112233445566778899aabbccddeeff01065fa79a46947f5028d9e884d9fc98304a43f318b13b2a9ac14ddbf7d68d898be03d7881bf347e5cdc209ab155d23cbe922430c71ae14902642310c50d7d9617f462b4674517abf1430b6bc7763299c6556e52187d6950f0ff71ca6a1dcffbce2415f074492dc511c6c489b1ea8ed5d34c46d8ce32cffb71567411b7f8da3a84b53b16bc4384bfbe3238e5487b18f4faf3b09ee83e53f537c699be3f9167fc27a7df92bea5ec4a20646100016cb6728a7a94d51184acbcfffab0b1ccf7f6ee61fc34df1dbfc98cc801e141111a88c5713748d929cbbd14d69613bacf12d60df4f4da0799f7b9b6a1200ad84732789ff629f63aef6e5ffd79741e6d9160a1350882de9a1c76aae13a2c70ed972200af3f4078837882a06501b514081cb465c6b69661c9a82459a59086d6bbda247d4eac9656d413fe88558ae6a4e6d94ed1d2bb87bfc75b68256291d7208bb5e890d30f4c53226e222288de73106b1d4ba3fb01addfb7b0b1b9fed386cfa5afdb8789f22e7cfe7f0b39c8c797d0b70fd33a89293df048820c86b80cfd6f8ed829483d9fadbb05b8b6b4c8680931d74ffc263a23ab7845fe92ab850638fae9c267a623e78afc26055864e0504a951451933f60a357e17d8471653dca6cc97b3ef92c1715" + }, + { + "body_hex": "c959c4df40bdacf257b106d1fc6e635c82208f4235621458afd2ad23ce1e799f596df9be514f25cfff1f65bebf46ff52c2f43cf7c2edae2873e70a938cb417aea136c2a3b9521863eb0ae97e7093b99ad9059b00ba08463ad3b68bf2a91b053c6119117aee992020cc288f0f1ab9ae3ee3618f83d03378a62282bc9fa93a47dfa5426114551bbf610b5549ba57d548001e8b67bf09a40878799321832ef266b588a4226854366f22822fd36a79443c17f52bfc88341405eef4b6f821ef255270e14a21ae11570f205b550268c69e9ea071f21cd599774ef2e0fe20ec7ab2146d36237b922b7e59b8b78d621872ff682d703829e627b04f23315c93d5304e991e584d29facf6704c4d444418b3af4c6514cc9aa63e396e976713077755f721ebb022013bb75b071308a71ac578dd5c124a0af521c7404f166c26a63116b377928cc5d971d230f1bdf445202869d5af5304fd5e743165c1d65f86cf46247898b43d6cbf65f2be6217421c6a6ba95df7fd0fda278855fcea67916d710511da70eaa8cfb0d630b21e725f5cb304d13d5ca39440edefecc3f394482c08399a6d62cf8", + "map_hash_hex": "d9769774", + "raw_hex": "0c0000112233445566778899aabbccddeeff01c959c4df40bdacf257b106d1fc6e635c82208f4235621458afd2ad23ce1e799f596df9be514f25cfff1f65bebf46ff52c2f43cf7c2edae2873e70a938cb417aea136c2a3b9521863eb0ae97e7093b99ad9059b00ba08463ad3b68bf2a91b053c6119117aee992020cc288f0f1ab9ae3ee3618f83d03378a62282bc9fa93a47dfa5426114551bbf610b5549ba57d548001e8b67bf09a40878799321832ef266b588a4226854366f22822fd36a79443c17f52bfc88341405eef4b6f821ef255270e14a21ae11570f205b550268c69e9ea071f21cd599774ef2e0fe20ec7ab2146d36237b922b7e59b8b78d621872ff682d703829e627b04f23315c93d5304e991e584d29facf6704c4d444418b3af4c6514cc9aa63e396e976713077755f721ebb022013bb75b071308a71ac578dd5c124a0af521c7404f166c26a63116b377928cc5d971d230f1bdf445202869d5af5304fd5e743165c1d65f86cf46247898b43d6cbf65f2be6217421c6a6ba95df7fd0fda278855fcea67916d710511da70eaa8cfb0d630b21e725f5cb304d13d5ca39440edefecc3f394482c08399a6d62cf8" + } + ] + }, + "rns_version_at_generation": "1.2.4", + "generator_script": "tools/regen_resources.py", + "verifies_spec_sections": [ + "10.2", + "10.3", + "10.4", + "10.8", + "10.12" + ] + } + ] +} diff --git a/todo.md b/todo.md index 93903b7..03ea9c4 100644 --- a/todo.md +++ b/todo.md @@ -21,12 +21,13 @@ Outstanding work for the spec repo. ## Test infrastructure -- [ ] **Deterministic Resource vectors and negative cases.** The Tier 1 audit +- [x] **Deterministic Resource vectors and negative cases.** The Tier 1 audit is recorded in `audits/resource-tier1-rns-1.2.4.md`; - `tools/verify_resource.py` now runtime-locks the first focused claim - set, and confirmed corrections are promoted into SPEC.md §10. Add - `test-vectors/resources.json` plus malformed ADV, wrong-`r`, corrupt - part, invalid-HMU-boundary, and oversized-decompression rejection cases. + `tools/verify_resource.py` runtime-locks sender and receiver behavior, + proof validation, multi-segment sizing, and malformed ADV, wrong-`r`, + corrupt-part, invalid-HMU-boundary, and oversized-decompression + rejection cases. `tools/regen_resources.py` regenerates the deterministic + `test-vectors/resources.json`. - [x] **Bootstrap `test-vectors/identities.json`** — Alice + Bob identities populated against RNS 1.2.0. Regenerator at `tools/regen_identities.py`. diff --git a/tools/README.md b/tools/README.md index 4235c77..2c8090c 100644 --- a/tools/README.md +++ b/tools/README.md @@ -56,10 +56,11 @@ Populated against RNS 1.2.4 / LXMF 0.9.7: | `verify_msgpack_quirk.py` | §9.3 — encoding name as bytes vs str affects upstream parsing | ✅ | | `verify_stamps.py` | §5.7 — workblock determinism, PoW stamp search/validate, ticket shortcut | ✅ | | `verify_ratchet_dedup.py` | §7.3 / §4.5 step 6.3 — confirms replay defence is keyed on `random_blob`, NOT on `(dest_hash, ratchet_pub)` | ✅ | -| `verify_resource.py` | §10.2, §10.4, §10.6, §10.7, §10.9, §10.11 — whole-stream encryption/slicing, hashes, ADV fields, exhausted REQ behavior, rejection/cancel distinction, multi-segment total size | ✅ | +| `verify_resource.py` | §10.2, §10.4, §10.6-§10.9, §10.11, §10.12 — vectors, whole-stream encryption/slicing, receiver assembly/proof, control behavior, multi-segment size, and negative cases | ✅ | | `regen_identities.py` | regenerates `test-vectors/identities.json` | ✅ | | `regen_announces.py` | regenerates `test-vectors/announces.json` (deterministic announce wire bytes, with and without ratchet) | ✅ | | `regen_lxmf.py` | regenerates `test-vectors/lxmf.json` (deterministic opportunistic-LXMF plaintext + Token ciphertext) | ✅ | | `regen_links.py` | regenerates `test-vectors/links.json` (deterministic LINKREQUEST + LRPROOF + derived session key) | ✅ | +| `regen_resources.py` | regenerates `test-vectors/resources.json` (deterministic Resource ciphertext, parts, ADV, and PRF body) | ✅ | See [`../agent.md`](../agent.md) §3 and [`../todo.md`](../todo.md) for the evidence model and remaining priority order. diff --git a/tools/regen_resources.py b/tools/regen_resources.py new file mode 100644 index 0000000..68153f5 --- /dev/null +++ b/tools/regen_resources.py @@ -0,0 +1,146 @@ +""" +Regenerator for test-vectors/resources.json. + +Builds a deterministic multi-part Resource over a fixed fake Link. The link +Token key, Token IV, throwaway prefix, and Resource random-hash salt are pinned +so every ciphertext byte, part packet, map hash, advertisement, and proof is +reproducible against the pinned RNS version. + +Run from repo root: + + python tools/regen_resources.py + +Updates `test-vectors/resources.json` in place. Exit 0 on success. +""" + +from __future__ import annotations + +import json +import os +import sys + +import RNS +from RNS.Cryptography.Token import Token +from RNS.Resource import Resource, ResourceAdvertisement + + +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +OUT_PATH = os.path.join(REPO_ROOT, "test-vectors", "resources.json") + +FIXED_TOKEN_KEY = bytes(range(64)) +FIXED_TOKEN_IV = bytes.fromhex("404142434445464748494a4b4c4d4e4f") +FIXED_PREFIX = bytes.fromhex("10111213") +FIXED_RANDOM_HASH = bytes.fromhex("20212223") +PLAINTEXT = bytes(range(256)) * 5 + + +class FakeLink: + """Minimum deterministic Link surface needed by Resource construction.""" + + def __init__(self): + self.type = RNS.Destination.LINK + self.status = RNS.Link.ACTIVE + self.mtu = RNS.Reticulum.MTU + self.mdu = RNS.Link.MDU + self.hash = bytes.fromhex("00112233445566778899aabbccddeeff") + self.link_id = self.hash + self.rtt = 0.1 + self.traffic_timeout_factor = 1 + self.last_outbound = 0 + self.tx = 0 + self.txbytes = 0 + self._token = Token(FIXED_TOKEN_KEY) + + def encrypt(self, data: bytes) -> bytes: + return self._token.encrypt(data) + + def decrypt(self, data: bytes) -> bytes: + return self._token.decrypt(data) + + +def main() -> None: + print(f"regen_resources.py against RNS {RNS.__version__}") + token_mod = sys.modules["RNS.Cryptography.Token"] + real_urandom = token_mod.os.urandom + real_get_random_hash = RNS.Identity.get_random_hash + random_hashes = [ + FIXED_PREFIX + bytes(28), + FIXED_RANDOM_HASH + bytes(28), + ] + + def fixed_urandom(length: int) -> bytes: + if length == 16: + return FIXED_TOKEN_IV + return real_urandom(length) + + def fixed_random_hash() -> bytes: + if not random_hashes: + raise RuntimeError("unexpected additional Resource random-hash request") + return random_hashes.pop(0) + + token_mod.os.urandom = fixed_urandom + RNS.Identity.get_random_hash = staticmethod(fixed_random_hash) + try: + link = FakeLink() + resource = Resource(PLAINTEXT, link, advertise=False, auto_compress=False) + finally: + token_mod.os.urandom = real_urandom + RNS.Identity.get_random_hash = staticmethod(real_get_random_hash) + + encrypted_stream = b"".join(part.data for part in resource.parts) + advertisement = ResourceAdvertisement(resource).pack() + proof = resource.hash + resource.expected_proof + + vector = { + "label": "fixed_link_uncompressed_multi_part", + "inputs": { + "plaintext_hex": PLAINTEXT.hex(), + "token_key_hex": FIXED_TOKEN_KEY.hex(), + "token_iv_hex": FIXED_TOKEN_IV.hex(), + "throwaway_prefix_hex": FIXED_PREFIX.hex(), + "random_hash_hex": FIXED_RANDOM_HASH.hex(), + "link_hash_hex": link.hash.hex(), + "mtu": link.mtu, + "auto_compress": False, + }, + "expected": { + "encrypted_stream_hex": encrypted_stream.hex(), + "resource_hash_hex": resource.hash.hex(), + "expected_proof_hex": resource.expected_proof.hex(), + "resource_prf_body_hex": proof.hex(), + "hashmap_hex": resource.hashmap.hex(), + "advertisement_plaintext_hex": advertisement.hex(), + "parts": [ + { + "body_hex": part.data.hex(), + "map_hash_hex": part.map_hash.hex(), + "raw_hex": part.raw.hex(), + } + for part in resource.parts + ], + }, + "rns_version_at_generation": RNS.__version__, + "generator_script": "tools/regen_resources.py", + "verifies_spec_sections": ["10.2", "10.3", "10.4", "10.8", "10.12"], + } + payload = { + "_about": ( + "Deterministic Resource transfer vector over a fixed Link Token key. " + "The Resource is uncompressed and spans multiple RESOURCE packets. " + "A clean-room implementation should encrypt `throwaway_prefix || " + "plaintext` once with the recorded Token key and IV, slice the " + "ciphertext at the Resource SDU, reproduce every part and map hash, " + "pack the advertisement, then assemble and emit the recorded PRF body." + ), + "vectors": [vector], + } + with open(OUT_PATH, "w", encoding="utf-8", newline="\n") as output: + json.dump(payload, output, indent=2, sort_keys=False) + output.write("\n") + + print(f"Wrote {OUT_PATH} with 1 vector") + print("ALL PASS") + + +if __name__ == "__main__": + main() diff --git a/tools/verify_resource.py b/tools/verify_resource.py index 70db4a3..5c572bb 100644 --- a/tools/verify_resource.py +++ b/tools/verify_resource.py @@ -14,14 +14,21 @@ Link and verifies: requested parts and emits RESOURCE_HMU. 7. RESOURCE_RCL is emitted for advertisement rejection, while an ordinary receiver-side cancel is local-only. + 8. Deterministic Resource vector bytes round-trip. + 9. Receiver assembly decrypts once, strips the prefix, handles metadata, + validates integrity, and emits RESOURCE_PRF. + 10. Malformed ADV, wrong-r, corrupt-part, invalid-HMU-boundary, + oversized-decompression, and incorrect-proof cases are rejected. Exit code 0 on PASS, non-zero on FAIL. """ from __future__ import annotations +import json import os import sys +import tempfile import time import LXMF @@ -32,6 +39,8 @@ from RNS.Resource import Resource, ResourceAdvertisement FIXED_TOKEN_KEY = bytes(range(64)) +REPO_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +VECTORS_PATH = os.path.join(REPO_ROOT, "test-vectors", "resources.json") def fail(msg: str) -> None: @@ -57,6 +66,8 @@ class FakeLink: self._token = Token(FIXED_TOKEN_KEY) self.cancelled_incoming = [] self.cancelled_outgoing = [] + self.concluded = [] + self.torn_down = False def encrypt(self, data: bytes) -> bytes: return self._token.encrypt(data) @@ -71,7 +82,10 @@ class FakeLink: self.cancelled_outgoing.append(resource) def resource_concluded(self, resource) -> None: - pass + self.concluded.append(resource) + + def teardown(self) -> None: + self.torn_down = True def verify_default_lxmf_threshold() -> None: @@ -239,6 +253,190 @@ def verify_receiver_reject_vs_cancel() -> None: print("PASS S10.9 RESOURCE_RCL rejects advertisements; ordinary receiver cancel is local-only") +def load_vector() -> dict: + try: + with open(VECTORS_PATH, "r", encoding="utf-8") as vector_file: + return json.load(vector_file)["vectors"][0] + except (OSError, KeyError, IndexError, json.JSONDecodeError) as exc: + fail(f"S10 Resource vector could not be loaded: {exc}") + + +def verify_deterministic_vector() -> None: + vector = load_vector() + inputs = vector["inputs"] + expected = vector["expected"] + plaintext = bytes.fromhex(inputs["plaintext_hex"]) + prefix = bytes.fromhex(inputs["throwaway_prefix_hex"]) + random_hash = bytes.fromhex(inputs["random_hash_hex"]) + encrypted_stream = bytes.fromhex(expected["encrypted_stream_hex"]) + + token = Token(bytes.fromhex(inputs["token_key_hex"])) + if token.decrypt(encrypted_stream) != prefix + plaintext: + fail("S10 vector ciphertext did not decrypt to prefix || plaintext") + + resource_hash = RNS.Identity.full_hash(plaintext + random_hash) + proof = RNS.Identity.full_hash(plaintext + resource_hash) + if resource_hash.hex() != expected["resource_hash_hex"]: + fail("S10 vector resource hash mismatch") + if proof.hex() != expected["expected_proof_hex"]: + fail("S10 vector expected proof mismatch") + if (resource_hash + proof).hex() != expected["resource_prf_body_hex"]: + fail("S10 vector RESOURCE_PRF body mismatch") + + part_bodies = [bytes.fromhex(part["body_hex"]) for part in expected["parts"]] + if b"".join(part_bodies) != encrypted_stream: + fail("S10 vector part bodies do not concatenate to encrypted stream") + map_hashes = b"".join( + RNS.Identity.full_hash(body + random_hash)[:Resource.MAPHASH_LEN] + for body in part_bodies + ) + if map_hashes.hex() != expected["hashmap_hex"]: + fail("S10 vector hashmap mismatch") + + adv = ResourceAdvertisement.unpack(bytes.fromhex(expected["advertisement_plaintext_hex"])) + if adv.h != resource_hash or adv.r != random_hash or adv.m != map_hashes: + fail("S10 vector advertisement hash, random-hash, or hashmap mismatch") + if adv.n != len(part_bodies) or adv.d != len(plaintext): + fail("S10 vector advertisement part count or logical size mismatch") + + print("PASS S10.2/S10.4/S10.8 deterministic Resource vector round-trip") + + +def make_incoming(outgoing: Resource, link: FakeLink, directory: str, parts=None) -> Resource: + incoming = Resource(None, link) + incoming.status = Resource.TRANSFERRING + incoming.initiator = False + incoming.callback = None + incoming.parts = parts if parts is not None else [part.data for part in outgoing.parts] + incoming.encrypted = outgoing.encrypted + incoming.compressed = outgoing.compressed + incoming.random_hash = outgoing.random_hash + incoming.hash = outgoing.hash + incoming.has_metadata = outgoing.has_metadata + incoming.segment_index = outgoing.segment_index + incoming.total_segments = outgoing.total_segments + incoming.storagepath = os.path.join(directory, "resource.data") + incoming.meta_storagepath = os.path.join(directory, "resource.meta") + incoming.max_decompressed_size = Resource.AUTO_COMPRESS_MAX_SIZE + incoming.data = None + + class AdvertisementPacket: + pass + + AdvertisementPacket.link = link + AdvertisementPacket.plaintext = ResourceAdvertisement(outgoing).pack() + incoming.advertisement_packet = AdvertisementPacket + return incoming + + +def verify_receiver_assembly_and_proof() -> None: + link = FakeLink() + payload = b"receiver-assembly-" * 160 + metadata = {"name": "fixture.bin", "kind": "resource"} + outgoing = Resource(payload, link, metadata=metadata, advertise=False, auto_compress=True) + captured: list[RNS.Packet] = [] + callback_result = {} + real_outbound = RNS.Transport.outbound + real_cache = RNS.Transport.cache + + def fake_outbound(packet): + captured.append(packet) + return True + + with tempfile.TemporaryDirectory(prefix="verify-resource-") as directory: + incoming = make_incoming(outgoing, link, directory) + + def assembled(resource): + callback_result["data"] = resource.data.read() + callback_result["metadata"] = resource.metadata + + incoming.callback = assembled + RNS.Transport.outbound = staticmethod(fake_outbound) + RNS.Transport.cache = staticmethod(lambda packet, force_cache=False: None) + try: + incoming.assemble() + finally: + RNS.Transport.outbound = real_outbound + RNS.Transport.cache = real_cache + + proofs = [packet for packet in captured if packet.context == RNS.Packet.RESOURCE_PRF] + if incoming.status != Resource.COMPLETE: + fail(f"S10.8 receiver assembly status is {incoming.status}, want COMPLETE") + if callback_result != {"data": payload, "metadata": metadata}: + fail("S10.8 receiver assembly did not recover payload and metadata") + if len(proofs) != 1 or proofs[0].data != outgoing.hash + outgoing.expected_proof: + fail("S10.8 receiver did not emit the expected RESOURCE_PRF body") + + outgoing.status = Resource.AWAITING_PROOF + outgoing.validate_proof(bytes(64)) + if outgoing.status != Resource.AWAITING_PROOF: + fail("S10.8 initiator accepted an incorrect Resource proof") + outgoing.validate_proof(outgoing.hash + outgoing.expected_proof) + if outgoing.status != Resource.COMPLETE: + fail("S10.8 initiator rejected the correct Resource proof") + + print("PASS S10.8 receiver assembly, metadata recovery, PRF emission, and proof validation") + + +def verify_negative_cases() -> None: + try: + ResourceAdvertisement.unpack(b"\x81\xa1t\x01") + except Exception: + pass + else: + fail("S10.4 malformed Resource advertisement was accepted") + + link = FakeLink() + outgoing = Resource(b"integrity-check-" * 100, link, advertise=False, auto_compress=False) + real_outbound = RNS.Transport.outbound + captured: list[RNS.Packet] = [] + RNS.Transport.outbound = staticmethod(lambda packet: captured.append(packet) or True) + try: + with tempfile.TemporaryDirectory(prefix="verify-resource-negative-") as directory: + wrong_r = make_incoming(outgoing, link, directory) + wrong_r.random_hash = bytes(Resource.RANDOM_HASH_SIZE) + wrong_r.assemble() + if wrong_r.status != Resource.CORRUPT: + fail("S10.8 Resource with wrong r was not marked CORRUPT") + + with tempfile.TemporaryDirectory(prefix="verify-resource-negative-") as directory: + corrupt_parts = [part.data for part in outgoing.parts] + corrupt_parts[0] = bytes([corrupt_parts[0][0] ^ 1]) + corrupt_parts[0][1:] + corrupt = make_incoming(outgoing, link, directory, corrupt_parts) + corrupt.assemble() + if corrupt.status != Resource.CORRUPT: + fail("S10.8 Resource with corrupt part was not marked CORRUPT") + + compressed = Resource(b"Z" * 5000, link, advertise=False, auto_compress=True) + with tempfile.TemporaryDirectory(prefix="verify-resource-negative-") as directory: + oversized = make_incoming(compressed, link, directory) + oversized.max_decompressed_size = 128 + oversized.assemble() + if oversized.status != Resource.CORRUPT or not link.torn_down: + fail("S10.8 oversized decompression was not rejected and link torn down") + + payload_size = ResourceAdvertisement.HASHMAP_MAX_LEN * outgoing.sdu + boundary = Resource(b"B" * payload_size, link, advertise=False, auto_compress=False) + boundary.status = Resource.TRANSFERRING + boundary.adv_sent = time.time() + boundary.rtt = 0.1 + invalid_request = ( + bytes([Resource.HASHMAP_IS_EXHAUSTED]) + + boundary.parts[0].map_hash + + boundary.hash + ) + captured.clear() + boundary.request(invalid_request) + if boundary.status != Resource.FAILED: + fail("S10.7 invalid HMU boundary did not cancel the Resource") + if any(packet.context == RNS.Packet.RESOURCE_HMU for packet in captured): + fail("S10.7 invalid HMU boundary emitted RESOURCE_HMU") + finally: + RNS.Transport.outbound = real_outbound + + print("PASS S10.4/S10.7/S10.8 malformed and corrupt Resource cases are rejected") + + def main() -> None: print(f"verify_resource.py against RNS {RNS.__version__} / LXMF {LXMF.__version__}") verify_default_lxmf_threshold() @@ -246,6 +444,9 @@ def main() -> None: verify_multisegment_total_size() verify_exhausted_request_with_parts() verify_receiver_reject_vs_cancel() + verify_deterministic_vector() + verify_receiver_assembly_and_proof() + verify_negative_cases() print("ALL PASS") diff --git a/tools/verify_stamps.py b/tools/verify_stamps.py index 13fce60..3614ab9 100644 --- a/tools/verify_stamps.py +++ b/tools/verify_stamps.py @@ -151,10 +151,17 @@ def verify_ticket_shortcut(target_cost=4): lxm = LXMessage(destination=dst, source=src, content=b"x", title=b"", fields={}, desired_method=LXMessage.OPPORTUNISTIC) - lxm.message_id = hashlib.sha256(b"ticket-shortcut-test").digest() - - ticket = os.urandom(LXMessage.TICKET_LENGTH) # 16B - expected_stamp = RNS.Identity.truncated_hash(ticket + lxm.message_id) + ticket = bytes(range(LXMessage.TICKET_LENGTH)) + nonce = 0 + while True: + lxm.message_id = hashlib.sha256( + b"ticket-shortcut-test" + nonce.to_bytes(4, "big") + ).digest() + expected_stamp = RNS.Identity.truncated_hash(ticket + lxm.message_id) + workblock = LXStamper.stamp_workblock(lxm.message_id) + if not LXStamper.stamp_valid(expected_stamp, target_cost, workblock): + break + nonce += 1 lxm.stamp = expected_stamp if not lxm.validate_stamp(target_cost, tickets=[ticket]): @@ -164,9 +171,9 @@ def verify_ticket_shortcut(target_cost=4): print("PASS S5.7.3 LXMessage.validate_stamp accepts ticket stamp shortcut") # With wrong ticket — must NOT match the ticket shortcut, and the - # PoW path won't validate either (because expected_stamp wasn't - # generated against the workblock). validate_stamp returns False. - wrong_ticket = os.urandom(LXMessage.TICKET_LENGTH) + # PoW path is known not to validate either; the deterministic fixture + # above explicitly selects a ticket stamp below target_cost. + wrong_ticket = bytes(reversed(ticket)) lxm.stamp_value = None if lxm.validate_stamp(target_cost, tickets=[wrong_ticket]): fail("S5.7.3 validate_stamp accepted with wrong ticket")