Fix and expand §1.3 — on-disk identity format (real spec bug!)

Closes Tier 1 #6 and the entire Tier 1 sweep. Previous §1.3 said the
on-disk byte order was Ed25519_priv(32) || X25519_priv(32) ("opposite
of the public_key concatenation"). That was WRONG.

Verified empirically against RNS 1.2.0 by round-tripping the existing
test vectors through Identity.to_file and reading the bytes back:

    disk = X25519_priv(32) || Ed25519_priv(32)    # same as public_key

This matches Identity.get_private_key() at RNS/Identity.py:694-698:
   return self.prv_bytes + self.sig_prv_bytes
where prv_bytes is X25519 (line 679) and sig_prv_bytes is Ed25519
(line 682). It also matches load_private_key at line 706-717.

Implementations following the prior spec wording would have written
identity files that fail to load on upstream RNS — a real interop
break that would have been very hard to debug because the failure is
in keypair-loading, before any signature operation runs.

§1.3 rewritten and expanded:

  - Correct byte order with citation to upstream code.
  - 64-byte raw-blob format with explicit "no header / no version /
    no checksum / no encryption".
  - File-system facts: no chmod, expected to live in OS-protected
    storage, filename is caller-controlled.
  - from_bytes HAZARD note: feeding raw random bytes skips the
    `cryptography` library's keypair-generation invariants
    (X25519 RFC 7748 §5 scalar clamping etc).
  - Cross-implementation portability follows automatically because
    there's nothing in the file but the bytes.
  - ⚠️ Spec correction callout warning future readers about the
    previous wording so the bug history is on record.

tools/verify_destination_hash.py extended with a §1.3 to_file /
from_file round-trip section. For each test vector it now:
  - writes the identity via to_file
  - asserts the on-disk file is exactly 64 bytes
  - asserts disk[:32] hex == expected x25519_priv_hex
  - asserts disk[32:64] hex == expected ed25519_priv_hex
  - reloads via from_file and asserts identity_hash invariance

This is what would have caught the bug if it had been there from the
start. tools/README.md updated to reflect §1.3 coverage.

Cumulative Tier 1 status: 6 of 6 done. A from-scratch client built
from §1-§9 + §10 + §11 + flows/ can now interop with upstream
Reticulum / LXMF / RNode for identity, announce, opportunistic LXMF
DATA, Resource fragmentation, regular PROOF receipts, link
handshakes with MTU/mode signalling, path-? discovery, and
KISS/HDLC/RNode-air-frame framing. Tiers 2 and 3 remain open in the
todo for follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
Rob 2026-05-03 11:54:54 -04:00
commit 537b1e8182
4 changed files with 106 additions and 9 deletions

View file

@ -23,7 +23,7 @@ Populated against RNS 1.2.0 / LXMF 0.9.6:
| Script | Verifies SPEC.md section | Status |
|---|---|---|
| `verify_destination_hash.py` | §1.1, §1.2 — identity composition + `dest_hash = SHA256(name_hash \|\| identity_hash)[:16]` | ✅ |
| `verify_destination_hash.py` | §1.1, §1.2, §1.3 — identity composition, `dest_hash = SHA256(name_hash \|\| identity_hash)[:16]`, on-disk private-key round-trip via `to_file`/`from_file` | ✅ |
| `verify_packet_header.py` | §2.1, §2.2, §2.3 — flag byte layout, HEADER_1/HEADER_2 form, originator HEADER_1→HEADER_2 conversion via upstream `Transport.outbound` | ✅ |
| `verify_announce_app_data.py` | §4.3 — LXMF announce app_data 2-element form, parser tolerance | ✅ |
| `verify_path_request.py` | §1.2 well-known hashes, §7.1 LXMF path-preamble gating | ✅ |

View file

@ -1,5 +1,5 @@
"""
Verifier for SPEC.md S1.1 (identity composition) and S1.2 (destination hash).
Verifier for SPEC.md S1.1, S1.2, S1.3.
Reads test-vectors/identities.json and, for each vector:
- Loads the private-key bytes via RNS.Identity.from_bytes (the upstream API
@ -12,6 +12,10 @@ Reads test-vectors/identities.json and, for each vector:
expected.destination_hash_hex.
- Cross-checks RNS.Destination.hash(identity_hash, *name.split(".")) ==
expected.destination_hash_hex (i.e. upstream agrees with the by-hand recipe).
- S1.3 round-trip: writes the identity via `to_file`, reads the bytes back
from disk, confirms they are exactly the 64-byte X25519||Ed25519 concat
with no header / version byte / checksum / encryption, then loads via
`from_file` and confirms the resulting identity hash matches the original.
Exit code 0 on PASS, non-zero on FAIL.
"""
@ -22,6 +26,7 @@ import hashlib
import json
import os
import sys
import tempfile
import RNS
@ -77,7 +82,42 @@ def verify_vector(v: dict) -> None:
f" RNS.Destination.hash: {rns_dest_hash.hex()}\n"
f" want: {expect['destination_hash_hex']}")
print(f"PASS {label}: identity, identity_hash, dest_hash for {full!r}")
# S1.3: to_file / from_file round-trip + on-disk byte-order verification
fp = tempfile.NamedTemporaryFile(delete=False)
fp.close()
try:
if not identity.to_file(fp.name):
fail(f"{label}: Identity.to_file returned False")
with open(fp.name, "rb") as fh:
disk = fh.read()
if len(disk) != 64:
fail(f"{label}: on-disk identity is {len(disk)} bytes, want 64 (no header/checksum)")
# Per S1.3 the on-disk order is X25519_priv(32) || Ed25519_priv(32)
if disk[:32].hex() != inputs["x25519_priv_hex"]:
fail(f"{label}: on-disk bytes [0:32] != X25519 priv\n"
f" got: {disk[:32].hex()}\n"
f" want: {inputs['x25519_priv_hex']}")
if disk[32:].hex() != inputs["ed25519_priv_hex"]:
fail(f"{label}: on-disk bytes [32:64] != Ed25519 priv\n"
f" got: {disk[32:].hex()}\n"
f" want: {inputs['ed25519_priv_hex']}")
# from_file must reconstitute the same identity_hash
reloaded = RNS.Identity.from_file(fp.name)
if reloaded is None:
fail(f"{label}: Identity.from_file returned None")
if reloaded.hash.hex() != expect["identity_hash_hex"]:
fail(f"{label}: from_file identity hash mismatch\n"
f" got: {reloaded.hash.hex()}\n"
f" want: {expect['identity_hash_hex']}")
finally:
os.unlink(fp.name)
print(f"PASS {label}: identity, identity_hash, dest_hash, "
"on-disk round-trip for {!r}".format(full))
def main():