reticiulum-specification/README.md

80 lines
6.7 KiB
Markdown
Raw Normal View History

# Reticulum Specifications
Byte-level interoperability specifications for the [Reticulum Network Stack](https://reticulum.network/) and [LXMF](https://github.com/markqvist/LXMF) — the parts that aren't in the upstream manuals but are needed to build a working client from scratch.
Upstream Reticulum has excellent operator-facing documentation (config, deployment, design philosophy). What's missing — and what every alternative implementation has had to reverse-engineer from the Python source — is an authoritative wire-level spec: header bit layouts, msgpack field types, signature input formats, the exact behavior of `Transport.outbound`, and the long list of "would never guess from reading the manual" gotchas that cost hours of debugging each.
This repo collects those findings in one place. The hope is that future client authors (Kotlin, Swift, Rust, Go, embedded C — pick your stack) can read this instead of re-deriving everything from `RNS/Transport.py`.
## Status
**Early days, contributions welcome.** Current content was bootstrapped from the working notes of two reverse-engineering efforts:
- The web-based Reticulum client at [`reticulum-lora-webclient`](https://github.com/thatSFguy/reticulum-lora-webclient)
- The native Android client at [`reticulum-mobile-app`](https://github.com/thatSFguy/reticulum-mobile-app)
Each finding is grounded in upstream source citations (file + line) so it can be re-verified as RNS evolves.
## What's here
- [`SPEC.md`](SPEC.md) — the single combined spec document, organized by protocol layer
docs: add playbook.md + AGENTS.md template for new Reticulum impls Two new files plus pointers from README.md and agent.md. playbook.md — companion to agent.md. Where agent.md governs what evidence is admissible as you add to the spec, playbook.md covers how to navigate the work itself: triage checklist for wire-format bugs, common debugging anti-patterns (the stale-sibling-binary trap, trusting LLM training data on Reticulum specifics, chasing intermittent symptoms with retries), the three layers of test trustworthiness, and how to work productively in a code-as-spec domain. Includes an incident registry seeded with the §6.2/§6.6 signed_data signalling bug surfaced in mobile-app today plus older HEADER_2, REQUEST path_hash, DEST_LINK, and stale-binary incidents. Append-only — every future interop fix gets a registry entry per §8. templates/AGENTS.md — drop-in boilerplate for new Reticulum implementation projects in any language. Uppercase plural matches the emerging AGENTS.md convention (Claude Code, Codex, Cursor, Copilot Workspace). Sections: read-these-first reading list, cardinal rules summary, project-specific FILL-IN placeholders, contributing-findings-back obligation, attribution. Project-specific bits use HTML comments so they're obvious to edit. §5 attribution points back to this repo and is mandatory per CC BY 4.0. templates/README.md — names the template, says where to put it, restates the attribution expectation. agent.md and README.md updated with pointers to the new files so anyone reading the front door of the repo finds them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-10 20:05:22 -04:00
- [`playbook.md`](playbook.md) — how to troubleshoot interop bugs, design tests that don't lie to you, and navigate the protocol's code-as-spec parts. **Read this if you're starting any Reticulum implementation work, not just contributing to this repo.** Includes an incident registry of past wire-format bugs and their fixes.
- [`agent.md`](agent.md) — verification rules for adding to this repo (markers, tools/, test-vectors)
- [`templates/`](templates/) — drop-in `AGENTS.md` for new Reticulum implementation projects in any language. Copy into your project root, edit the marked sections, and the next agent or contributor lands on the right docs automatically.
- [`flows/`](flows/) — chronological end-to-end narratives (e.g. "send a message"), cross-referencing SPEC.md sections
Re-anchor against RNS 1.2.4 / LXMF 0.9.7 + track upstream distribution shift Upstream RNS 1.2.4 (2026-05-07) announces it is "probably the last release that is also published to GitHub" — pip continues until rnpkg is complete and RNS is self-hosting. All 13 verifiers pass against 1.2.4 / 0.9.7; no wire-format, signing, or protocol behavior changed between 1.2.0 and 1.2.4, so the changes here are purely currency: - Pin tools/requirements.txt to rns==1.2.4 / lxmf==0.9.7 so the verifier stays reproducible if upstream stops mirroring to PyPI before the migration is ready. - Add an "Upstream distribution shift" watch-list to todo.md (local Reticulum node, repo destination hash, rnpkg install/upgrade commands, rsg signature verification, mirroring source citations). - Bump SPEC.md frontmatter and re-anchor ~50 line citations across Identity.py, Transport.py, Resource.py, Link.py, Reticulum.py, Packet.py, and LXMF/* (Identity.py drift was the heaviest at +13 to +31 lines; Transport.py was variable). Fix one numeric (MAX_RANDOM_BLOBS = 32 → 64) and one semantic (§6.6.3 LRPROOF MTU clamp citation pointed at the wrong location — corrected to point at the transit-relay clamp at Transport.py:1539-1556). - Update §10.4 decompression-bomb hazard to note upstream's 1.1.9 cap adoption, with citations to Resource.py:686-691 and Buffer.py:95-97 plus a "do not use one-shot bz2.decompress()" warning. - Re-anchor 11 flows/ files (version pins + ~30 line citations). - Bump version labels in tools/README.md, test-vectors/README.md, and 4 verifier docstrings + 2 hardcoded print strings. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-08 07:42:25 -04:00
- [`tools/`](tools/) — self-contained Python verifier scripts that test SPEC.md claims against upstream RNS / LXMF. Pinned via [`tools/requirements.txt`](tools/requirements.txt) to the upstream versions the scripts were last re-verified against
- [`test-vectors/`](test-vectors/) — known-good byte sequences each implementation should be able to round-trip (intent: grow into a compliance suite)
As content grows, `SPEC.md` will be split into per-layer files (packet header, identity, announce, token-crypto, LXMF, link, resource, transport).
## Spec corrections
Errata that may invalidate code built against an earlier revision of `SPEC.md`. Newest first. Feature additions and ordinary edits live in `git log` — this section is reserved for cases where the spec said one thing, that turned out to be wrong, and an implementer who pulled the bad version needs to fix their code.
- **2026-05-17 — §10.2 Resource integrity hash: the 4-byte prefix is NOT `r`, and is NOT in the hash input.**
Bad text introduced in [`95823ad`](../../commit/95823ad); on master from 2026-05-03 to 2026-05-17. §10.2 step 3 wrongly equated the random-hash *prefix* prepended to the Resource body with the advertisement's `r` field, and step 5 wrongly fed that prefix into `hash`/`expected_proof` (claiming `hash = SHA256(random_hash || body || random_hash)`). Upstream `RNS/Resource.py` (1.2.4) uses *two distinct* `get_random_hash()[:4]` values: a throwaway prefix the receiver strips and discards (`:405`/`412`, `:682`), and `self.random_hash` — the advertisement's `r` field (`:440`, `:1285`). The integrity hash is `SHA256(uncompressed_plaintext || r)` over the prefix-stripped, decompressed body (`:441`, `:694`) — exactly as §10.8 already stated. An implementer who trusted §10.2 step 5 computes a hash no spec-compliant peer accepts; every Resource is rejected as `CORRUPT`. §10.2 corrected to agree with §10.8; §10.12's wire-layering block fixed to match. Surfaced by [issue #9](../../issues/9).
- **2026-05-06 — §2.1 flag byte: bit 7 is the IFAC flag, not part of `header_type`.**
Bad text introduced in [`8c4d550`](../../commit/8c4d550), corrected in [`0c2021e`](../../commit/0c2021e); on master from 2026-05-04 to 2026-05-06. The corrected layout is `ifac_flag(bit 7) | header_type(bit 6) | context_flag(5) | transport_type(4) | destination_type(3-2) | packet_type(1-0)`, matching the official manual §4.6.3 and upstream `RNS/Packet.py:246` (parse mask `0b01000000 >> 6`) / `RNS/Transport.py:1003` (IFAC setter `raw[0] | 0x80`). Implementers who consumed the bad version will mis-parse every IFAC-protected packet as `header_type ∈ {2, 3}` and drop it. Surfaced by [issue #4](../../issues/4) item #1.
## Scope
**In scope:**
- Wire formats: byte layouts, field encodings, framing
- Signing inputs and what's hashed where
- Cross-cutting behaviors required for interop (path requests, ratchet rotation, retransmit semantics)
- "Gotchas" — things upstream code does that aren't obvious from the manual or RFC-style sketches
- Test vectors that any implementation must be able to round-trip
**Out of scope:**
- Operator/user documentation — see [the official manual](https://markqvist.github.io/Reticulum/manual/)
- API design choices for any specific implementation
- Networking layer config (interfaces, transport modes) — already well documented
## Source citations
Where a finding cites upstream Python code, the path is relative to a standard `pip install rns lxmf` installation, e.g. `RNS/Transport.py`, `LXMF/LXMF.py`. Where the bundled `umsgpack` is referenced, the path is `RNS/vendor/umsgpack.py`.
When upstream code changes such that a citation no longer matches, file an issue or PR — the goal is to track the de-facto wire spec as it actually behaves, not as it was at any single snapshot.
## Contributing
If you've debugged a Reticulum interop problem and the answer wasn't in the upstream docs, please add it. Format:
```markdown
### N.M Short description of the finding
**Symptom:** what you observed that prompted the investigation.
**What's happening:** the actual mechanism, ideally with upstream source citation (file + line).
**Implication / fix:** what an implementation must do to interop.
**Source:** upstream file paths and approximate line numbers.
```
Add a worked test vector to `test-vectors/` if the finding is byte-level.
## License
[CC BY 4.0](LICENSE) — use freely, attribution appreciated.