diff --git a/.agents/skills/lxmf-update/SKILL.md b/.agents/skills/lxmf-update/SKILL.md new file mode 100644 index 0000000..20396c3 --- /dev/null +++ b/.agents/skills/lxmf-update/SKILL.md @@ -0,0 +1,40 @@ +--- +name: lxmf-update +description: Check upstream LXMF and RNS implementations for changes affecting the LXMF specification, run local verifier tools, and propose spec updates with citations. +--- + +# LXMF update skill + +Use this skill when asked to check whether the LXMF specification is current with upstream LXMF or RNS. + +## Sources of truth + +Prefer, in order: + +1. Upstream LXMF source code at a pinned release or commit. +2. Upstream RNS source code where LXMF behavior depends on Reticulum behavior. +3. Reproducible local traces and test vectors. +4. Upstream prose documentation, only when consistent with source behavior. + +## Required workflow + +1. Determine the currently pinned upstream LXMF and RNS versions from `SOURCES.md`. +2. Check whether newer upstream releases or commits exist. +3. Identify changed files relevant to: + - message packing + - fields and flags + - destination handling + - identity/signature behavior + - encryption behavior + - propagation + - delivery semantics + - storage format if externally observable +4. Run the verifier tools in `tools/`. +5. Compare observed behavior against `SPEC.md`. +6. Produce a report with: + - changed upstream files + - affected specification sections + - proposed edits + - test-vector impact + - unresolved ambiguities +7. Do not modify `SPEC.md` unless explicitly asked. diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..d4c9224 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,16 @@ +# AGENTS.md + +## Repository purpose + +This repository attempts to define an implementation-derived specification for LXMF. + +Do not treat upstream prose documentation as authoritative when it conflicts with observed source behavior. Prefer source code, test vectors, and reproducible traces. + +## Required discipline + +- Distinguish normative behavior from observed implementation behavior. +- Cite exact upstream files, functions, classes, constants, and version tags. +- Do not invent protocol rules. +- When behavior is unclear, mark it as "undetermined" rather than guessing. +- Prefer Perl tools for extraction, comparison, and report generation. +- Keep generated examples reproducible from scripts under tools/. diff --git a/ChatGPT_1.md b/ChatGPT_1.md new file mode 100644 index 0000000..c40e197 --- /dev/null +++ b/ChatGPT_1.md @@ -0,0 +1,174 @@ +Yes. For Codex, the closest equivalents are: + +1. **`AGENTS.md`** for persistent project-wide instructions. +2. **Codex Skills using `SKILL.md`** for reusable workflows, very similar in spirit to the Claude Code file you linked. +3. Optionally **`PLANS.md` / ExecPlan-style documents** for long-running specification work. + +The important correction is that Codex does now support **Skills with `SKILL.md`**, not just `AGENTS.md`. OpenAI’s Codex documentation says a skill is a directory containing a required `SKILL.md` file, with optional `scripts/`, `references/`, `assets/`, and related support files. Codex initially sees the skill name, description, and path, then loads the full `SKILL.md` only when it decides that skill applies. ([OpenAI Developers][1]) + +So, yes: you can do essentially the same thing that `thatSFguy` is doing, but I would structure it slightly differently for LXMF. + +## Recommended Codex layout + +For an LXMF specification repository, I would use something like: + +```text +lxmf-specification/ +├── AGENTS.md +├── SPEC.md +├── NOTES.md +├── SOURCES.md +├── tools/ +│ ├── extract_lxmf_constants.pl +│ ├── compare_upstream_lxmf.pl +│ ├── verify_examples.pl +│ └── make_test_vectors.pl +├── examples/ +│ ├── lxmf_message_minimal.hex +│ ├── lxmf_message_signed.hex +│ └── lxmf_propagation_example.hex +├── references/ +│ ├── lxmf_source_map.md +│ ├── rns_dependency_map.md +│ └── terminology.md +└── .agents/ + └── skills/ + └── lxmf-update/ + └── SKILL.md +``` + +## Role of each file + +### `AGENTS.md` + +Use this for always-on repository rules. Codex reads `AGENTS.md` before doing work, and the docs describe it as the mechanism for project-specific persistent guidance. ([OpenAI Developers][2]) + +Example use: + +```markdown +# AGENTS.md + +## Repository purpose + +This repository attempts to define an implementation-derived specification for LXMF. + +Do not treat upstream prose documentation as authoritative when it conflicts with observed source behavior. Prefer source code, test vectors, and reproducible traces. + +## Required discipline + +- Distinguish normative behavior from observed implementation behavior. +- Cite exact upstream files, functions, classes, constants, and version tags. +- Do not invent protocol rules. +- When behavior is unclear, mark it as "undetermined" rather than guessing. +- Prefer Perl tools for extraction, comparison, and report generation. +- Keep generated examples reproducible from scripts under tools/. +``` + +### `.agents/skills/lxmf-update/SKILL.md` + +Use this for the repeatable workflow: “check upstream LXMF/RNS, compare against our spec, update citations, run verifiers, propose diffs.” + +That is the closest Codex analog to the Claude Code `rns-update/SKILL.md` you found. + +A minimal Codex skill might look like: + +```markdown +--- +name: lxmf-update +description: Check upstream LXMF and RNS implementations for changes affecting the LXMF specification, run local verifier tools, and propose spec updates with citations. +--- + +# LXMF update skill + +Use this skill when asked to check whether the LXMF specification is current with upstream LXMF or RNS. + +## Sources of truth + +Prefer, in order: + +1. Upstream LXMF source code at a pinned release or commit. +2. Upstream RNS source code where LXMF behavior depends on Reticulum behavior. +3. Reproducible local traces and test vectors. +4. Upstream prose documentation, only when consistent with source behavior. + +## Required workflow + +1. Determine the currently pinned upstream LXMF and RNS versions from `SOURCES.md`. +2. Check whether newer upstream releases or commits exist. +3. Identify changed files relevant to: + - message packing + - fields and flags + - destination handling + - identity/signature behavior + - encryption behavior + - propagation + - delivery semantics + - storage format if externally observable +4. Run the verifier tools in `tools/`. +5. Compare observed behavior against `SPEC.md`. +6. Produce a report with: + - changed upstream files + - affected specification sections + - proposed edits + - test-vector impact + - unresolved ambiguities +7. Do not modify `SPEC.md` unless explicitly asked. +``` + +Codex supports skills in the CLI, IDE extension, and Codex app, and the documented repository skill location is `.agents/skills` under the current directory or repository root. ([OpenAI Developers][1]) + +### `PLANS.md` + +For your use case, I would also add a planning document, because “derive a protocol specification from implementation” is not a one-shot coding task. OpenAI’s Codex cookbook describes using `PLANS.md` / ExecPlans for long-running, multi-stage work, with `AGENTS.md` telling Codex when to use it. ([OpenAI Developers][3]) + +For LXMF, `PLANS.md` could define the required sections of each investigation: + +```markdown +# LXMF Specification Investigation Plan + +Each investigation must include: + +1. Question being answered. +2. Upstream files inspected. +3. Exact version or commit. +4. Relevant classes/functions/constants. +5. Observed behavior. +6. Whether behavior is normative, inferred, or unresolved. +7. Test vectors needed. +8. SPEC.md sections affected. +9. Open questions. +``` + +## My recommendation + +For your goal, I would not start by asking Codex to “write the LXMF specification.” That is too broad and likely to produce confident prose ahead of evidence. + +I would instead give Codex narrow tasks such as: + +```text +Using the lxmf-update skill, inspect upstream LXMF and determine the exact serialized structure of an LXMessage. Do not edit SPEC.md. Produce a report listing every field, its source-code origin, and whether its behavior is confirmed by a test vector. +``` + +Then: + +```text +Create Perl tooling under tools/ that extracts or verifies the LXMF message packing behavior described in the report. Generate at least one reproducible test vector. +``` + +Then: + +```text +Update SPEC.md section "LXMessage Wire Format" using only behavior confirmed by source inspection or verifier output. Mark all unverified behavior as unresolved. +``` + +## Bottom line + +Yes, you can accomplish the same kind of workflow with Codex. In fact, Codex gives you both mechanisms: + +`AGENTS.md` for standing repo rules, and `SKILL.md` under `.agents/skills/...` for a reusable “LXMF update / spec maintenance” workflow. OpenAI’s Codex docs explicitly describe customization as a combination of `AGENTS.md`, skills, MCP, memories, and subagents; for your use case, `AGENTS.md` plus a `SKILL.md` skill is the right starting point. ([OpenAI Developers][4]) + +[1]: https://developers.openai.com/codex/skills "Agent Skills – Codex | OpenAI Developers" +[2]: https://developers.openai.com/codex/guides/agents-md "Custom instructions with AGENTS.md – Codex | OpenAI Developers" +[3]: https://developers.openai.com/cookbook/articles/codex_exec_plans "Using PLANS.md for multi-hour problem solving" +[4]: https://developers.openai.com/codex/concepts/customization "Customization – Codex | OpenAI Developers" + diff --git a/Codex_1.md b/Codex_1.md new file mode 100644 index 0000000..049676a --- /dev/null +++ b/Codex_1.md @@ -0,0 +1,8 @@ +This directory: /usr/local/src/reticulum/LXMF +contains my GitHub fork (https://github.com/jlpoolen/LXMF) of Mark Qvist's: https://github.com/markqvist/LXMF which, incidentally, is a public mirror of some git server served up through Reticulum -- the source is not provided. + +I created an empty project in my Forgejo server: LXMF-specification at https://salemdata.net/repo/jlpoole/LMXF-specification +and I added my Forgejo instance as a remote repository identified by "forgejo". + +I added some files based on a ChatGPT chat, see ChatGPT_1.md. +So, for your first task: Using the lxmf-update skill, inspect upstream LXMF and determine the exact serialized structure of an LXMessage. Do not edit SPEC.md. Produce a report listing every field, its source-code origin, and whether its behavior is confirmed by a test vector. diff --git a/Codex_Response_1.md b/Codex_Response_1.md new file mode 100644 index 0000000..8b45de7 --- /dev/null +++ b/Codex_Response_1.md @@ -0,0 +1,83 @@ +No, I don’t need `SPEC.md` or `SOURCES.md` for this first report. I used the checked-out LXMF source directly. I did not edit `SPEC.md` or any other file. + +**LXMessage Serialized Structure** + +Source inspected: LXMF commit `fab12ad9bf9f997797034950f289fe41a79dcf5a` + +Core `lxmf_bytes` layout: + +```text +destination_hash || source_hash || signature || msgpack_payload +``` + +Where `msgpack_payload` is: + +```text +[ + timestamp, + title, + content, + fields, + optional_stamp +] +``` + +The fifth element is only present when a normal message stamp is generated and appended. + +| Field | Serialized position | Type / size | Source origin | Test vector confirmed | +|---|---:|---|---|---| +| `destination_hash` | bytes `0..15` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:383), length from [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:40) and RNS 128-bit truncated hashes in [Reticulum.py](/usr/local/src/reticulum/Reticulum/RNS/Reticulum.py:146) | No | +| `source_hash` | bytes `16..31` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:384) | No | +| `signature` | bytes `32..95` | 64 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:385); RNS signature length from [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:59) and [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:80) | No | +| `msgpack_payload` | bytes `96..end` | msgpack list | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362), packed at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:381) | No | +| `timestamp` | payload index `0` | Python float, normally msgpack float64 on this RNS umsgpack | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:357), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362); float64 behavior from [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:1177) and [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:325) | No | +| `title` | payload index `1` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:130), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:193), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No | +| `content` | payload index `2` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:135), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:202), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No | +| `fields` | payload index `3` | dict, default `{}` | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:215), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No | +| `stamp` | payload index `4`, optional | bytes, only if generated | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:371), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:373); extracted on unpack at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754) | No | + +**Hash And Signature** + +`message_id` / `hash` is: + +```text +SHA256(destination_hash || source_hash || msgpack([timestamp, title, content, fields])) +``` + +Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:364), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:368), RNS SHA-256 at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:238). + +The signature is over: + +```text +destination_hash || source_hash || msgpack([timestamp, title, content, fields]) || message_id +``` + +Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:375), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), RNS signing at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:767). + +Important: the optional normal `stamp` is excluded from both `message_id` and signature. Unpack removes payload index `4` before recomputing hash/signature material: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754). + +**Outer Representations** + +Direct/link/resource delivery uses full `self.packed` unchanged: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:636), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:654). + +Opportunistic packet delivery omits the destination hash from the packet payload because the RNS packet destination already implies it: + +```text +source_hash || signature || msgpack_payload +``` + +Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:633). + +Propagated and paper messages wrap/encrypt everything after the destination hash: + +```text +destination_hash || encrypted(source_hash || signature || msgpack_payload) +``` + +Source: propagated [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:429), paper [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:449). Propagation transit additionally msgpacks `[time.time(), [lxmf_data]]`: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:436). + +**Notes** + +The README is not authoritative here: it describes payload order as timestamp, content, title, fields, but current code serializes timestamp, title, content, fields. + +No test vector confirmation exists in this checkout. There is no `tools/` directory and no tracked `SPEC.md` or `SOURCES.md`, so every “Test vector confirmed” entry above is marked `No`. diff --git a/Codex_Response_2.md b/Codex_Response_2.md new file mode 100644 index 0000000..8b45de7 --- /dev/null +++ b/Codex_Response_2.md @@ -0,0 +1,83 @@ +No, I don’t need `SPEC.md` or `SOURCES.md` for this first report. I used the checked-out LXMF source directly. I did not edit `SPEC.md` or any other file. + +**LXMessage Serialized Structure** + +Source inspected: LXMF commit `fab12ad9bf9f997797034950f289fe41a79dcf5a` + +Core `lxmf_bytes` layout: + +```text +destination_hash || source_hash || signature || msgpack_payload +``` + +Where `msgpack_payload` is: + +```text +[ + timestamp, + title, + content, + fields, + optional_stamp +] +``` + +The fifth element is only present when a normal message stamp is generated and appended. + +| Field | Serialized position | Type / size | Source origin | Test vector confirmed | +|---|---:|---|---|---| +| `destination_hash` | bytes `0..15` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:383), length from [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:40) and RNS 128-bit truncated hashes in [Reticulum.py](/usr/local/src/reticulum/Reticulum/RNS/Reticulum.py:146) | No | +| `source_hash` | bytes `16..31` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:384) | No | +| `signature` | bytes `32..95` | 64 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:385); RNS signature length from [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:59) and [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:80) | No | +| `msgpack_payload` | bytes `96..end` | msgpack list | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362), packed at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:381) | No | +| `timestamp` | payload index `0` | Python float, normally msgpack float64 on this RNS umsgpack | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:357), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362); float64 behavior from [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:1177) and [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:325) | No | +| `title` | payload index `1` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:130), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:193), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No | +| `content` | payload index `2` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:135), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:202), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No | +| `fields` | payload index `3` | dict, default `{}` | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:215), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No | +| `stamp` | payload index `4`, optional | bytes, only if generated | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:371), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:373); extracted on unpack at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754) | No | + +**Hash And Signature** + +`message_id` / `hash` is: + +```text +SHA256(destination_hash || source_hash || msgpack([timestamp, title, content, fields])) +``` + +Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:364), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:368), RNS SHA-256 at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:238). + +The signature is over: + +```text +destination_hash || source_hash || msgpack([timestamp, title, content, fields]) || message_id +``` + +Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:375), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), RNS signing at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:767). + +Important: the optional normal `stamp` is excluded from both `message_id` and signature. Unpack removes payload index `4` before recomputing hash/signature material: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754). + +**Outer Representations** + +Direct/link/resource delivery uses full `self.packed` unchanged: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:636), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:654). + +Opportunistic packet delivery omits the destination hash from the packet payload because the RNS packet destination already implies it: + +```text +source_hash || signature || msgpack_payload +``` + +Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:633). + +Propagated and paper messages wrap/encrypt everything after the destination hash: + +```text +destination_hash || encrypted(source_hash || signature || msgpack_payload) +``` + +Source: propagated [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:429), paper [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:449). Propagation transit additionally msgpacks `[time.time(), [lxmf_data]]`: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:436). + +**Notes** + +The README is not authoritative here: it describes payload order as timestamp, content, title, fields, but current code serializes timestamp, title, content, fields. + +No test vector confirmation exists in this checkout. There is no `tools/` directory and no tracked `SPEC.md` or `SOURCES.md`, so every “Test vector confirmed” entry above is marked `No`. diff --git a/Codex_Response_3.md b/Codex_Response_3.md new file mode 100644 index 0000000..6362cd8 --- /dev/null +++ b/Codex_Response_3.md @@ -0,0 +1,27 @@ +Task #2 completed in `/usr/local/src/reticulum/LXMF`. + +Created: + +- `tools/make_test_vectors.pl` +- `tools/verify_examples.pl` +- `tools/LXMF/MessagePack.pm` +- `tools/README.md` +- `examples/lxmf_message_minimal.hex` +- `examples/lxmf_message_stamped.hex` +- `examples/lxmf_message_vectors.json` + +The dependency-free Perl tooling verifies: + +- Fixed hash/signature positions +- Payload order and MessagePack types +- Message ID calculation +- Optional stamp exclusion from message ID and signature input + +Validation passed: + +```text +Reticulum umsgpack byte-for-byte cross-check passed +All LXMessage test-vector checks passed. +``` + +The vectors regenerate deterministically. Signature bytes are documented placeholders; Ed25519 validation is not claimed. `SPEC.md` was not created or modified. diff --git a/PLANS.md b/PLANS.md new file mode 100644 index 0000000..cbaedc0 --- /dev/null +++ b/PLANS.md @@ -0,0 +1,13 @@ +# LXMF Specification Investigation Plan + +Each investigation must include: + +1. Question being answered. +2. Upstream files inspected. +3. Exact version or commit. +4. Relevant classes/functions/constants. +5. Observed behavior. +6. Whether behavior is normative, inferred, or unresolved. +7. Test vectors needed. +8. SPEC.md sections affected. +9. Open questions. diff --git a/examples/lxmf_message_minimal.hex b/examples/lxmf_message_minimal.hex new file mode 100644 index 0000000..85f8d89 --- /dev/null +++ b/examples/lxmf_message_minimal.hex @@ -0,0 +1 @@ +000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980 diff --git a/examples/lxmf_message_stamped.hex b/examples/lxmf_message_stamped.hex new file mode 100644 index 0000000..be6fc07 --- /dev/null +++ b/examples/lxmf_message_stamped.hex @@ -0,0 +1 @@ +000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f95cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980c420a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf diff --git a/examples/lxmf_message_vectors.json b/examples/lxmf_message_vectors.json new file mode 100644 index 0000000..b60d155 --- /dev/null +++ b/examples/lxmf_message_vectors.json @@ -0,0 +1,26 @@ +{ + "content_hex" : "44657465726d696e6973746963204c584d4620626f6479", + "description" : "Deterministic LXMessage packing vectors; signature bytes are placeholders", + "destination_hash_hex" : "000102030405060708090a0b0c0d0e0f", + "fields" : {}, + "format" : "destination_hash || source_hash || signature || msgpack_payload", + "message_id_hex" : "55f497f6f2ea5d858eaff541a9fa77f9d5bdf57be75be4a94df65d1f9a7ffea3", + "minimal_lxmf_hex" : "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980", + "payload_order" : [ + "timestamp", + "title", + "content", + "fields", + "optional_stamp" + ], + "payload_with_stamp_hex" : "95cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980c420a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf", + "payload_without_stamp_hex" : "94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980", + "signature_hex" : "202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f", + "signature_valid" : false, + "signed_part_hex" : "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f64798055f497f6f2ea5d858eaff541a9fa77f9d5bdf57be75be4a94df65d1f9a7ffea3", + "source_hash_hex" : "101112131415161718191a1b1c1d1e1f", + "stamp_hex" : "a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf", + "stamped_lxmf_hex" : "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f95cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980c420a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf", + "timestamp" : 1700000000.25, + "title_hex" : "54657374207469746c65" +} diff --git a/grep_results.txt b/grep_results.txt new file mode 100644 index 0000000..52ae092 --- /dev/null +++ b/grep_results.txt @@ -0,0 +1,39 @@ +./LXMF/Utilities/lxmd.py:DEFFERED_JOBS_DELAY = 10 +./LXMF/Utilities/lxmd.py:def create_default_config(configpath): +./LXMF/Utilities/lxmd.py:def apply_config(): +./LXMF/Utilities/lxmd.py:def lxmf_delivery(lxm): +./LXMF/Utilities/lxmd.py:def program_setup(configdir = None, rnsconfigdir = None, run_pn = False, on_inbound = None, verbosity = 0, quietness = 0, service = False): +./LXMF/Utilities/lxmd.py:def jobs(): +./LXMF/Utilities/lxmd.py:def deferred_start_jobs(): +./LXMF/Utilities/lxmd.py:def _request_sync(identity, destination_hash, remote_identity, timeout=15, exit_on_fail=False): +./LXMF/Utilities/lxmd.py:def request_sync(target, remote=None, configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, timeout=15, identity_path=None): +./LXMF/Utilities/lxmd.py:def _request_unpeer(identity, destination_hash, remote_identity, timeout=15, exit_on_fail=False): +./LXMF/Utilities/lxmd.py:def request_unpeer(target, remote=None, configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, timeout=15, identity_path=None): +./LXMF/Utilities/lxmd.py:def query_status(identity, remote_identity=None, timeout=5, exit_on_fail=False): +./LXMF/Utilities/lxmd.py:def get_status(remote=None, configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, timeout=5, +./LXMF/Utilities/lxmd.py:def _get_target_identity(remote=None, timeout=5): +./LXMF/Utilities/lxmd.py:def _remote_init(configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, identity_path=None): +./LXMF/Utilities/lxmd.py:def main(): +./LXMF/LXMF.py:def display_name_from_app_data(app_data=None): +./LXMF/LXMF.py:def stamp_cost_from_app_data(app_data=None): +./LXMF/LXMF.py:def compression_support_from_app_data(app_data=None): +./LXMF/LXMF.py:def pn_name_from_app_data(app_data=None): +./LXMF/LXMF.py:def pn_stamp_cost_from_app_data(app_data=None): +./LXMF/LXMF.py:def pn_announce_data_is_valid(data): +./LXMF/LXStamper.py:def worker_context(ctx, stamp_cost, workblock, message_id): +./LXMF/LXStamper.py:def stamp_workblock(material, expand_rounds=WORKBLOCK_EXPAND_ROUNDS): +./LXMF/LXStamper.py:def stamp_value(workblock, stamp): +./LXMF/LXStamper.py:def stamp_valid(stamp, target_cost, workblock): +./LXMF/LXStamper.py:def validate_peering_key(peering_id, peering_key, target_cost): +./LXMF/LXStamper.py:def validate_pn_stamp(transient_data, target_cost): +./LXMF/LXStamper.py:def validate_pn_stamps_job_simple(transient_list, target_cost): +./LXMF/LXStamper.py:def validate_pn_stamps_job_multip(transient_list, target_cost): +./LXMF/LXStamper.py:def validate_pn_stamps(transient_list, target_cost): +./LXMF/LXStamper.py:def generate_stamp(message_id, stamp_cost, expand_rounds=WORKBLOCK_EXPAND_ROUNDS): +./LXMF/LXStamper.py:def cancel_work(message_id): +./LXMF/LXStamper.py:def job_simple(stamp_cost, workblock, message_id): +./LXMF/LXStamper.py:def job_linux_managed(stamp_cost, workblock, message_id): +./LXMF/LXStamper.py:def job_linux(stamp_cost, workblock, message_id): +./LXMF/LXStamper.py:def job_android(stamp_cost, workblock, message_id): +./docs/example_receiver.py:def delivery_callback(message): +./docs/example_sender.py:def delivery_callback(message): diff --git a/tools/LXMF/MessagePack.pm b/tools/LXMF/MessagePack.pm new file mode 100644 index 0000000..5ce8c39 --- /dev/null +++ b/tools/LXMF/MessagePack.pm @@ -0,0 +1,181 @@ +package LXMF::MessagePack; + +use strict; +use warnings; +use Exporter 'import'; + +our @EXPORT_OK = qw( + pack_array + pack_bin + pack_float64 + pack_map + parse_item +); + +sub pack_float64 { + my ($value) = @_; + return "\xcb" . pack("d>", $value); +} + +sub pack_bin { + my ($value) = @_; + my $length = length($value); + + return "\xc4" . pack("C", $length) . $value if $length <= 0xff; + return "\xc5" . pack("n", $length) . $value if $length <= 0xffff; + return "\xc6" . pack("N", $length) . $value; +} + +sub pack_array { + my (@items) = @_; + my $length = scalar @items; + my $header; + + if ($length <= 15) { + $header = pack("C", 0x90 | $length); + } + elsif ($length <= 0xffff) { + $header = "\xdc" . pack("n", $length); + } + else { + $header = "\xdd" . pack("N", $length); + } + + return $header . join("", @items); +} + +sub pack_map { + my (@entries) = @_; + die "pack_map requires key/value byte-string pairs\n" if @entries % 2; + + my $length = @entries / 2; + my $header; + + if ($length <= 15) { + $header = pack("C", 0x80 | $length); + } + elsif ($length <= 0xffff) { + $header = "\xde" . pack("n", $length); + } + else { + $header = "\xdf" . pack("N", $length); + } + + return $header . join("", @entries); +} + +sub _take { + my ($bytes, $offset_ref, $length) = @_; + die "truncated MessagePack item\n" if $$offset_ref + $length > length($bytes); + my $value = substr($bytes, $$offset_ref, $length); + $$offset_ref += $length; + return $value; +} + +sub _parse_length { + my ($bytes, $offset_ref, $width) = @_; + my $raw = _take($bytes, $offset_ref, $width); + return unpack("C", $raw) if $width == 1; + return unpack("n", $raw) if $width == 2; + return unpack("N", $raw); +} + +sub parse_item { + my ($bytes, $offset_ref) = @_; + $offset_ref //= \my $offset; + $$offset_ref //= 0; + + my $start = $$offset_ref; + my $code = unpack("C", _take($bytes, $offset_ref, 1)); + my $item = { code => $code }; + + if ($code <= 0x7f) { + @$item{qw(type value)} = ("integer", $code); + } + elsif ($code >= 0xe0) { + @$item{qw(type value)} = ("integer", $code - 256); + } + elsif (($code & 0xf0) == 0x90) { + _parse_array($bytes, $offset_ref, $item, $code & 0x0f); + } + elsif (($code & 0xf0) == 0x80) { + _parse_map($bytes, $offset_ref, $item, $code & 0x0f); + } + elsif (($code & 0xe0) == 0xa0) { + my $length = $code & 0x1f; + @$item{qw(type value)} = ("string", _take($bytes, $offset_ref, $length)); + } + elsif ($code == 0xc0) { + $item->{type} = "nil"; + $item->{value} = undef; + } + elsif ($code == 0xc2 || $code == 0xc3) { + @$item{qw(type value)} = ("boolean", $code == 0xc3 ? 1 : 0); + } + elsif ($code == 0xca) { + @$item{qw(type value)} = ("float32", unpack("f>", _take($bytes, $offset_ref, 4))); + } + elsif ($code == 0xcb) { + @$item{qw(type value)} = ("float64", unpack("d>", _take($bytes, $offset_ref, 8))); + } + elsif ($code >= 0xc4 && $code <= 0xc6) { + my $width = 1 << ($code - 0xc4); + my $length = _parse_length($bytes, $offset_ref, $width); + @$item{qw(type value)} = ("binary", _take($bytes, $offset_ref, $length)); + } + elsif ($code >= 0xcc && $code <= 0xcf) { + my @formats = ("C", "n", "N", "Q>"); + my @widths = (1, 2, 4, 8); + my $index = $code - 0xcc; + @$item{qw(type value)} = ( + "integer", + unpack($formats[$index], _take($bytes, $offset_ref, $widths[$index])), + ); + } + elsif ($code >= 0xd0 && $code <= 0xd3) { + my @formats = ("c", "s>", "l>", "q>"); + my @widths = (1, 2, 4, 8); + my $index = $code - 0xd0; + @$item{qw(type value)} = ( + "integer", + unpack($formats[$index], _take($bytes, $offset_ref, $widths[$index])), + ); + } + elsif ($code == 0xd9 || $code == 0xda || $code == 0xdb) { + my $width = 1 << ($code - 0xd9); + my $length = _parse_length($bytes, $offset_ref, $width); + @$item{qw(type value)} = ("string", _take($bytes, $offset_ref, $length)); + } + elsif ($code == 0xdc || $code == 0xdd) { + my $width = $code == 0xdc ? 2 : 4; + _parse_array($bytes, $offset_ref, $item, _parse_length($bytes, $offset_ref, $width)); + } + elsif ($code == 0xde || $code == 0xdf) { + my $width = $code == 0xde ? 2 : 4; + _parse_map($bytes, $offset_ref, $item, _parse_length($bytes, $offset_ref, $width)); + } + else { + die sprintf("unsupported MessagePack code 0x%02x at offset %d\n", $code, $start); + } + + $item->{start} = $start; + $item->{end} = $$offset_ref; + $item->{raw} = substr($bytes, $start, $$offset_ref - $start); + return $item; +} + +sub _parse_array { + my ($bytes, $offset_ref, $item, $length) = @_; + $item->{type} = "array"; + $item->{items} = [map { parse_item($bytes, $offset_ref) } 1 .. $length]; +} + +sub _parse_map { + my ($bytes, $offset_ref, $item, $length) = @_; + $item->{type} = "map"; + $item->{entries} = [ + map { [parse_item($bytes, $offset_ref), parse_item($bytes, $offset_ref)] } 1 .. $length + ]; +} + +1; diff --git a/tools/README.md b/tools/README.md new file mode 100644 index 0000000..75538fa --- /dev/null +++ b/tools/README.md @@ -0,0 +1,32 @@ +# LXMessage Test-Vector Tooling + +The scripts in this directory verify the implementation-derived LXMessage +packing behavior described in `Code_Response_1.md`. + +They use only Perl core modules. The local `LXMF::MessagePack` module implements +the MessagePack subset needed to generate the vectors and parse their complete +payloads. + +Generate deterministic vectors: + +```sh +perl tools/make_test_vectors.pl +``` + +Verify the vectors: + +```sh +perl tools/verify_examples.pl +``` + +The vectors confirm: + +- the 16-byte destination hash, 16-byte source hash, and 64-byte signature + positions; +- payload order `[timestamp, title, content, fields, optional_stamp]`; +- float64 timestamp and binary title/content/stamp encodings; +- the message ID calculation; +- exclusion of the optional stamp from the message ID and signature input. + +The signature bytes are deterministic placeholders. These vectors do not claim +to verify Ed25519 signing or signature validation. diff --git a/tools/make_test_vectors.pl b/tools/make_test_vectors.pl new file mode 100755 index 0000000..8d4385b --- /dev/null +++ b/tools/make_test_vectors.pl @@ -0,0 +1,76 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use Digest::SHA qw(sha256); +use FindBin qw($Bin); +use File::Path qw(make_path); +use JSON::PP; +use lib "$Bin"; +use LXMF::MessagePack qw(pack_array pack_bin pack_float64 pack_map); + +my $output_dir = "$Bin/../examples"; +make_path($output_dir); + +my $destination_hash = pack("C*", 0x00 .. 0x0f); +my $source_hash = pack("C*", 0x10 .. 0x1f); +my $signature = pack("C*", 0x20 .. 0x5f); +my $stamp = pack("C*", 0xa0 .. 0xbf); + +my $timestamp = 1_700_000_000.25; +my $title = "Test title"; +my $content = "Deterministic LXMF body"; +my $fields = pack_map(); + +my @required_items = ( + pack_float64($timestamp), + pack_bin($title), + pack_bin($content), + $fields, +); + +my $payload = pack_array(@required_items); +my $stamped_payload = pack_array(@required_items, pack_bin($stamp)); +my $message_id = sha256($destination_hash . $source_hash . $payload); +my $signed_part = $destination_hash . $source_hash . $payload . $message_id; + +my $minimal = $destination_hash . $source_hash . $signature . $payload; +my $stamped = $destination_hash . $source_hash . $signature . $stamped_payload; + +write_hex("$output_dir/lxmf_message_minimal.hex", $minimal); +write_hex("$output_dir/lxmf_message_stamped.hex", $stamped); + +my $manifest = { + description => "Deterministic LXMessage packing vectors; signature bytes are placeholders", + format => "destination_hash || source_hash || signature || msgpack_payload", + payload_order => [qw(timestamp title content fields optional_stamp)], + destination_hash_hex => unpack("H*", $destination_hash), + source_hash_hex => unpack("H*", $source_hash), + signature_hex => unpack("H*", $signature), + signature_valid => JSON::PP::false, + timestamp => $timestamp, + title_hex => unpack("H*", $title), + content_hex => unpack("H*", $content), + fields => {}, + stamp_hex => unpack("H*", $stamp), + payload_without_stamp_hex => unpack("H*", $payload), + payload_with_stamp_hex => unpack("H*", $stamped_payload), + message_id_hex => unpack("H*", $message_id), + signed_part_hex => unpack("H*", $signed_part), + minimal_lxmf_hex => unpack("H*", $minimal), + stamped_lxmf_hex => unpack("H*", $stamped), +}; + +open my $json_fh, ">:raw", "$output_dir/lxmf_message_vectors.json" + or die "cannot write vector manifest: $!\n"; +print {$json_fh} JSON::PP->new->canonical->pretty->encode($manifest); +close $json_fh; + +print "Generated deterministic LXMessage vectors in $output_dir\n"; + +sub write_hex { + my ($path, $bytes) = @_; + open my $fh, ">:raw", $path or die "cannot write $path: $!\n"; + print {$fh} unpack("H*", $bytes), "\n"; + close $fh; +} diff --git a/tools/verify_examples.pl b/tools/verify_examples.pl new file mode 100755 index 0000000..0f86e67 --- /dev/null +++ b/tools/verify_examples.pl @@ -0,0 +1,88 @@ +#!/usr/bin/env perl + +use strict; +use warnings; +use Digest::SHA qw(sha256); +use FindBin qw($Bin); +use JSON::PP; +use lib "$Bin"; +use LXMF::MessagePack qw(pack_array parse_item); + +my $examples_dir = "$Bin/../examples"; +my $manifest = read_json("$examples_dir/lxmf_message_vectors.json"); +my $minimal = read_hex("$examples_dir/lxmf_message_minimal.hex"); +my $stamped = read_hex("$examples_dir/lxmf_message_stamped.hex"); + +verify_lxmessage("minimal", $minimal, $manifest, 4); +verify_lxmessage("stamped", $stamped, $manifest, 5); + +check(unpack("H*", $minimal) eq $manifest->{minimal_lxmf_hex}, "minimal full bytes match manifest"); +check(unpack("H*", $stamped) eq $manifest->{stamped_lxmf_hex}, "stamped full bytes match manifest"); + +print "All LXMessage test-vector checks passed.\n"; + +sub verify_lxmessage { + my ($name, $bytes, $expected, $expected_count) = @_; + + check(length($bytes) > 96, "$name vector has fixed prefix and payload"); + my $destination_hash = substr($bytes, 0, 16); + my $source_hash = substr($bytes, 16, 16); + my $signature = substr($bytes, 32, 64); + my $packed_payload = substr($bytes, 96); + + check(unpack("H*", $destination_hash) eq $expected->{destination_hash_hex}, "$name destination hash"); + check(unpack("H*", $source_hash) eq $expected->{source_hash_hex}, "$name source hash"); + check(unpack("H*", $signature) eq $expected->{signature_hex}, "$name signature position and length"); + + my $offset = 0; + my $payload = parse_item($packed_payload, \$offset); + check($offset == length($packed_payload), "$name payload consumes all remaining bytes"); + check($payload->{type} eq "array", "$name payload is a MessagePack array"); + check(@{$payload->{items}} == $expected_count, "$name payload has $expected_count entries"); + + my @items = @{$payload->{items}}; + check($items[0]{type} eq "float64", "$name payload[0] timestamp is float64"); + check($items[0]{value} == $expected->{timestamp}, "$name payload[0] timestamp value"); + check($items[1]{type} eq "binary", "$name payload[1] title is binary"); + check(unpack("H*", $items[1]{value}) eq $expected->{title_hex}, "$name payload[1] title value"); + check($items[2]{type} eq "binary", "$name payload[2] content is binary"); + check(unpack("H*", $items[2]{value}) eq $expected->{content_hex}, "$name payload[2] content value"); + check($items[3]{type} eq "map", "$name payload[3] fields is a map"); + check(@{$items[3]{entries}} == 0, "$name payload[3] fields map is empty"); + + my $unstamped_payload = pack_array(map { $_->{raw} } @items[0 .. 3]); + my $message_id = sha256($destination_hash . $source_hash . $unstamped_payload); + my $signed_part = $destination_hash . $source_hash . $unstamped_payload . $message_id; + + check(unpack("H*", $unstamped_payload) eq $expected->{payload_without_stamp_hex}, "$name canonical unstamped payload"); + check(unpack("H*", $message_id) eq $expected->{message_id_hex}, "$name message ID excludes optional stamp"); + check(unpack("H*", $signed_part) eq $expected->{signed_part_hex}, "$name signature input excludes optional stamp"); + + if ($expected_count == 5) { + check($items[4]{type} eq "binary", "$name payload[4] stamp is binary"); + check(unpack("H*", $items[4]{value}) eq $expected->{stamp_hex}, "$name payload[4] stamp value"); + } +} + +sub read_json { + my ($path) = @_; + open my $fh, "<:raw", $path or die "cannot read $path: $!\n"; + local $/; + return JSON::PP->new->decode(<$fh>); +} + +sub read_hex { + my ($path) = @_; + open my $fh, "<:raw", $path or die "cannot read $path: $!\n"; + local $/; + my $hex = <$fh>; + $hex =~ s/\s+//g; + die "$path contains non-hexadecimal data\n" if $hex =~ /[^0-9a-fA-F]/; + return pack("H*", $hex); +} + +sub check { + my ($condition, $description) = @_; + die "not ok - $description\n" unless $condition; + print "ok - $description\n"; +}