After accomplishing Task 2, see Codex_Response... for details.

This commit is contained in:
John Poole 2026-06-06 09:41:48 -07:00
commit d483d40d5c
16 changed files with 888 additions and 0 deletions

View file

@ -0,0 +1,40 @@
---
name: lxmf-update
description: Check upstream LXMF and RNS implementations for changes affecting the LXMF specification, run local verifier tools, and propose spec updates with citations.
---
# LXMF update skill
Use this skill when asked to check whether the LXMF specification is current with upstream LXMF or RNS.
## Sources of truth
Prefer, in order:
1. Upstream LXMF source code at a pinned release or commit.
2. Upstream RNS source code where LXMF behavior depends on Reticulum behavior.
3. Reproducible local traces and test vectors.
4. Upstream prose documentation, only when consistent with source behavior.
## Required workflow
1. Determine the currently pinned upstream LXMF and RNS versions from `SOURCES.md`.
2. Check whether newer upstream releases or commits exist.
3. Identify changed files relevant to:
- message packing
- fields and flags
- destination handling
- identity/signature behavior
- encryption behavior
- propagation
- delivery semantics
- storage format if externally observable
4. Run the verifier tools in `tools/`.
5. Compare observed behavior against `SPEC.md`.
6. Produce a report with:
- changed upstream files
- affected specification sections
- proposed edits
- test-vector impact
- unresolved ambiguities
7. Do not modify `SPEC.md` unless explicitly asked.

16
AGENTS.md Normal file
View file

@ -0,0 +1,16 @@
# AGENTS.md
## Repository purpose
This repository attempts to define an implementation-derived specification for LXMF.
Do not treat upstream prose documentation as authoritative when it conflicts with observed source behavior. Prefer source code, test vectors, and reproducible traces.
## Required discipline
- Distinguish normative behavior from observed implementation behavior.
- Cite exact upstream files, functions, classes, constants, and version tags.
- Do not invent protocol rules.
- When behavior is unclear, mark it as "undetermined" rather than guessing.
- Prefer Perl tools for extraction, comparison, and report generation.
- Keep generated examples reproducible from scripts under tools/.

174
ChatGPT_1.md Normal file
View file

@ -0,0 +1,174 @@
Yes. For Codex, the closest equivalents are:
1. **`AGENTS.md`** for persistent project-wide instructions.
2. **Codex Skills using `SKILL.md`** for reusable workflows, very similar in spirit to the Claude Code file you linked.
3. Optionally **`PLANS.md` / ExecPlan-style documents** for long-running specification work.
The important correction is that Codex does now support **Skills with `SKILL.md`**, not just `AGENTS.md`. OpenAIs Codex documentation says a skill is a directory containing a required `SKILL.md` file, with optional `scripts/`, `references/`, `assets/`, and related support files. Codex initially sees the skill name, description, and path, then loads the full `SKILL.md` only when it decides that skill applies. ([OpenAI Developers][1])
So, yes: you can do essentially the same thing that `thatSFguy` is doing, but I would structure it slightly differently for LXMF.
## Recommended Codex layout
For an LXMF specification repository, I would use something like:
```text
lxmf-specification/
├── AGENTS.md
├── SPEC.md
├── NOTES.md
├── SOURCES.md
├── tools/
│ ├── extract_lxmf_constants.pl
│ ├── compare_upstream_lxmf.pl
│ ├── verify_examples.pl
│ └── make_test_vectors.pl
├── examples/
│ ├── lxmf_message_minimal.hex
│ ├── lxmf_message_signed.hex
│ └── lxmf_propagation_example.hex
├── references/
│ ├── lxmf_source_map.md
│ ├── rns_dependency_map.md
│ └── terminology.md
└── .agents/
└── skills/
└── lxmf-update/
└── SKILL.md
```
## Role of each file
### `AGENTS.md`
Use this for always-on repository rules. Codex reads `AGENTS.md` before doing work, and the docs describe it as the mechanism for project-specific persistent guidance. ([OpenAI Developers][2])
Example use:
```markdown
# AGENTS.md
## Repository purpose
This repository attempts to define an implementation-derived specification for LXMF.
Do not treat upstream prose documentation as authoritative when it conflicts with observed source behavior. Prefer source code, test vectors, and reproducible traces.
## Required discipline
- Distinguish normative behavior from observed implementation behavior.
- Cite exact upstream files, functions, classes, constants, and version tags.
- Do not invent protocol rules.
- When behavior is unclear, mark it as "undetermined" rather than guessing.
- Prefer Perl tools for extraction, comparison, and report generation.
- Keep generated examples reproducible from scripts under tools/.
```
### `.agents/skills/lxmf-update/SKILL.md`
Use this for the repeatable workflow: “check upstream LXMF/RNS, compare against our spec, update citations, run verifiers, propose diffs.”
That is the closest Codex analog to the Claude Code `rns-update/SKILL.md` you found.
A minimal Codex skill might look like:
```markdown
---
name: lxmf-update
description: Check upstream LXMF and RNS implementations for changes affecting the LXMF specification, run local verifier tools, and propose spec updates with citations.
---
# LXMF update skill
Use this skill when asked to check whether the LXMF specification is current with upstream LXMF or RNS.
## Sources of truth
Prefer, in order:
1. Upstream LXMF source code at a pinned release or commit.
2. Upstream RNS source code where LXMF behavior depends on Reticulum behavior.
3. Reproducible local traces and test vectors.
4. Upstream prose documentation, only when consistent with source behavior.
## Required workflow
1. Determine the currently pinned upstream LXMF and RNS versions from `SOURCES.md`.
2. Check whether newer upstream releases or commits exist.
3. Identify changed files relevant to:
- message packing
- fields and flags
- destination handling
- identity/signature behavior
- encryption behavior
- propagation
- delivery semantics
- storage format if externally observable
4. Run the verifier tools in `tools/`.
5. Compare observed behavior against `SPEC.md`.
6. Produce a report with:
- changed upstream files
- affected specification sections
- proposed edits
- test-vector impact
- unresolved ambiguities
7. Do not modify `SPEC.md` unless explicitly asked.
```
Codex supports skills in the CLI, IDE extension, and Codex app, and the documented repository skill location is `.agents/skills` under the current directory or repository root. ([OpenAI Developers][1])
### `PLANS.md`
For your use case, I would also add a planning document, because “derive a protocol specification from implementation” is not a one-shot coding task. OpenAIs Codex cookbook describes using `PLANS.md` / ExecPlans for long-running, multi-stage work, with `AGENTS.md` telling Codex when to use it. ([OpenAI Developers][3])
For LXMF, `PLANS.md` could define the required sections of each investigation:
```markdown
# LXMF Specification Investigation Plan
Each investigation must include:
1. Question being answered.
2. Upstream files inspected.
3. Exact version or commit.
4. Relevant classes/functions/constants.
5. Observed behavior.
6. Whether behavior is normative, inferred, or unresolved.
7. Test vectors needed.
8. SPEC.md sections affected.
9. Open questions.
```
## My recommendation
For your goal, I would not start by asking Codex to “write the LXMF specification.” That is too broad and likely to produce confident prose ahead of evidence.
I would instead give Codex narrow tasks such as:
```text
Using the lxmf-update skill, inspect upstream LXMF and determine the exact serialized structure of an LXMessage. Do not edit SPEC.md. Produce a report listing every field, its source-code origin, and whether its behavior is confirmed by a test vector.
```
Then:
```text
Create Perl tooling under tools/ that extracts or verifies the LXMF message packing behavior described in the report. Generate at least one reproducible test vector.
```
Then:
```text
Update SPEC.md section "LXMessage Wire Format" using only behavior confirmed by source inspection or verifier output. Mark all unverified behavior as unresolved.
```
## Bottom line
Yes, you can accomplish the same kind of workflow with Codex. In fact, Codex gives you both mechanisms:
`AGENTS.md` for standing repo rules, and `SKILL.md` under `.agents/skills/...` for a reusable “LXMF update / spec maintenance” workflow. OpenAIs Codex docs explicitly describe customization as a combination of `AGENTS.md`, skills, MCP, memories, and subagents; for your use case, `AGENTS.md` plus a `SKILL.md` skill is the right starting point. ([OpenAI Developers][4])
[1]: https://developers.openai.com/codex/skills "Agent Skills Codex | OpenAI Developers"
[2]: https://developers.openai.com/codex/guides/agents-md "Custom instructions with AGENTS.md Codex | OpenAI Developers"
[3]: https://developers.openai.com/cookbook/articles/codex_exec_plans "Using PLANS.md for multi-hour problem solving"
[4]: https://developers.openai.com/codex/concepts/customization "Customization Codex | OpenAI Developers"

8
Codex_1.md Normal file
View file

@ -0,0 +1,8 @@
This directory: /usr/local/src/reticulum/LXMF
contains my GitHub fork (https://github.com/jlpoolen/LXMF) of Mark Qvist's: https://github.com/markqvist/LXMF which, incidentally, is a public mirror of some git server served up through Reticulum -- the source is not provided.
I created an empty project in my Forgejo server: LXMF-specification at https://salemdata.net/repo/jlpoole/LMXF-specification
and I added my Forgejo instance as a remote repository identified by "forgejo".
I added some files based on a ChatGPT chat, see ChatGPT_1.md.
So, for your first task: Using the lxmf-update skill, inspect upstream LXMF and determine the exact serialized structure of an LXMessage. Do not edit SPEC.md. Produce a report listing every field, its source-code origin, and whether its behavior is confirmed by a test vector.

83
Codex_Response_1.md Normal file
View file

@ -0,0 +1,83 @@
No, I dont need `SPEC.md` or `SOURCES.md` for this first report. I used the checked-out LXMF source directly. I did not edit `SPEC.md` or any other file.
**LXMessage Serialized Structure**
Source inspected: LXMF commit `fab12ad9bf9f997797034950f289fe41a79dcf5a`
Core `lxmf_bytes` layout:
```text
destination_hash || source_hash || signature || msgpack_payload
```
Where `msgpack_payload` is:
```text
[
timestamp,
title,
content,
fields,
optional_stamp
]
```
The fifth element is only present when a normal message stamp is generated and appended.
| Field | Serialized position | Type / size | Source origin | Test vector confirmed |
|---|---:|---|---|---|
| `destination_hash` | bytes `0..15` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:383), length from [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:40) and RNS 128-bit truncated hashes in [Reticulum.py](/usr/local/src/reticulum/Reticulum/RNS/Reticulum.py:146) | No |
| `source_hash` | bytes `16..31` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:384) | No |
| `signature` | bytes `32..95` | 64 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:385); RNS signature length from [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:59) and [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:80) | No |
| `msgpack_payload` | bytes `96..end` | msgpack list | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362), packed at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:381) | No |
| `timestamp` | payload index `0` | Python float, normally msgpack float64 on this RNS umsgpack | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:357), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362); float64 behavior from [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:1177) and [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:325) | No |
| `title` | payload index `1` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:130), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:193), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No |
| `content` | payload index `2` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:135), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:202), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No |
| `fields` | payload index `3` | dict, default `{}` | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:215), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No |
| `stamp` | payload index `4`, optional | bytes, only if generated | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:371), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:373); extracted on unpack at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754) | No |
**Hash And Signature**
`message_id` / `hash` is:
```text
SHA256(destination_hash || source_hash || msgpack([timestamp, title, content, fields]))
```
Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:364), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:368), RNS SHA-256 at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:238).
The signature is over:
```text
destination_hash || source_hash || msgpack([timestamp, title, content, fields]) || message_id
```
Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:375), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), RNS signing at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:767).
Important: the optional normal `stamp` is excluded from both `message_id` and signature. Unpack removes payload index `4` before recomputing hash/signature material: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754).
**Outer Representations**
Direct/link/resource delivery uses full `self.packed` unchanged: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:636), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:654).
Opportunistic packet delivery omits the destination hash from the packet payload because the RNS packet destination already implies it:
```text
source_hash || signature || msgpack_payload
```
Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:633).
Propagated and paper messages wrap/encrypt everything after the destination hash:
```text
destination_hash || encrypted(source_hash || signature || msgpack_payload)
```
Source: propagated [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:429), paper [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:449). Propagation transit additionally msgpacks `[time.time(), [lxmf_data]]`: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:436).
**Notes**
The README is not authoritative here: it describes payload order as timestamp, content, title, fields, but current code serializes timestamp, title, content, fields.
No test vector confirmation exists in this checkout. There is no `tools/` directory and no tracked `SPEC.md` or `SOURCES.md`, so every “Test vector confirmed” entry above is marked `No`.

83
Codex_Response_2.md Normal file
View file

@ -0,0 +1,83 @@
No, I dont need `SPEC.md` or `SOURCES.md` for this first report. I used the checked-out LXMF source directly. I did not edit `SPEC.md` or any other file.
**LXMessage Serialized Structure**
Source inspected: LXMF commit `fab12ad9bf9f997797034950f289fe41a79dcf5a`
Core `lxmf_bytes` layout:
```text
destination_hash || source_hash || signature || msgpack_payload
```
Where `msgpack_payload` is:
```text
[
timestamp,
title,
content,
fields,
optional_stamp
]
```
The fifth element is only present when a normal message stamp is generated and appended.
| Field | Serialized position | Type / size | Source origin | Test vector confirmed |
|---|---:|---|---|---|
| `destination_hash` | bytes `0..15` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:383), length from [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:40) and RNS 128-bit truncated hashes in [Reticulum.py](/usr/local/src/reticulum/Reticulum/RNS/Reticulum.py:146) | No |
| `source_hash` | bytes `16..31` | 16 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:384) | No |
| `signature` | bytes `32..95` | 64 bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:385); RNS signature length from [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:59) and [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:80) | No |
| `msgpack_payload` | bytes `96..end` | msgpack list | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362), packed at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:381) | No |
| `timestamp` | payload index `0` | Python float, normally msgpack float64 on this RNS umsgpack | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:357), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362); float64 behavior from [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:1177) and [umsgpack.py](/usr/local/src/reticulum/Reticulum/RNS/vendor/umsgpack.py:325) | No |
| `title` | payload index `1` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:130), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:193), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No |
| `content` | payload index `2` | bytes | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:135), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:202), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No |
| `fields` | payload index `3` | dict, default `{}` | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:215), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:362) | No |
| `stamp` | payload index `4`, optional | bytes, only if generated | [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:371), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:373); extracted on unpack at [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754) | No |
**Hash And Signature**
`message_id` / `hash` is:
```text
SHA256(destination_hash || source_hash || msgpack([timestamp, title, content, fields]))
```
Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:364), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:368), RNS SHA-256 at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:238).
The signature is over:
```text
destination_hash || source_hash || msgpack([timestamp, title, content, fields]) || message_id
```
Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:375), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:378), RNS signing at [Identity.py](/usr/local/src/reticulum/Reticulum/RNS/Identity.py:767).
Important: the optional normal `stamp` is excluded from both `message_id` and signature. Unpack removes payload index `4` before recomputing hash/signature material: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:754).
**Outer Representations**
Direct/link/resource delivery uses full `self.packed` unchanged: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:636), [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:654).
Opportunistic packet delivery omits the destination hash from the packet payload because the RNS packet destination already implies it:
```text
source_hash || signature || msgpack_payload
```
Source: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:633).
Propagated and paper messages wrap/encrypt everything after the destination hash:
```text
destination_hash || encrypted(source_hash || signature || msgpack_payload)
```
Source: propagated [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:429), paper [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:449). Propagation transit additionally msgpacks `[time.time(), [lxmf_data]]`: [LXMessage.py](/usr/local/src/reticulum/LXMF/LXMF/LXMessage.py:436).
**Notes**
The README is not authoritative here: it describes payload order as timestamp, content, title, fields, but current code serializes timestamp, title, content, fields.
No test vector confirmation exists in this checkout. There is no `tools/` directory and no tracked `SPEC.md` or `SOURCES.md`, so every “Test vector confirmed” entry above is marked `No`.

27
Codex_Response_3.md Normal file
View file

@ -0,0 +1,27 @@
Task #2 completed in `/usr/local/src/reticulum/LXMF`.
Created:
- `tools/make_test_vectors.pl`
- `tools/verify_examples.pl`
- `tools/LXMF/MessagePack.pm`
- `tools/README.md`
- `examples/lxmf_message_minimal.hex`
- `examples/lxmf_message_stamped.hex`
- `examples/lxmf_message_vectors.json`
The dependency-free Perl tooling verifies:
- Fixed hash/signature positions
- Payload order and MessagePack types
- Message ID calculation
- Optional stamp exclusion from message ID and signature input
Validation passed:
```text
Reticulum umsgpack byte-for-byte cross-check passed
All LXMessage test-vector checks passed.
```
The vectors regenerate deterministically. Signature bytes are documented placeholders; Ed25519 validation is not claimed. `SPEC.md` was not created or modified.

13
PLANS.md Normal file
View file

@ -0,0 +1,13 @@
# LXMF Specification Investigation Plan
Each investigation must include:
1. Question being answered.
2. Upstream files inspected.
3. Exact version or commit.
4. Relevant classes/functions/constants.
5. Observed behavior.
6. Whether behavior is normative, inferred, or unresolved.
7. Test vectors needed.
8. SPEC.md sections affected.
9. Open questions.

View file

@ -0,0 +1 @@
000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980

View file

@ -0,0 +1 @@
000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f95cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980c420a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf

View file

@ -0,0 +1,26 @@
{
"content_hex" : "44657465726d696e6973746963204c584d4620626f6479",
"description" : "Deterministic LXMessage packing vectors; signature bytes are placeholders",
"destination_hash_hex" : "000102030405060708090a0b0c0d0e0f",
"fields" : {},
"format" : "destination_hash || source_hash || signature || msgpack_payload",
"message_id_hex" : "55f497f6f2ea5d858eaff541a9fa77f9d5bdf57be75be4a94df65d1f9a7ffea3",
"minimal_lxmf_hex" : "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980",
"payload_order" : [
"timestamp",
"title",
"content",
"fields",
"optional_stamp"
],
"payload_with_stamp_hex" : "95cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980c420a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf",
"payload_without_stamp_hex" : "94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980",
"signature_hex" : "202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f",
"signature_valid" : false,
"signed_part_hex" : "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f94cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f64798055f497f6f2ea5d858eaff541a9fa77f9d5bdf57be75be4a94df65d1f9a7ffea3",
"source_hash_hex" : "101112131415161718191a1b1c1d1e1f",
"stamp_hex" : "a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf",
"stamped_lxmf_hex" : "000102030405060708090a0b0c0d0e0f101112131415161718191a1b1c1d1e1f202122232425262728292a2b2c2d2e2f303132333435363738393a3b3c3d3e3f404142434445464748494a4b4c4d4e4f505152535455565758595a5b5c5d5e5f95cb41d954fc40100000c40a54657374207469746c65c41744657465726d696e6973746963204c584d4620626f647980c420a0a1a2a3a4a5a6a7a8a9aaabacadaeafb0b1b2b3b4b5b6b7b8b9babbbcbdbebf",
"timestamp" : 1700000000.25,
"title_hex" : "54657374207469746c65"
}

39
grep_results.txt Normal file
View file

@ -0,0 +1,39 @@
./LXMF/Utilities/lxmd.py:DEFFERED_JOBS_DELAY = 10
./LXMF/Utilities/lxmd.py:def create_default_config(configpath):
./LXMF/Utilities/lxmd.py:def apply_config():
./LXMF/Utilities/lxmd.py:def lxmf_delivery(lxm):
./LXMF/Utilities/lxmd.py:def program_setup(configdir = None, rnsconfigdir = None, run_pn = False, on_inbound = None, verbosity = 0, quietness = 0, service = False):
./LXMF/Utilities/lxmd.py:def jobs():
./LXMF/Utilities/lxmd.py:def deferred_start_jobs():
./LXMF/Utilities/lxmd.py:def _request_sync(identity, destination_hash, remote_identity, timeout=15, exit_on_fail=False):
./LXMF/Utilities/lxmd.py:def request_sync(target, remote=None, configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, timeout=15, identity_path=None):
./LXMF/Utilities/lxmd.py:def _request_unpeer(identity, destination_hash, remote_identity, timeout=15, exit_on_fail=False):
./LXMF/Utilities/lxmd.py:def request_unpeer(target, remote=None, configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, timeout=15, identity_path=None):
./LXMF/Utilities/lxmd.py:def query_status(identity, remote_identity=None, timeout=5, exit_on_fail=False):
./LXMF/Utilities/lxmd.py:def get_status(remote=None, configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, timeout=5,
./LXMF/Utilities/lxmd.py:def _get_target_identity(remote=None, timeout=5):
./LXMF/Utilities/lxmd.py:def _remote_init(configdir=None, rnsconfigdir=None, verbosity=0, quietness=0, identity_path=None):
./LXMF/Utilities/lxmd.py:def main():
./LXMF/LXMF.py:def display_name_from_app_data(app_data=None):
./LXMF/LXMF.py:def stamp_cost_from_app_data(app_data=None):
./LXMF/LXMF.py:def compression_support_from_app_data(app_data=None):
./LXMF/LXMF.py:def pn_name_from_app_data(app_data=None):
./LXMF/LXMF.py:def pn_stamp_cost_from_app_data(app_data=None):
./LXMF/LXMF.py:def pn_announce_data_is_valid(data):
./LXMF/LXStamper.py:def worker_context(ctx, stamp_cost, workblock, message_id):
./LXMF/LXStamper.py:def stamp_workblock(material, expand_rounds=WORKBLOCK_EXPAND_ROUNDS):
./LXMF/LXStamper.py:def stamp_value(workblock, stamp):
./LXMF/LXStamper.py:def stamp_valid(stamp, target_cost, workblock):
./LXMF/LXStamper.py:def validate_peering_key(peering_id, peering_key, target_cost):
./LXMF/LXStamper.py:def validate_pn_stamp(transient_data, target_cost):
./LXMF/LXStamper.py:def validate_pn_stamps_job_simple(transient_list, target_cost):
./LXMF/LXStamper.py:def validate_pn_stamps_job_multip(transient_list, target_cost):
./LXMF/LXStamper.py:def validate_pn_stamps(transient_list, target_cost):
./LXMF/LXStamper.py:def generate_stamp(message_id, stamp_cost, expand_rounds=WORKBLOCK_EXPAND_ROUNDS):
./LXMF/LXStamper.py:def cancel_work(message_id):
./LXMF/LXStamper.py:def job_simple(stamp_cost, workblock, message_id):
./LXMF/LXStamper.py:def job_linux_managed(stamp_cost, workblock, message_id):
./LXMF/LXStamper.py:def job_linux(stamp_cost, workblock, message_id):
./LXMF/LXStamper.py:def job_android(stamp_cost, workblock, message_id):
./docs/example_receiver.py:def delivery_callback(message):
./docs/example_sender.py:def delivery_callback(message):

181
tools/LXMF/MessagePack.pm Normal file
View file

@ -0,0 +1,181 @@
package LXMF::MessagePack;
use strict;
use warnings;
use Exporter 'import';
our @EXPORT_OK = qw(
pack_array
pack_bin
pack_float64
pack_map
parse_item
);
sub pack_float64 {
my ($value) = @_;
return "\xcb" . pack("d>", $value);
}
sub pack_bin {
my ($value) = @_;
my $length = length($value);
return "\xc4" . pack("C", $length) . $value if $length <= 0xff;
return "\xc5" . pack("n", $length) . $value if $length <= 0xffff;
return "\xc6" . pack("N", $length) . $value;
}
sub pack_array {
my (@items) = @_;
my $length = scalar @items;
my $header;
if ($length <= 15) {
$header = pack("C", 0x90 | $length);
}
elsif ($length <= 0xffff) {
$header = "\xdc" . pack("n", $length);
}
else {
$header = "\xdd" . pack("N", $length);
}
return $header . join("", @items);
}
sub pack_map {
my (@entries) = @_;
die "pack_map requires key/value byte-string pairs\n" if @entries % 2;
my $length = @entries / 2;
my $header;
if ($length <= 15) {
$header = pack("C", 0x80 | $length);
}
elsif ($length <= 0xffff) {
$header = "\xde" . pack("n", $length);
}
else {
$header = "\xdf" . pack("N", $length);
}
return $header . join("", @entries);
}
sub _take {
my ($bytes, $offset_ref, $length) = @_;
die "truncated MessagePack item\n" if $$offset_ref + $length > length($bytes);
my $value = substr($bytes, $$offset_ref, $length);
$$offset_ref += $length;
return $value;
}
sub _parse_length {
my ($bytes, $offset_ref, $width) = @_;
my $raw = _take($bytes, $offset_ref, $width);
return unpack("C", $raw) if $width == 1;
return unpack("n", $raw) if $width == 2;
return unpack("N", $raw);
}
sub parse_item {
my ($bytes, $offset_ref) = @_;
$offset_ref //= \my $offset;
$$offset_ref //= 0;
my $start = $$offset_ref;
my $code = unpack("C", _take($bytes, $offset_ref, 1));
my $item = { code => $code };
if ($code <= 0x7f) {
@$item{qw(type value)} = ("integer", $code);
}
elsif ($code >= 0xe0) {
@$item{qw(type value)} = ("integer", $code - 256);
}
elsif (($code & 0xf0) == 0x90) {
_parse_array($bytes, $offset_ref, $item, $code & 0x0f);
}
elsif (($code & 0xf0) == 0x80) {
_parse_map($bytes, $offset_ref, $item, $code & 0x0f);
}
elsif (($code & 0xe0) == 0xa0) {
my $length = $code & 0x1f;
@$item{qw(type value)} = ("string", _take($bytes, $offset_ref, $length));
}
elsif ($code == 0xc0) {
$item->{type} = "nil";
$item->{value} = undef;
}
elsif ($code == 0xc2 || $code == 0xc3) {
@$item{qw(type value)} = ("boolean", $code == 0xc3 ? 1 : 0);
}
elsif ($code == 0xca) {
@$item{qw(type value)} = ("float32", unpack("f>", _take($bytes, $offset_ref, 4)));
}
elsif ($code == 0xcb) {
@$item{qw(type value)} = ("float64", unpack("d>", _take($bytes, $offset_ref, 8)));
}
elsif ($code >= 0xc4 && $code <= 0xc6) {
my $width = 1 << ($code - 0xc4);
my $length = _parse_length($bytes, $offset_ref, $width);
@$item{qw(type value)} = ("binary", _take($bytes, $offset_ref, $length));
}
elsif ($code >= 0xcc && $code <= 0xcf) {
my @formats = ("C", "n", "N", "Q>");
my @widths = (1, 2, 4, 8);
my $index = $code - 0xcc;
@$item{qw(type value)} = (
"integer",
unpack($formats[$index], _take($bytes, $offset_ref, $widths[$index])),
);
}
elsif ($code >= 0xd0 && $code <= 0xd3) {
my @formats = ("c", "s>", "l>", "q>");
my @widths = (1, 2, 4, 8);
my $index = $code - 0xd0;
@$item{qw(type value)} = (
"integer",
unpack($formats[$index], _take($bytes, $offset_ref, $widths[$index])),
);
}
elsif ($code == 0xd9 || $code == 0xda || $code == 0xdb) {
my $width = 1 << ($code - 0xd9);
my $length = _parse_length($bytes, $offset_ref, $width);
@$item{qw(type value)} = ("string", _take($bytes, $offset_ref, $length));
}
elsif ($code == 0xdc || $code == 0xdd) {
my $width = $code == 0xdc ? 2 : 4;
_parse_array($bytes, $offset_ref, $item, _parse_length($bytes, $offset_ref, $width));
}
elsif ($code == 0xde || $code == 0xdf) {
my $width = $code == 0xde ? 2 : 4;
_parse_map($bytes, $offset_ref, $item, _parse_length($bytes, $offset_ref, $width));
}
else {
die sprintf("unsupported MessagePack code 0x%02x at offset %d\n", $code, $start);
}
$item->{start} = $start;
$item->{end} = $$offset_ref;
$item->{raw} = substr($bytes, $start, $$offset_ref - $start);
return $item;
}
sub _parse_array {
my ($bytes, $offset_ref, $item, $length) = @_;
$item->{type} = "array";
$item->{items} = [map { parse_item($bytes, $offset_ref) } 1 .. $length];
}
sub _parse_map {
my ($bytes, $offset_ref, $item, $length) = @_;
$item->{type} = "map";
$item->{entries} = [
map { [parse_item($bytes, $offset_ref), parse_item($bytes, $offset_ref)] } 1 .. $length
];
}
1;

32
tools/README.md Normal file
View file

@ -0,0 +1,32 @@
# LXMessage Test-Vector Tooling
The scripts in this directory verify the implementation-derived LXMessage
packing behavior described in `Code_Response_1.md`.
They use only Perl core modules. The local `LXMF::MessagePack` module implements
the MessagePack subset needed to generate the vectors and parse their complete
payloads.
Generate deterministic vectors:
```sh
perl tools/make_test_vectors.pl
```
Verify the vectors:
```sh
perl tools/verify_examples.pl
```
The vectors confirm:
- the 16-byte destination hash, 16-byte source hash, and 64-byte signature
positions;
- payload order `[timestamp, title, content, fields, optional_stamp]`;
- float64 timestamp and binary title/content/stamp encodings;
- the message ID calculation;
- exclusion of the optional stamp from the message ID and signature input.
The signature bytes are deterministic placeholders. These vectors do not claim
to verify Ed25519 signing or signature validation.

76
tools/make_test_vectors.pl Executable file
View file

@ -0,0 +1,76 @@
#!/usr/bin/env perl
use strict;
use warnings;
use Digest::SHA qw(sha256);
use FindBin qw($Bin);
use File::Path qw(make_path);
use JSON::PP;
use lib "$Bin";
use LXMF::MessagePack qw(pack_array pack_bin pack_float64 pack_map);
my $output_dir = "$Bin/../examples";
make_path($output_dir);
my $destination_hash = pack("C*", 0x00 .. 0x0f);
my $source_hash = pack("C*", 0x10 .. 0x1f);
my $signature = pack("C*", 0x20 .. 0x5f);
my $stamp = pack("C*", 0xa0 .. 0xbf);
my $timestamp = 1_700_000_000.25;
my $title = "Test title";
my $content = "Deterministic LXMF body";
my $fields = pack_map();
my @required_items = (
pack_float64($timestamp),
pack_bin($title),
pack_bin($content),
$fields,
);
my $payload = pack_array(@required_items);
my $stamped_payload = pack_array(@required_items, pack_bin($stamp));
my $message_id = sha256($destination_hash . $source_hash . $payload);
my $signed_part = $destination_hash . $source_hash . $payload . $message_id;
my $minimal = $destination_hash . $source_hash . $signature . $payload;
my $stamped = $destination_hash . $source_hash . $signature . $stamped_payload;
write_hex("$output_dir/lxmf_message_minimal.hex", $minimal);
write_hex("$output_dir/lxmf_message_stamped.hex", $stamped);
my $manifest = {
description => "Deterministic LXMessage packing vectors; signature bytes are placeholders",
format => "destination_hash || source_hash || signature || msgpack_payload",
payload_order => [qw(timestamp title content fields optional_stamp)],
destination_hash_hex => unpack("H*", $destination_hash),
source_hash_hex => unpack("H*", $source_hash),
signature_hex => unpack("H*", $signature),
signature_valid => JSON::PP::false,
timestamp => $timestamp,
title_hex => unpack("H*", $title),
content_hex => unpack("H*", $content),
fields => {},
stamp_hex => unpack("H*", $stamp),
payload_without_stamp_hex => unpack("H*", $payload),
payload_with_stamp_hex => unpack("H*", $stamped_payload),
message_id_hex => unpack("H*", $message_id),
signed_part_hex => unpack("H*", $signed_part),
minimal_lxmf_hex => unpack("H*", $minimal),
stamped_lxmf_hex => unpack("H*", $stamped),
};
open my $json_fh, ">:raw", "$output_dir/lxmf_message_vectors.json"
or die "cannot write vector manifest: $!\n";
print {$json_fh} JSON::PP->new->canonical->pretty->encode($manifest);
close $json_fh;
print "Generated deterministic LXMessage vectors in $output_dir\n";
sub write_hex {
my ($path, $bytes) = @_;
open my $fh, ">:raw", $path or die "cannot write $path: $!\n";
print {$fh} unpack("H*", $bytes), "\n";
close $fh;
}

88
tools/verify_examples.pl Executable file
View file

@ -0,0 +1,88 @@
#!/usr/bin/env perl
use strict;
use warnings;
use Digest::SHA qw(sha256);
use FindBin qw($Bin);
use JSON::PP;
use lib "$Bin";
use LXMF::MessagePack qw(pack_array parse_item);
my $examples_dir = "$Bin/../examples";
my $manifest = read_json("$examples_dir/lxmf_message_vectors.json");
my $minimal = read_hex("$examples_dir/lxmf_message_minimal.hex");
my $stamped = read_hex("$examples_dir/lxmf_message_stamped.hex");
verify_lxmessage("minimal", $minimal, $manifest, 4);
verify_lxmessage("stamped", $stamped, $manifest, 5);
check(unpack("H*", $minimal) eq $manifest->{minimal_lxmf_hex}, "minimal full bytes match manifest");
check(unpack("H*", $stamped) eq $manifest->{stamped_lxmf_hex}, "stamped full bytes match manifest");
print "All LXMessage test-vector checks passed.\n";
sub verify_lxmessage {
my ($name, $bytes, $expected, $expected_count) = @_;
check(length($bytes) > 96, "$name vector has fixed prefix and payload");
my $destination_hash = substr($bytes, 0, 16);
my $source_hash = substr($bytes, 16, 16);
my $signature = substr($bytes, 32, 64);
my $packed_payload = substr($bytes, 96);
check(unpack("H*", $destination_hash) eq $expected->{destination_hash_hex}, "$name destination hash");
check(unpack("H*", $source_hash) eq $expected->{source_hash_hex}, "$name source hash");
check(unpack("H*", $signature) eq $expected->{signature_hex}, "$name signature position and length");
my $offset = 0;
my $payload = parse_item($packed_payload, \$offset);
check($offset == length($packed_payload), "$name payload consumes all remaining bytes");
check($payload->{type} eq "array", "$name payload is a MessagePack array");
check(@{$payload->{items}} == $expected_count, "$name payload has $expected_count entries");
my @items = @{$payload->{items}};
check($items[0]{type} eq "float64", "$name payload[0] timestamp is float64");
check($items[0]{value} == $expected->{timestamp}, "$name payload[0] timestamp value");
check($items[1]{type} eq "binary", "$name payload[1] title is binary");
check(unpack("H*", $items[1]{value}) eq $expected->{title_hex}, "$name payload[1] title value");
check($items[2]{type} eq "binary", "$name payload[2] content is binary");
check(unpack("H*", $items[2]{value}) eq $expected->{content_hex}, "$name payload[2] content value");
check($items[3]{type} eq "map", "$name payload[3] fields is a map");
check(@{$items[3]{entries}} == 0, "$name payload[3] fields map is empty");
my $unstamped_payload = pack_array(map { $_->{raw} } @items[0 .. 3]);
my $message_id = sha256($destination_hash . $source_hash . $unstamped_payload);
my $signed_part = $destination_hash . $source_hash . $unstamped_payload . $message_id;
check(unpack("H*", $unstamped_payload) eq $expected->{payload_without_stamp_hex}, "$name canonical unstamped payload");
check(unpack("H*", $message_id) eq $expected->{message_id_hex}, "$name message ID excludes optional stamp");
check(unpack("H*", $signed_part) eq $expected->{signed_part_hex}, "$name signature input excludes optional stamp");
if ($expected_count == 5) {
check($items[4]{type} eq "binary", "$name payload[4] stamp is binary");
check(unpack("H*", $items[4]{value}) eq $expected->{stamp_hex}, "$name payload[4] stamp value");
}
}
sub read_json {
my ($path) = @_;
open my $fh, "<:raw", $path or die "cannot read $path: $!\n";
local $/;
return JSON::PP->new->decode(<$fh>);
}
sub read_hex {
my ($path) = @_;
open my $fh, "<:raw", $path or die "cannot read $path: $!\n";
local $/;
my $hex = <$fh>;
$hex =~ s/\s+//g;
die "$path contains non-hexadecimal data\n" if $hex =~ /[^0-9a-fA-F]/;
return pack("H*", $hex);
}
sub check {
my ($condition, $description) = @_;
die "not ok - $description\n" unless $condition;
print "ok - $description\n";
}