Compare commits
8 commits
76496efcb3
...
65a3b73014
| Author | SHA1 | Date | |
|---|---|---|---|
| 65a3b73014 | |||
| 36bc86baa9 | |||
| 3f3f3668f1 | |||
| 67e69d5ce7 | |||
| 41a5c86a8e | |||
| f47cc42d10 | |||
| e771515b43 | |||
| b7232b5e1e |
24 changed files with 3130 additions and 8 deletions
4
migration/.gitignore
vendored
Normal file
4
migration/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
*.sqlite
|
||||
*.so
|
||||
__pycache__/
|
||||
protocol_core/build/
|
||||
47
migration/Codex_prompt_20250616_1449.md
Normal file
47
migration/Codex_prompt_20250616_1449.md
Normal file
|
|
@ -0,0 +1,47 @@
|
|||
I am trying to extract the protocol-only portion of this BLE Reticulum implementation into a future C++ library callable from Python.
|
||||
|
||||
Use /usr/local/src/ble-reticulum/src/ble_symbols_20260516_1429.txt as the master inventory. For every symbol listed in that file, inspect the corresponding source code and classify it as:
|
||||
|
||||
[CORE]
|
||||
Protocol logic that may belong in a future C++/Rust library.
|
||||
|
||||
[GLUE]
|
||||
Python Reticulum integration that should probably stay Python initially.
|
||||
|
||||
[PLATFORM]
|
||||
Linux/BLE/BlueZ/Bleak/bluezero/DBus/GATT/scanning/advertising code.
|
||||
|
||||
[TEST]
|
||||
Examples, proof scripts, diagnostics, command-line tools, or test harnesses.
|
||||
|
||||
[UNKNOWN]
|
||||
Not enough information to classify safely.
|
||||
|
||||
For each symbol, produce a Markdown table with:
|
||||
|
||||
- file path
|
||||
- symbol name
|
||||
- line number
|
||||
- tag
|
||||
- phase-1 C++ candidate: yes/no
|
||||
- rationale
|
||||
- important callers, if easily found
|
||||
- important callees, if easily found
|
||||
|
||||
Do not modify files.
|
||||
|
||||
At the end, produce:
|
||||
|
||||
1. A "Phase 1 C++ extraction candidates" list.
|
||||
2. A "Do not port yet" list.
|
||||
3. A short explanation of where the Python/C++ boundary should probably be.
|
||||
4. Any symbols whose classification depends on Reticulum internals.
|
||||
|
||||
For information about where reticulum is installed, see /usr/local/src/ble-reticulum/reticulum_info.txt
|
||||
|
||||
The main tree is /usr/local/src/ble-reticulum which is pulled from my Forgejo instance. I have created a branch "c++migration" which is the current branch.
|
||||
|
||||
Treat ble_dual_node_echo.py as an acceptance test/harness, not as the source of the protocol-core design.
|
||||
|
||||
Prefer conservative classification. If a symbol touches RNS.Transport, RNS.Interface, RNS.Link, BleakClient, bluezero, DBus, sockets, threads, or OS-specific objects, do not mark it as phase-1 C++ unless the protocol logic can be cleanly separated from those dependencies.
|
||||
|
||||
154
migration/Codex_response_20260616_1514.md
Normal file
154
migration/Codex_response_20260616_1514.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
Read-only inspection completed. Note: the prompt’s Reticulum info path was not present as named; I used `/usr/local/src/ble-reticulum/reticulum_info_20260516_1444.txt`. Installed RNS is `rns 1.1.3` at `/home/jlpoole/rnsenv/lib/python3.13/site-packages/RNS`.
|
||||
|
||||
**BLEFragmentation.py**
|
||||
|
||||
| file path | symbol | line | tag | phase-1 C++ candidate | rationale | important callers | important callees |
|
||||
|---|---:|---:|---|---|---|---|---|
|
||||
| `src/ble_reticulum/BLEFragmentation.py` | `BLEFragmenter` | 52 | [CORE] | yes | Pure BLE packet fragmentation format; RNS only logging. | `BLEInterface._mtu_negotiated_callback`, `_handle_identity_handshake`, `BLEPeerInterface.process_outgoing` | `struct.pack` |
|
||||
| same | `BLEFragmenter.__init__` | 68 | [CORE] | yes | MTU/header sizing logic. | fragmenter construction | none |
|
||||
| same | `BLEFragmenter.fragment_packet` | 82 | [CORE] | yes | Core packet-to-fragments encoding. | `BLEPeerInterface.process_outgoing` | `struct.pack` |
|
||||
| same | `BLEFragmenter.get_fragment_overhead` | 158 | [CORE] | yes | Pure sizing/overhead calculation. | tests/diagnostics likely | none |
|
||||
| same | `BLEReassembler` | 176 | [CORE] | yes | Core fragment state machine; RNS only logging. | `BLEInterface` receive paths | `_reassemble`, `cleanup_stale_buffers` |
|
||||
| same | `BLEReassembler.__init__` | 187 | [CORE] | yes | Protocol reassembly state. | reassembler construction | none |
|
||||
| same | `BLEReassembler.receive_fragment` | 205 | [CORE] | yes | Core fragment validation/dedup/reassembly trigger. | `_handle_ble_data`, `handle_peripheral_data` | `struct.unpack`, `_reassemble` |
|
||||
| same | `BLEReassembler._reassemble` | 380 | [CORE] | yes | Pure ordered fragment join. | `receive_fragment` | none |
|
||||
| same | `BLEReassembler.cleanup_stale_buffers` | 402 | [CORE] | yes | Timeout policy for incomplete protocol frames. | cleanup timer, receive paths | `time.time` |
|
||||
| same | `BLEReassembler.get_statistics` | 429 | [CORE] | yes | Protocol stats. | receive/logging paths | none |
|
||||
| same | `BLEReassembler.reset_statistics` | 443 | [CORE] | yes | Protocol stats reset. | tests likely | none |
|
||||
| same | `HDLCFramer` | 450 | [CORE] | yes | Pure byte-stuffing frame format. | not obviously used | static methods |
|
||||
| same | `HDLCFramer.frame_packet` | 464 | [CORE] | yes | Pure HDLC-style escaping. | tests/possible future use | none |
|
||||
| same | `HDLCFramer.deframe_packet` | 491 | [CORE] | yes | Pure HDLC-style unescaping/validation. | tests/possible future use | none |
|
||||
|
||||
**BLEInterface.py**
|
||||
|
||||
| file path | symbol | line | tag | phase-1 C++ candidate | rationale | important callers | important callees |
|
||||
|---|---:|---:|---|---|---|---|---|
|
||||
| `src/ble_reticulum/BLEInterface.py` | `DiscoveredPeer` | 118 | [PLATFORM] | no | BLE discovery/connection scoring state. | `_device_discovered_callback`, `_select_peers_to_connect` | `time.time` |
|
||||
| same | `DiscoveredPeer.__init__` | 146 | [PLATFORM] | no | Stores BLE address/RSSI/timing. | discovery callback | `time.time` |
|
||||
| same | `update_rssi` | 167 | [PLATFORM] | no | Discovery cache update. | discovery callback | `time.time` |
|
||||
| same | `record_connection_attempt` | 172 | [PLATFORM] | no | BLE connection policy state. | discovery callback | `time.time` |
|
||||
| same | `record_connection_success` | 177 | [PLATFORM] | no | BLE connection policy state. | `_record_connection_success` | none |
|
||||
| same | `record_connection_failure` | 181 | [PLATFORM] | no | BLE connection policy state. | `_record_connection_failure` | none |
|
||||
| same | `get_success_rate` | 185 | [PLATFORM] | no | BLE peer scoring helper. | `_score_peer`, `__repr__` | none |
|
||||
| same | `__repr__` | 196 | [TEST] | no | Diagnostic formatting only. | logging/debug | `get_success_rate` |
|
||||
| same | `BLEInterface` | 202 | [GLUE] | no | Reticulum `Interface` implementation binding BLE driver to RNS Transport. | Reticulum external interface loader | many |
|
||||
| same | `BLEInterface.__init__` | 267 | [GLUE] | no | Heavy RNS config, driver, locks, timers, callbacks. | Reticulum loader | `Interface.get_config_obj`, driver ctor |
|
||||
| same | `start` | 479 | [GLUE] | no | Starts driver/scanning and touches Transport path workaround. | `__init__` | `driver.start`, `driver.start_scanning`, `_clear_stale_ble_paths` |
|
||||
| same | `final_init` | 516 | [GLUE] | no | Reticulum lifecycle hook. | Reticulum | `_start_advertising_when_identity_ready` |
|
||||
| same | `_setup_logging_redirect` | 529 | [GLUE] | no | RNS logging integration. | `__init__` | `logging`, `RNS.log` |
|
||||
| same | `_start_advertising_when_identity_ready` | 583 | [GLUE] | no | Polls `RNS.Transport.identity`; starts platform advertising. | `final_init` thread | `driver.set_identity`, `driver.start_advertising` |
|
||||
| same | `_clear_stale_ble_paths` | 630 | [GLUE] | no | Depends on `RNS.Transport.path_table` internals. | `start` | `Transport.path_table` |
|
||||
| same | `_start_cleanup_timer` | 686 | [GLUE] | no | Thread timer around Python state. | `__init__`, cleanup task | `threading.Timer` |
|
||||
| same | `_periodic_cleanup_task` | 701 | [GLUE] | no | Coordinates reassembly cleanup plus interface lifecycle. | timer | `cleanup_stale_buffers`, `_validate_spawned_interfaces` |
|
||||
| same | `_cleanup_pending_identity_connections` | 741 | [GLUE] | no | Driver disconnect policy around identity handshake. | cleanup task | `driver.disconnect` |
|
||||
| same | `_process_pending_detaches` | 771 | [GLUE] | no | Reticulum peer interface lifecycle. | cleanup task | `peer_if.detach`, `_get_fragmenter_key` |
|
||||
| same | `_validate_spawned_interfaces` | 829 | [GLUE] | no | Compares RNS spawned interfaces to driver connections. | cleanup task | `driver.connected_peers`, `peer_if.detach` |
|
||||
| same | `_device_discovered_callback` | 912 | [PLATFORM] | no | BLE scan callback and connection initiation. | driver callback | `DiscoveredPeer`, `_select_peers_to_connect`, `driver.connect` |
|
||||
| same | `_device_connected_callback` | 964 | [GLUE] | no | Driver callback; maps identity to Reticulum peer interface. | driver callback | `_compute_identity_hash`, `_mtu_negotiated_callback` |
|
||||
| same | `_check_duplicate_identity` | 1023 | [GLUE] | no | Identity dedup policy tied to driver state and MAC rotation. | driver callback | `_cleanup_stale_address`, `driver.disconnect` |
|
||||
| same | `_mtu_negotiated_callback` | 1112 | [GLUE] | no | Creates protocol fragmenters but tied to driver/RNS interface spawn. | driver callback | `BLEFragmenter`, `BLEReassembler`, `_spawn_peer_interface` |
|
||||
| same | `_handle_identity_handshake` | 1174 | [GLUE] | no | 16-byte identity handshake is protocol-ish, but intertwined with driver/RNS state. | `_data_received_callback` | `_check_duplicate_identity`, `_spawn_peer_interface` |
|
||||
| same | `_data_received_callback` | 1281 | [GLUE] | no | Driver callback dispatch. | driver callback | `_handle_identity_handshake`, `_handle_ble_data` |
|
||||
| same | `_device_disconnected_callback` | 1295 | [GLUE] | no | Reticulum/driver cleanup. | driver callback | `_compute_identity_hash`, detach scheduling |
|
||||
| same | `_cleanup_stale_address` | 1380 | [GLUE] | no | MAC rotation mapping cleanup. | duplicate/address-change paths | dict cleanup |
|
||||
| same | `_address_changed_callback` | 1411 | [GLUE] | no | Driver MAC rotation callback. | driver callback | `_get_fragmenter_key`, `_compute_identity_hash` |
|
||||
| same | `_error_callback` | 1479 | [GLUE] | no | Driver error logging/blacklisting. | driver callback | `_record_connection_failure` |
|
||||
| same | `_score_peer` | 1534 | [PLATFORM] | no | BLE RSSI/recency scoring policy. | `_select_peers_to_connect` | `peer.get_success_rate` |
|
||||
| same | `_select_peers_to_connect` | 1620 | [PLATFORM] | no | BLE connection selection and MAC sorting. | discovery callback | `_score_peer`, `_is_blacklisted` |
|
||||
| same | `_is_blacklisted` | 1761 | [PLATFORM] | no | BLE retry/backoff policy. | `_select_peers_to_connect` | `time.time` |
|
||||
| same | `_record_connection_success` | 1785 | [PLATFORM] | no | BLE discovery/backoff state. | connection callback | `DiscoveredPeer.record_connection_success` |
|
||||
| same | `_record_connection_failure` | 1800 | [PLATFORM] | no | BLE retry/backoff plus BlueZ cleanup hook. | error callback | `driver._remove_bluez_device` |
|
||||
| same | `_get_fragmenter_key` | 1858 | [CORE] | yes | Pure identity-to-fragment-state key; cleanly separable. | many send/receive paths | `bytes.hex` |
|
||||
| same | `_compute_identity_hash` | 1871 | [CORE] | yes | Pure identity truncation rule; cleanly separable. | identity mapping paths | `bytes.hex` |
|
||||
| same | `_spawn_peer_interface` | 1892 | [GLUE] | no | Creates Reticulum `Interface`, mutates `RNS.Transport.interfaces`. | MTU/handshake paths | `BLEPeerInterface`, `RNS.Transport.interfaces.append` |
|
||||
| same | `_handle_ble_data` | 1957 | [GLUE] | no | Uses core reassembly, but routes into Reticulum peer interfaces. | data callback | `BLEReassembler.receive_fragment`, `peer_if.process_incoming` |
|
||||
| same | `handle_peripheral_data` | 2085 | [GLUE] | no | Older peripheral data path; handshake/reassembly/RNS routing mixed. | `BLEGATTServer` callback | `BLEFragmenter`, `BLEReassembler`, `_spawn_peer_interface` |
|
||||
| same | `handle_central_connected` | 2188 | [GLUE] | no | GATT server to RNS interface lifecycle. | `BLEGATTServer` callback | `_spawn_peer_interface` |
|
||||
| same | `handle_central_disconnected` | 2222 | [GLUE] | no | GATT server cleanup of Reticulum peer mapping. | `BLEGATTServer` callback | `_compute_identity_hash` |
|
||||
| same | `process_incoming` | 2291 | [GLUE] | no | Calls Reticulum owner inbound. | peer interfaces | `owner.inbound` |
|
||||
| same | `process_outgoing` | 2305 | [GLUE] | no | Broadcasts Reticulum packet to peer interfaces. | Reticulum Transport | `peer_if.process_outgoing` |
|
||||
| same | `detach` | 2335 | [GLUE] | no | Reticulum/driver shutdown. | Reticulum | `driver.stop`, `peer_if.detach` |
|
||||
| same | `should_ingress_limit` | 2365 | [GLUE] | no | Reticulum Interface policy. | Reticulum | none |
|
||||
| same | `__str__` | 2376 | [TEST] | no | Diagnostic formatting. | logging | none |
|
||||
| same | `BLEPeerInterface` | 2380 | [GLUE] | no | Reticulum per-peer interface wrapper. | `_spawn_peer_interface` | `Interface` |
|
||||
| same | `BLEPeerInterface.__init__` | 2388 | [GLUE] | no | Reticulum interface fields/statistics. | `_spawn_peer_interface` | `Interface.__init__` |
|
||||
| same | `BLEPeerInterface.process_incoming` | 2420 | [GLUE] | no | Hands data to `owner.inbound`. | `_handle_ble_data` | `owner.inbound` |
|
||||
| same | `BLEPeerInterface.process_outgoing` | 2437 | [GLUE] | no | Calls core fragmenter, then platform driver send. | parent `process_outgoing`, Reticulum | `fragment_packet`, `driver.send` |
|
||||
| same | `BLEPeerInterface.detach` | 2483 | [GLUE] | no | Removes interface from `RNS.Transport.interfaces`. | cleanup paths | `RNS.Transport.interfaces.remove` |
|
||||
| same | `BLEPeerInterface.should_ingress_limit` | 2493 | [GLUE] | no | Delegates Reticulum policy. | Reticulum | parent method |
|
||||
| same | `BLEPeerInterface.connection_id` | 2498 | [GLUE] | no | Reticulum/RNS identity display helper. | Reticulum/status | `RNS.Identity.full_hash` |
|
||||
| same | `BLEPeerInterface.__str__` | 2510 | [TEST] | no | Diagnostic formatting. | logging | none |
|
||||
|
||||
**BLEAgent.py, BLEGATTServer.py, bluetooth_driver.py**
|
||||
|
||||
| file path | symbol | line | tag | phase-1 C++ candidate | rationale | important callers | important callees |
|
||||
|---|---:|---:|---|---|---|---|---|
|
||||
| `src/ble_reticulum/BLEAgent.py` | `BLEAgent` + all methods | 55 | [PLATFORM] | no | BlueZ DBus pairing agent. | `register_agent`, GATT servers | `dbus.service`, `RNS.log` |
|
||||
| same | `register_agent` | 202 | [PLATFORM] | no | Registers BlueZ agent on system DBus. | GATT server startup | `DBusGMainLoop`, `dbus.SystemBus` |
|
||||
| same | `unregister_agent` | 262 | [PLATFORM] | no | Unregisters BlueZ agent. | GATT server shutdown | `dbus.SystemBus` |
|
||||
| `src/ble_reticulum/BLEGATTServer.py` | `BLEGATTServer` + all methods | 51 | [PLATFORM] | no | bluezero GATT server, threads, callbacks, notifications. | older/alternate peripheral path | `bluezero.peripheral`, `register_agent` |
|
||||
| same | `set_transport_identity` | 484 | [GLUE] | no | Validates Reticulum identity bytes but serves them over GATT. | startup | none |
|
||||
| `src/ble_reticulum/bluetooth_driver.py` | `BLEDevice` | 10 | [PLATFORM] | no | BLE discovery DTO. | Linux driver scan | dataclass |
|
||||
| same | `DriverState` | 18 | [PLATFORM] | no | Driver state enum. | driver state property | enum |
|
||||
| same | `BLEDriverInterface` + all abstract methods | 29 | [PLATFORM] | no | Platform BLE backend contract, not protocol core. | `BLEInterface` | abstract methods |
|
||||
|
||||
**linux_bluetooth_driver.py**
|
||||
|
||||
| file path | symbol | line | tag | phase-1 C++ candidate | rationale | important callers | important callees |
|
||||
|---|---:|---:|---|---|---|---|---|
|
||||
| `src/ble_reticulum/linux_bluetooth_driver.py` | `_rns_showwarning` | 142 | [GLUE] | no | Routes Python warnings to RNS logging. | `warnings.showwarning` | `RNS.log` |
|
||||
| same | `apply_bluez_services_resolved_patch` | 220 | [PLATFORM] | no | Monkey-patches Bleak BlueZ manager. | `LinuxBluetoothDriver.__init__` | `BlueZManager._wait_for_services_discovery` |
|
||||
| same | `PeerConnection` | 287 | [PLATFORM] | no | BLE connection runtime DTO with `BleakClient`. | driver connection state | dataclass |
|
||||
| same | `LinuxBluetoothDriver` + lifecycle/scanning/connection/GATT methods | 297 | [PLATFORM] | no | Linux BLE backend using Bleak, bluezero, DBus, asyncio, threads. | `BLEInterface` | `BleakScanner`, `BleakClient`, DBus, `BluezeroGATTServer` |
|
||||
| same | `_connect_to_peer` | 988 | [PLATFORM] | no | Central connection flow, service discovery, GATT reads/writes. | `connect` | `BleakClient`, `_negotiate_mtu`, callbacks |
|
||||
| same | `_connect_via_dbus_le` | 1253 | [PLATFORM] | no | BlueZ DBus LE-only workaround. | `_connect_to_peer` | `MessageBus`, `Adapter1.ConnectDevice` |
|
||||
| same | `_negotiate_mtu` | 1291 | [PLATFORM] | no | Bleak/BlueZ MTU probing. | `_connect_to_peer` | `client._backend`, `client.mtu_size` |
|
||||
| same | `_handle_notification` | 1332 | [PLATFORM] | no | Bleak notification callback adapter. | `start_notify` callback | `on_data_received` |
|
||||
| same | `BluezeroGATTServer` + all methods | 1578 | [PLATFORM] | no | bluezero peripheral server, DBus monitoring, stale polling. | `LinuxBluetoothDriver.start_advertising` | `bluezero`, DBus, threads |
|
||||
| same | `_handle_write_rx` | 2328 | [PLATFORM] | no | GATT write callback; forwards bytes upward. | bluezero | driver callbacks |
|
||||
| same | `_handle_read_identity` | 2374 | [PLATFORM] | no | GATT identity characteristic read. | bluezero | none |
|
||||
| same | `_handle_central_connected` | 2388 | [PLATFORM] | no | Peripheral connection state plus callbacks. | `_handle_write_rx` | `PeerConnection`, driver callbacks |
|
||||
| same | `_handle_central_disconnected` | 2430 | [PLATFORM] | no | DBus/poll disconnect cleanup. | DBus monitor, stale poll | driver callback |
|
||||
| same | `send_notification` | 2479 | [PLATFORM] | no | bluezero notification send. | `LinuxBluetoothDriver.send` | `tx_characteristic.set_value` |
|
||||
|
||||
For `linux_bluetooth_driver.py`, the inventory includes many individual platform methods under `LinuxBluetoothDriver` and `BluezeroGATTServer`; all are [PLATFORM]/no for the same reason: they directly touch Bleak, bluezero, DBus, BlueZ, sockets/processes, asyncio event loops, or threads.
|
||||
|
||||
**Phase 1 C++ Extraction Candidates**
|
||||
- `BLEFragmenter`
|
||||
- `BLEFragmenter.fragment_packet`
|
||||
- `BLEFragmenter.get_fragment_overhead`
|
||||
- `BLEReassembler`
|
||||
- `BLEReassembler.receive_fragment`
|
||||
- `BLEReassembler.cleanup_stale_buffers`
|
||||
- `BLEReassembler.get_statistics`
|
||||
- `BLEReassembler.reset_statistics`
|
||||
- `HDLCFramer.frame_packet`
|
||||
- `HDLCFramer.deframe_packet`
|
||||
- Extracted pure equivalents of `BLEInterface._get_fragmenter_key`
|
||||
- Extracted pure equivalents of `BLEInterface._compute_identity_hash`
|
||||
|
||||
**Do Not Port Yet**
|
||||
- `BLEInterface` and `BLEPeerInterface` as classes.
|
||||
- All BlueZ/Bleak/bluezero/DBus code: `BLEAgent`, `BLEGATTServer`, `LinuxBluetoothDriver`, `BluezeroGATTServer`.
|
||||
- `bluetooth_driver.py` driver abstraction unless designing a cross-language platform backend API later.
|
||||
- Peer discovery/scoring/backoff/MAC-rotation orchestration in `BLEInterface`; it is BLE transport policy and Reticulum glue, not a clean protocol core.
|
||||
- `ble_dual_node_echo.py`: treat as acceptance harness, as requested.
|
||||
|
||||
**Likely Python/C++ Boundary**
|
||||
The clean first boundary is a small protocol library that accepts/returns bytes and simple metadata only: fragment packet, parse fragment, reassemble by sender/session key, timeout stale buffers, and compute identity-derived keys. Python should continue owning Reticulum `Interface` integration, `RNS.Transport` mutation, driver callbacks, scanning, advertising, GATT characteristics, DBus pairing, threading, timers, and logging.
|
||||
|
||||
**Classification Depends On Reticulum Internals**
|
||||
- `BLEInterface.__init__`, `start`, `final_init`
|
||||
- `_start_advertising_when_identity_ready`
|
||||
- `_clear_stale_ble_paths`
|
||||
- `_spawn_peer_interface`
|
||||
- `_handle_ble_data`
|
||||
- `handle_peripheral_data`
|
||||
- `process_incoming`
|
||||
- `process_outgoing`
|
||||
- `detach`
|
||||
- `BLEPeerInterface.process_incoming`
|
||||
- `BLEPeerInterface.detach`
|
||||
- `BLEPeerInterface.connection_id`
|
||||
|
||||
The strongest dependency is `_clear_stale_ble_paths`, which assumes `RNS.Transport.path_table` tuple indexes and BLE interface type names.
|
||||
17
migration/README.md
Normal file
17
migration/README.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
This directory concerns the migration to C++ the ble-reticulum project.
|
||||
|
||||
## Test environment note
|
||||
|
||||
The project pytest configuration expects `pytest-asyncio`. 2 warning had issued, I had to rem the --disable-warnings flag in pytest.ini for the warnings to appear, when they appeared is became obvious that the absence of pytest-asyncio was creating the problem.
|
||||
|
||||
FIX: Install in the active virtual environment:
|
||||
|
||||
```bash
|
||||
python3 -m pip install pytest-asyncio
|
||||
```
|
||||
|
||||
Equivalence test:
|
||||
```bash
|
||||
cd /usr/local/src/ble-reticulum
|
||||
python3 -m pytest migration/tests/test_fragmentation_cpp_equivalence.py -vv
|
||||
```
|
||||
182
migration/ble_symbols_20260516_1429.txt
Normal file
182
migration/ble_symbols_20260516_1429.txt
Normal file
|
|
@ -0,0 +1,182 @@
|
|||
BLEAgent.py
|
||||
class BLEAgent line 55
|
||||
def __init__(self, bus, capability) line 68
|
||||
def _log(self, message, level) line 82
|
||||
def Release(self) line 101
|
||||
def AuthorizeService(self, device, uuid) line 111
|
||||
def RequestAuthorization(self, device) line 127
|
||||
def RequestConfirmation(self, device, passkey) line 139
|
||||
def RequestPasskey(self, device) line 155
|
||||
def Cancel(self) line 172
|
||||
def _format_device_path(self, device_path) line 182
|
||||
def register_agent(capability) line 202
|
||||
def unregister_agent(agent) line 262
|
||||
BLEFragmentation.py
|
||||
class BLEFragmenter line 52
|
||||
def __init__(self, mtu) line 68
|
||||
def fragment_packet(self, packet) line 82
|
||||
def get_fragment_overhead(self, packet_size) line 158
|
||||
class BLEReassembler line 176
|
||||
def __init__(self, timeout) line 187
|
||||
def receive_fragment(self, fragment, sender_id) line 205
|
||||
def _reassemble(self, buffer) line 380
|
||||
def cleanup_stale_buffers(self) line 402
|
||||
def get_statistics(self) line 429
|
||||
def reset_statistics(self) line 443
|
||||
class HDLCFramer line 450
|
||||
def frame_packet(packet) line 464
|
||||
def deframe_packet(frame) line 491
|
||||
BLEGATTServer.py
|
||||
class BLEGATTServer line 51
|
||||
def __init__(self, interface, device_name, agent_capability) line 77
|
||||
def _log(self, message, level) line 127
|
||||
def _handle_write_rx(self, value, options) line 148
|
||||
def _handle_read_identity(self, options) line 215
|
||||
def _handle_central_connected(self, central_address, mtu) line 242
|
||||
def _handle_central_disconnected(self, central_address) line 279
|
||||
def _run_server_thread(self) line 306
|
||||
async def start(self) line 441
|
||||
def set_transport_identity(self, identity_hash) line 484
|
||||
async def stop(self) line 503
|
||||
async def send_notification(self, data, central_address) line 543
|
||||
def is_connected(self, central_address) line 607
|
||||
def get_connected_centrals(self) line 620
|
||||
def get_connection_info(self, central_address) line 630
|
||||
def get_central_mtu(self, central_address) line 643
|
||||
def get_statistics(self) line 658
|
||||
def __str__(self) line 678
|
||||
def __repr__(self) line 684
|
||||
BLEInterface.py
|
||||
class DiscoveredPeer line 118
|
||||
def __init__(self, address, name, rssi) line 146
|
||||
def update_rssi(self, rssi) line 167
|
||||
def record_connection_attempt(self) line 172
|
||||
def record_connection_success(self) line 177
|
||||
def record_connection_failure(self) line 181
|
||||
def get_success_rate(self) line 185
|
||||
def __repr__(self) line 196
|
||||
class BLEInterface line 202
|
||||
def __init__(self, owner, configuration) line 267
|
||||
def start(self) line 479
|
||||
def final_init(self) line 516
|
||||
def _setup_logging_redirect(self) line 529
|
||||
def _start_advertising_when_identity_ready(self) line 583
|
||||
def _clear_stale_ble_paths(self) line 630
|
||||
def _start_cleanup_timer(self) line 686
|
||||
def _periodic_cleanup_task(self) line 701
|
||||
def _cleanup_pending_identity_connections(self) line 741
|
||||
def _process_pending_detaches(self) line 771
|
||||
def _validate_spawned_interfaces(self) line 829
|
||||
def _device_discovered_callback(self, device) line 912
|
||||
def _device_connected_callback(self, address, peer_identity) line 964
|
||||
def _check_duplicate_identity(self, address, peer_identity) line 1023
|
||||
def _mtu_negotiated_callback(self, address, mtu) line 1112
|
||||
def _handle_identity_handshake(self, address, data) line 1174
|
||||
def _data_received_callback(self, address, data) line 1281
|
||||
def _device_disconnected_callback(self, address) line 1295
|
||||
def _cleanup_stale_address(self, identity_hash, old_address) line 1380
|
||||
def _address_changed_callback(self, old_address, new_address, identity_hash) line 1411
|
||||
def _error_callback(self, severity, message, exc) line 1479
|
||||
def _score_peer(self, peer) line 1534
|
||||
def _select_peers_to_connect(self) line 1620
|
||||
def _is_blacklisted(self, address) line 1761
|
||||
def _record_connection_success(self, address) line 1785
|
||||
def _record_connection_failure(self, address) line 1800
|
||||
def _get_fragmenter_key(self, peer_identity, peer_address) line 1858
|
||||
def _compute_identity_hash(self, peer_identity) line 1871
|
||||
def _spawn_peer_interface(self, address, name, peer_identity, client, mtu, connection_type) line 1892
|
||||
def _handle_ble_data(self, peer_address, data) line 1957
|
||||
def handle_peripheral_data(self, data, sender_address) line 2085
|
||||
def handle_central_connected(self, address) line 2188
|
||||
def handle_central_disconnected(self, address) line 2222
|
||||
def process_incoming(self, data) line 2291
|
||||
def process_outgoing(self, data) line 2305
|
||||
def detach(self) line 2335
|
||||
def should_ingress_limit(self) line 2365
|
||||
def __str__(self) line 2376
|
||||
class BLEPeerInterface line 2380
|
||||
def __init__(self, parent, peer_address, peer_name, peer_identity) line 2388
|
||||
def process_incoming(self, data) line 2420
|
||||
def process_outgoing(self, data) line 2437
|
||||
def detach(self) line 2483
|
||||
def should_ingress_limit(self) line 2493
|
||||
def connection_id(self) line 2498
|
||||
def __str__(self) line 2510
|
||||
__init__.py
|
||||
bluetooth_driver.py
|
||||
class BLEDevice line 10
|
||||
class DriverState line 18
|
||||
class BLEDriverInterface line 29
|
||||
def start(self, service_uuid, rx_char_uuid, tx_char_uuid, identity_char_uuid) line 57
|
||||
def stop(self) line 66
|
||||
def set_identity(self, identity_bytes) line 74
|
||||
def state(self) line 85
|
||||
def connected_peers(self) line 91
|
||||
def start_scanning(self) line 98
|
||||
def stop_scanning(self) line 106
|
||||
def start_advertising(self, device_name, identity) line 111
|
||||
def stop_advertising(self) line 124
|
||||
def connect(self, address) line 129
|
||||
def disconnect(self, address) line 137
|
||||
def send(self, address, data) line 142
|
||||
def read_characteristic(self, address, char_uuid) line 157
|
||||
def write_characteristic(self, address, char_uuid, data) line 165
|
||||
def start_notify(self, address, char_uuid, callback) line 173
|
||||
def get_local_address(self) line 183
|
||||
def get_peer_role(self, address) line 191
|
||||
def set_service_discovery_delay(self, seconds) line 204
|
||||
def set_power_mode(self, mode) line 212
|
||||
linux_bluetooth_driver.py
|
||||
def _rns_showwarning(message, category, filename, lineno, file, line) line 142
|
||||
def apply_bluez_services_resolved_patch() line 220
|
||||
class PeerConnection line 287
|
||||
class LinuxBluetoothDriver line 297
|
||||
def __init__(self, discovery_interval, connection_timeout, min_rssi, service_discovery_delay, max_peers, adapter_index, agent_capability) line 314
|
||||
def _log(self, message, level) line 403
|
||||
def start(self, service_uuid, rx_char_uuid, tx_char_uuid, identity_char_uuid) line 426
|
||||
def stop(self) line 489
|
||||
def set_identity(self, identity_bytes) line 531
|
||||
def state(self) line 551
|
||||
def connected_peers(self) line 556
|
||||
def start_scanning(self) line 565
|
||||
def stop_scanning(self) line 582
|
||||
def _should_pause_scanning(self) line 593
|
||||
async def _scan_loop(self) line 607
|
||||
async def _perform_scan(self) line 636
|
||||
def start_advertising(self, device_name, identity) line 749
|
||||
def stop_advertising(self) line 784
|
||||
def connect(self, address) line 806
|
||||
def disconnect(self, address) line 876
|
||||
def _handle_peripheral_disconnected(self, address) line 909
|
||||
async def _remove_bluez_device(self, address) line 942
|
||||
async def _connect_to_peer(self, address) line 988
|
||||
async def _connect_via_dbus_le(self, peer_address) line 1253
|
||||
async def _negotiate_mtu(self, client) line 1291
|
||||
def _handle_notification(self, address, data) line 1332
|
||||
def send(self, address, data) line 1344
|
||||
def read_characteristic(self, address, char_uuid) line 1386
|
||||
def write_characteristic(self, address, char_uuid, data) line 1409
|
||||
def start_notify(self, address, char_uuid, callback) line 1431
|
||||
def get_local_address(self) line 1464
|
||||
def get_peer_role(self, address) line 1468
|
||||
def get_peer_mtu(self, address) line 1475
|
||||
def set_service_discovery_delay(self, seconds) line 1494
|
||||
def set_power_mode(self, mode) line 1499
|
||||
def _run_event_loop(self) line 1511
|
||||
async def _get_local_adapter_address(self) line 1523
|
||||
def _detect_bluez_version(self) line 1556
|
||||
class BluezeroGATTServer line 1578
|
||||
def __init__(self, driver, service_uuid, rx_char_uuid, tx_char_uuid, identity_char_uuid, adapter_index, agent_capability) line 1589
|
||||
def _log(self, message, level) line 1646
|
||||
def set_identity(self, identity_bytes) line 1650
|
||||
def _verify_services_on_dbus(self, timeout) line 1662
|
||||
def _monitor_device_disconnections(self) line 1730
|
||||
def _poll_stale_connections(self) line 1965
|
||||
def start(self, device_name) line 2072
|
||||
def stop(self) line 2160
|
||||
def _run_server_thread(self, device_name) line 2206
|
||||
def _handle_write_rx(self, value, options) line 2328
|
||||
def _handle_read_identity(self, options) line 2374
|
||||
def _handle_central_connected(self, central_address, mtu) line 2388
|
||||
def _handle_central_disconnected(self, central_address) line 2430
|
||||
def send_notification(self, central_address, data) line 2479
|
||||
489
migration/protocol_core/ble_protocol_core.cpp
Normal file
489
migration/protocol_core/ble_protocol_core.cpp
Normal file
|
|
@ -0,0 +1,489 @@
|
|||
#include <pybind11/pybind11.h>
|
||||
#include <pybind11/stl.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <cstdint>
|
||||
#include <map>
|
||||
#include <stdexcept>
|
||||
#include <string>
|
||||
#include <tuple>
|
||||
#include <vector>
|
||||
|
||||
namespace py = pybind11;
|
||||
|
||||
namespace {
|
||||
|
||||
using Bytes = std::string;
|
||||
|
||||
constexpr uint8_t TYPE_START = 0x01;
|
||||
constexpr uint8_t TYPE_CONTINUE = 0x02;
|
||||
constexpr uint8_t TYPE_END = 0x03;
|
||||
constexpr size_t HEADER_SIZE = 5;
|
||||
|
||||
double now_seconds() {
|
||||
py::object time = py::module_::import("time").attr("time");
|
||||
return time().cast<double>();
|
||||
}
|
||||
|
||||
Bytes py_bytes_to_string(const py::bytes &value) {
|
||||
return value.cast<std::string>();
|
||||
}
|
||||
|
||||
py::bytes string_to_py_bytes(const Bytes &value) {
|
||||
return py::bytes(value);
|
||||
}
|
||||
|
||||
struct Buffer {
|
||||
std::map<uint16_t, Bytes> fragments;
|
||||
uint16_t total = 0;
|
||||
double start_time = 0.0;
|
||||
std::string sender_id;
|
||||
};
|
||||
|
||||
struct BufferKey {
|
||||
std::string sender_id;
|
||||
uint16_t packet_index = 0;
|
||||
uint16_t total = 0;
|
||||
|
||||
bool operator<(const BufferKey &other) const {
|
||||
return std::tie(sender_id, packet_index, total) <
|
||||
std::tie(other.sender_id, other.packet_index, other.total);
|
||||
}
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
class BLEFragmenterCpp {
|
||||
public:
|
||||
static constexpr uint8_t TYPE_START_VALUE = TYPE_START;
|
||||
static constexpr uint8_t TYPE_CONTINUE_VALUE = TYPE_CONTINUE;
|
||||
static constexpr uint8_t TYPE_END_VALUE = TYPE_END;
|
||||
static constexpr size_t HEADER_SIZE_VALUE = HEADER_SIZE;
|
||||
|
||||
explicit BLEFragmenterCpp(int mtu_value = 185) {
|
||||
mtu = std::max(mtu_value, 20);
|
||||
payload_size = mtu - static_cast<int>(HEADER_SIZE);
|
||||
if (payload_size < 1) {
|
||||
throw py::value_error("MTU " + std::to_string(mtu_value) +
|
||||
" too small for fragmentation (min " +
|
||||
std::to_string(HEADER_SIZE + 1) + ")");
|
||||
}
|
||||
}
|
||||
|
||||
std::vector<py::bytes> fragment_packet(const py::bytes &packet_obj) const {
|
||||
Bytes packet = py_bytes_to_string(packet_obj);
|
||||
if (packet.empty()) {
|
||||
throw py::value_error("Cannot fragment empty packet");
|
||||
}
|
||||
|
||||
const size_t packet_size = packet.size();
|
||||
const size_t num_fragments =
|
||||
(packet_size + static_cast<size_t>(payload_size) - 1) /
|
||||
static_cast<size_t>(payload_size);
|
||||
|
||||
if (num_fragments > 65535) {
|
||||
throw py::value_error(
|
||||
"Packet requires " + std::to_string(num_fragments) +
|
||||
" fragments, exceeds max (65535). Packet size too large for BLE MTU " +
|
||||
std::to_string(mtu) + ". Maximum supported: " +
|
||||
std::to_string(65535 * payload_size) + " bytes");
|
||||
}
|
||||
|
||||
std::vector<py::bytes> fragments;
|
||||
fragments.reserve(num_fragments);
|
||||
|
||||
for (size_t i = 0; i < num_fragments; ++i) {
|
||||
uint8_t frag_type = TYPE_CONTINUE;
|
||||
if (i == 0) {
|
||||
frag_type = TYPE_START;
|
||||
} else if (i == num_fragments - 1) {
|
||||
frag_type = TYPE_END;
|
||||
}
|
||||
|
||||
const size_t start_idx = i * static_cast<size_t>(payload_size);
|
||||
const size_t end_idx =
|
||||
std::min(start_idx + static_cast<size_t>(payload_size), packet_size);
|
||||
const Bytes data = packet.substr(start_idx, end_idx - start_idx);
|
||||
|
||||
Bytes fragment;
|
||||
fragment.reserve(HEADER_SIZE + data.size());
|
||||
fragment.push_back(static_cast<char>(frag_type));
|
||||
fragment.push_back(static_cast<char>((i >> 8) & 0xff));
|
||||
fragment.push_back(static_cast<char>(i & 0xff));
|
||||
fragment.push_back(static_cast<char>((num_fragments >> 8) & 0xff));
|
||||
fragment.push_back(static_cast<char>(num_fragments & 0xff));
|
||||
fragment.append(data);
|
||||
|
||||
fragments.emplace_back(string_to_py_bytes(fragment));
|
||||
}
|
||||
|
||||
return fragments;
|
||||
}
|
||||
|
||||
py::tuple get_fragment_overhead(size_t packet_size) const {
|
||||
const size_t num_fragments =
|
||||
(packet_size + static_cast<size_t>(payload_size) - 1) /
|
||||
static_cast<size_t>(payload_size);
|
||||
const size_t overhead_bytes = num_fragments * HEADER_SIZE;
|
||||
const double overhead_pct =
|
||||
packet_size > 0 ? (static_cast<double>(overhead_bytes) /
|
||||
static_cast<double>(packet_size)) *
|
||||
100.0
|
||||
: 0.0;
|
||||
return py::make_tuple(num_fragments, overhead_bytes, overhead_pct);
|
||||
}
|
||||
|
||||
int mtu = 185;
|
||||
int payload_size = 180;
|
||||
};
|
||||
|
||||
class BLEReassemblerCpp {
|
||||
public:
|
||||
static constexpr double DEFAULT_TIMEOUT_VALUE = 30.0;
|
||||
|
||||
explicit BLEReassemblerCpp(py::object timeout_obj = py::none()) {
|
||||
if (timeout_obj.is_none()) {
|
||||
timeout = DEFAULT_TIMEOUT_VALUE;
|
||||
} else {
|
||||
timeout = timeout_obj.cast<double>();
|
||||
}
|
||||
}
|
||||
|
||||
py::object receive_fragment(const py::bytes &fragment_obj,
|
||||
py::object sender_obj = py::none()) {
|
||||
const Bytes fragment = py_bytes_to_string(fragment_obj);
|
||||
if (fragment.size() < HEADER_SIZE) {
|
||||
throw py::value_error("Fragment too short: " +
|
||||
std::to_string(fragment.size()) +
|
||||
" bytes (min " + std::to_string(HEADER_SIZE) + ")");
|
||||
}
|
||||
|
||||
std::string sender_id = "default";
|
||||
if (!sender_obj.is_none()) {
|
||||
sender_id = py::str(sender_obj).cast<std::string>();
|
||||
}
|
||||
|
||||
fragments_received += 1;
|
||||
|
||||
const uint8_t frag_type = static_cast<uint8_t>(fragment[0]);
|
||||
const uint16_t sequence =
|
||||
(static_cast<uint16_t>(static_cast<uint8_t>(fragment[1])) << 8) |
|
||||
static_cast<uint16_t>(static_cast<uint8_t>(fragment[2]));
|
||||
const uint16_t total =
|
||||
(static_cast<uint16_t>(static_cast<uint8_t>(fragment[3])) << 8) |
|
||||
static_cast<uint16_t>(static_cast<uint8_t>(fragment[4]));
|
||||
const Bytes data = fragment.substr(HEADER_SIZE);
|
||||
|
||||
if (frag_type != TYPE_START && frag_type != TYPE_CONTINUE &&
|
||||
frag_type != TYPE_END) {
|
||||
char buffer[64];
|
||||
snprintf(buffer, sizeof(buffer), "Invalid fragment type: 0x%02x",
|
||||
frag_type);
|
||||
throw py::value_error(buffer);
|
||||
}
|
||||
|
||||
if (sequence >= total) {
|
||||
throw py::value_error("Invalid sequence " + std::to_string(sequence) +
|
||||
" >= total " + std::to_string(total));
|
||||
}
|
||||
|
||||
if (total == 0) {
|
||||
throw py::value_error("Total fragments cannot be zero");
|
||||
}
|
||||
|
||||
BufferKey packet_key{sender_id, static_cast<uint16_t>(sequence / total),
|
||||
total};
|
||||
|
||||
if (sequence == 0) {
|
||||
Buffer buffer;
|
||||
buffer.fragments[sequence] = data;
|
||||
buffer.total = total;
|
||||
buffer.start_time = now_seconds();
|
||||
buffer.sender_id = sender_id;
|
||||
reassembly_buffers[packet_key] = buffer;
|
||||
} else {
|
||||
bool found = false;
|
||||
BufferKey buffer_key;
|
||||
const double current_time = now_seconds();
|
||||
for (const auto &entry : reassembly_buffers) {
|
||||
if (entry.first.sender_id == sender_id &&
|
||||
entry.second.total == total &&
|
||||
current_time - entry.second.start_time < timeout) {
|
||||
buffer_key = entry.first;
|
||||
found = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (!found) {
|
||||
if (reassembly_buffers.find(packet_key) == reassembly_buffers.end()) {
|
||||
Buffer buffer;
|
||||
buffer.total = total;
|
||||
buffer.start_time = now_seconds();
|
||||
buffer.sender_id = sender_id;
|
||||
reassembly_buffers[packet_key] = buffer;
|
||||
}
|
||||
Buffer &buffer = reassembly_buffers[packet_key];
|
||||
if (buffer.fragments.find(sequence) != buffer.fragments.end()) {
|
||||
check_duplicate_or_throw(packet_key, sequence, data);
|
||||
return py::none();
|
||||
}
|
||||
buffer.fragments[sequence] = data;
|
||||
return py::none();
|
||||
}
|
||||
|
||||
packet_key = buffer_key;
|
||||
Buffer &buffer = reassembly_buffers[packet_key];
|
||||
if (buffer.total != total) {
|
||||
reassembly_buffers.erase(packet_key);
|
||||
throw py::value_error("Fragment total mismatch for " + sender_id +
|
||||
": expected " +
|
||||
std::to_string(buffer.total) + ", got " +
|
||||
std::to_string(total));
|
||||
}
|
||||
|
||||
if (buffer.fragments.find(sequence) != buffer.fragments.end()) {
|
||||
check_duplicate_or_throw(packet_key, sequence, data);
|
||||
return py::none();
|
||||
}
|
||||
|
||||
buffer.fragments[sequence] = data;
|
||||
}
|
||||
|
||||
Buffer &buffer = reassembly_buffers[packet_key];
|
||||
if (buffer.fragments.size() == total) {
|
||||
for (uint16_t i = 0; i < total; ++i) {
|
||||
if (buffer.fragments.find(i) == buffer.fragments.end()) {
|
||||
return py::none();
|
||||
}
|
||||
}
|
||||
|
||||
Bytes packet = reassemble(buffer);
|
||||
reassembly_buffers.erase(packet_key);
|
||||
packets_reassembled += 1;
|
||||
return string_to_py_bytes(packet);
|
||||
}
|
||||
|
||||
return py::none();
|
||||
}
|
||||
|
||||
int cleanup_stale_buffers() {
|
||||
const double current_time = now_seconds();
|
||||
std::vector<BufferKey> stale_keys;
|
||||
|
||||
for (const auto &entry : reassembly_buffers) {
|
||||
if (current_time - entry.second.start_time > timeout) {
|
||||
stale_keys.push_back(entry.first);
|
||||
}
|
||||
}
|
||||
|
||||
for (const auto &key : stale_keys) {
|
||||
reassembly_buffers.erase(key);
|
||||
packets_timeout += 1;
|
||||
}
|
||||
|
||||
return static_cast<int>(stale_keys.size());
|
||||
}
|
||||
|
||||
py::bytes _reassemble(py::dict py_buffer) const {
|
||||
if (!py_buffer.contains("fragments") || !py_buffer.contains("total")) {
|
||||
throw py::key_error("fragments");
|
||||
}
|
||||
|
||||
py::dict py_fragments = py_buffer["fragments"].cast<py::dict>();
|
||||
const uint16_t total = py_buffer["total"].cast<uint16_t>();
|
||||
|
||||
Bytes packet;
|
||||
for (uint16_t i = 0; i < total; ++i) {
|
||||
py::object key = py::int_(i);
|
||||
if (!py_fragments.contains(key)) {
|
||||
throw py::value_error("Missing fragment " + std::to_string(i) +
|
||||
" during reassembly");
|
||||
}
|
||||
packet.append(py_fragments[key].cast<std::string>());
|
||||
}
|
||||
|
||||
return string_to_py_bytes(packet);
|
||||
}
|
||||
|
||||
py::dict get_statistics() const {
|
||||
py::dict stats;
|
||||
stats["packets_reassembled"] = packets_reassembled;
|
||||
stats["packets_timeout"] = packets_timeout;
|
||||
stats["fragments_received"] = fragments_received;
|
||||
stats["pending_packets"] = reassembly_buffers.size();
|
||||
return stats;
|
||||
}
|
||||
|
||||
void reset_statistics() {
|
||||
packets_reassembled = 0;
|
||||
packets_timeout = 0;
|
||||
fragments_received = 0;
|
||||
}
|
||||
|
||||
py::dict buffers_as_dict() const {
|
||||
py::dict result;
|
||||
for (const auto &entry : reassembly_buffers) {
|
||||
py::tuple key =
|
||||
py::make_tuple(entry.first.sender_id, entry.first.packet_index,
|
||||
entry.first.total);
|
||||
py::dict value;
|
||||
py::dict fragments;
|
||||
for (const auto &frag : entry.second.fragments) {
|
||||
fragments[py::int_(frag.first)] = string_to_py_bytes(frag.second);
|
||||
}
|
||||
value["fragments"] = fragments;
|
||||
value["total"] = entry.second.total;
|
||||
value["start_time"] = entry.second.start_time;
|
||||
value["sender_id"] = entry.second.sender_id;
|
||||
result[key] = value;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
double timeout = DEFAULT_TIMEOUT_VALUE;
|
||||
int packets_reassembled = 0;
|
||||
int packets_timeout = 0;
|
||||
int fragments_received = 0;
|
||||
|
||||
private:
|
||||
void check_duplicate_or_throw(const BufferKey &packet_key, uint16_t sequence,
|
||||
const Bytes &data) {
|
||||
Buffer &buffer = reassembly_buffers[packet_key];
|
||||
const Bytes &existing_data = buffer.fragments[sequence];
|
||||
if (existing_data == data) {
|
||||
return;
|
||||
}
|
||||
|
||||
reassembly_buffers.erase(packet_key);
|
||||
throw py::value_error("Fragment " + std::to_string(sequence) + " from " +
|
||||
packet_key.sender_id +
|
||||
" received twice with different data! Possible corruption.");
|
||||
}
|
||||
|
||||
static Bytes reassemble(const Buffer &buffer) {
|
||||
Bytes packet;
|
||||
for (uint16_t i = 0; i < buffer.total; ++i) {
|
||||
auto iter = buffer.fragments.find(i);
|
||||
if (iter == buffer.fragments.end()) {
|
||||
throw py::value_error("Missing fragment " + std::to_string(i) +
|
||||
" during reassembly");
|
||||
}
|
||||
packet.append(iter->second);
|
||||
}
|
||||
return packet;
|
||||
}
|
||||
|
||||
std::map<BufferKey, Buffer> reassembly_buffers;
|
||||
};
|
||||
|
||||
class HDLCFramerCpp {
|
||||
public:
|
||||
static constexpr uint8_t FLAG_VALUE = 0x7E;
|
||||
static constexpr uint8_t ESCAPE_VALUE = 0x7D;
|
||||
static constexpr uint8_t ESCAPE_XOR_VALUE = 0x20;
|
||||
|
||||
static py::bytes frame_packet(const py::bytes &packet_obj) {
|
||||
const Bytes packet = py_bytes_to_string(packet_obj);
|
||||
Bytes stuffed;
|
||||
stuffed.reserve(packet.size() + 2);
|
||||
|
||||
for (unsigned char byte : packet) {
|
||||
if (byte == FLAG_VALUE || byte == ESCAPE_VALUE) {
|
||||
stuffed.push_back(static_cast<char>(ESCAPE_VALUE));
|
||||
stuffed.push_back(static_cast<char>(byte ^ ESCAPE_XOR_VALUE));
|
||||
} else {
|
||||
stuffed.push_back(static_cast<char>(byte));
|
||||
}
|
||||
}
|
||||
|
||||
Bytes frame;
|
||||
frame.reserve(stuffed.size() + 2);
|
||||
frame.push_back(static_cast<char>(FLAG_VALUE));
|
||||
frame.append(stuffed);
|
||||
frame.push_back(static_cast<char>(FLAG_VALUE));
|
||||
return string_to_py_bytes(frame);
|
||||
}
|
||||
|
||||
static py::bytes deframe_packet(const py::bytes &frame_obj) {
|
||||
const Bytes frame = py_bytes_to_string(frame_obj);
|
||||
if (frame.size() < 2) {
|
||||
throw py::value_error("Frame too short (minimum 2 bytes for delimiters)");
|
||||
}
|
||||
|
||||
if (static_cast<uint8_t>(frame.front()) != FLAG_VALUE ||
|
||||
static_cast<uint8_t>(frame.back()) != FLAG_VALUE) {
|
||||
throw py::value_error("Invalid frame: missing FLAG delimiters");
|
||||
}
|
||||
|
||||
Bytes unstuffed;
|
||||
bool escape_next = false;
|
||||
|
||||
for (size_t i = 1; i + 1 < frame.size(); ++i) {
|
||||
const uint8_t byte = static_cast<uint8_t>(frame[i]);
|
||||
if (escape_next) {
|
||||
unstuffed.push_back(static_cast<char>(byte ^ ESCAPE_XOR_VALUE));
|
||||
escape_next = false;
|
||||
} else if (byte == ESCAPE_VALUE) {
|
||||
escape_next = true;
|
||||
} else if (byte == FLAG_VALUE) {
|
||||
throw py::value_error("Unexpected FLAG in frame data");
|
||||
} else {
|
||||
unstuffed.push_back(static_cast<char>(byte));
|
||||
}
|
||||
}
|
||||
|
||||
if (escape_next) {
|
||||
throw py::value_error("Frame ends with ESCAPE character");
|
||||
}
|
||||
|
||||
return string_to_py_bytes(unstuffed);
|
||||
}
|
||||
};
|
||||
|
||||
PYBIND11_MODULE(ble_protocol_core_cpp, m) {
|
||||
m.doc() = "C++ protocol-core implementation for BLE Reticulum fragmentation";
|
||||
|
||||
py::class_<BLEFragmenterCpp>(m, "BLEFragmenter")
|
||||
.def(py::init<int>(), py::arg("mtu") = 185)
|
||||
.def("fragment_packet", &BLEFragmenterCpp::fragment_packet,
|
||||
py::arg("packet"))
|
||||
.def("get_fragment_overhead", &BLEFragmenterCpp::get_fragment_overhead,
|
||||
py::arg("packet_size"))
|
||||
.def_readwrite("mtu", &BLEFragmenterCpp::mtu)
|
||||
.def_readwrite("payload_size", &BLEFragmenterCpp::payload_size)
|
||||
.def_readonly_static("TYPE_START", &BLEFragmenterCpp::TYPE_START_VALUE)
|
||||
.def_readonly_static("TYPE_CONTINUE",
|
||||
&BLEFragmenterCpp::TYPE_CONTINUE_VALUE)
|
||||
.def_readonly_static("TYPE_END", &BLEFragmenterCpp::TYPE_END_VALUE)
|
||||
.def_readonly_static("HEADER_SIZE", &BLEFragmenterCpp::HEADER_SIZE_VALUE);
|
||||
|
||||
py::class_<BLEReassemblerCpp>(m, "BLEReassembler")
|
||||
.def(py::init<py::object>(), py::arg("timeout") = py::none())
|
||||
.def("receive_fragment", &BLEReassemblerCpp::receive_fragment,
|
||||
py::arg("fragment"), py::arg("sender_id") = py::none())
|
||||
.def("_reassemble", &BLEReassemblerCpp::_reassemble, py::arg("buffer"))
|
||||
.def("cleanup_stale_buffers", &BLEReassemblerCpp::cleanup_stale_buffers)
|
||||
.def("get_statistics", &BLEReassemblerCpp::get_statistics)
|
||||
.def("reset_statistics", &BLEReassemblerCpp::reset_statistics)
|
||||
.def_readwrite("timeout", &BLEReassemblerCpp::timeout)
|
||||
.def_readwrite("packets_reassembled",
|
||||
&BLEReassemblerCpp::packets_reassembled)
|
||||
.def_readwrite("packets_timeout", &BLEReassemblerCpp::packets_timeout)
|
||||
.def_readwrite("fragments_received",
|
||||
&BLEReassemblerCpp::fragments_received)
|
||||
.def_property_readonly("reassembly_buffers",
|
||||
&BLEReassemblerCpp::buffers_as_dict)
|
||||
.def_readonly_static("DEFAULT_TIMEOUT",
|
||||
&BLEReassemblerCpp::DEFAULT_TIMEOUT_VALUE);
|
||||
|
||||
py::class_<HDLCFramerCpp>(m, "HDLCFramer")
|
||||
.def(py::init<>())
|
||||
.def_static("frame_packet", &HDLCFramerCpp::frame_packet,
|
||||
py::arg("packet"))
|
||||
.def_static("deframe_packet", &HDLCFramerCpp::deframe_packet,
|
||||
py::arg("frame"))
|
||||
.def_readonly_static("FLAG", &HDLCFramerCpp::FLAG_VALUE)
|
||||
.def_readonly_static("ESCAPE", &HDLCFramerCpp::ESCAPE_VALUE)
|
||||
.def_readonly_static("ESCAPE_XOR", &HDLCFramerCpp::ESCAPE_XOR_VALUE);
|
||||
}
|
||||
22
migration/protocol_core/setup.py
Normal file
22
migration/protocol_core/setup.py
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
from pathlib import Path
|
||||
|
||||
from pybind11.setup_helpers import Pybind11Extension, build_ext
|
||||
from setuptools import setup
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
setup(
|
||||
name="ble-protocol-core-cpp",
|
||||
version="0.0.1",
|
||||
ext_modules=[
|
||||
Pybind11Extension(
|
||||
"ble_protocol_core_cpp",
|
||||
[str(ROOT / "ble_protocol_core.cpp")],
|
||||
cxx_std=17,
|
||||
)
|
||||
],
|
||||
cmdclass={"build_ext": build_ext},
|
||||
zip_safe=False,
|
||||
)
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
# C++ Protocol Core Fragmentation Equivalence Test
|
||||
|
||||
Date: 2026-05-16
|
||||
Host: jp
|
||||
Python environment: rnsenv
|
||||
Repository: /usr/local/src/ble-reticulum
|
||||
|
||||
## Command
|
||||
|
||||
```bash
|
||||
cd /usr/local/src/ble-reticulum
|
||||
python3 -m pytest migration/tests/test_fragmentation_cpp_equivalence.py -vv
|
||||
```
|
||||
## Result
|
||||
|
||||
28 tests passed.
|
||||
|
||||
Coverage Summary
|
||||
|
||||
The C++ implementation was compared against the existing Python implementation for:
|
||||
|
||||
``` bash
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ cd /usr/local/src/ble-reticulum
|
||||
python3 -m pytest migration/tests/test_fragmentation_cpp_equivalence.py -vv
|
||||
======================================================================================== test session starts ========================================================================================
|
||||
platform linux -- Python 3.13.12, pytest-9.0.3, pluggy-1.6.0 -- /home/jlpoole/rnsenv/bin/python3
|
||||
cachedir: .pytest_cache
|
||||
rootdir: /usr/local/src/ble-reticulum
|
||||
configfile: pytest.ini
|
||||
plugins: anyio-4.12.1, asyncio-1.3.0
|
||||
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
|
||||
collected 28 items
|
||||
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_single_fragment_packets PASSED [ 3%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_multi_fragment_packets PASSED [ 7%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_mtu_boundary_sizes[20] PASSED [ 10%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_mtu_boundary_sizes[23] PASSED [ 14%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_mtu_boundary_sizes[50] PASSED [ 17%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_mtu_boundary_sizes[185] PASSED [ 21%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEFragmenterCppEquivalence::test_empty_and_non_bytes_packet_errors PASSED [ 25%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_single_fragment_reassembly PASSED [ 28%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_multi_fragment_reassembly PASSED [ 32%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_out_of_order_fragments_with_start_first PASSED [ 35%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_malformed_fragments PASSED [ 39%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_duplicate_fragments_same_data PASSED [ 42%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_duplicate_fragments_different_data PASSED [ 46%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_stale_buffer_cleanup PASSED [ 50%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_statistics_reset PASSED [ 53%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestBLEReassemblerCppEquivalence::test_internal_reassemble_method_matches_python PASSED [ 57%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_frame_deframe_round_trips[] PASSED [ 60%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_frame_deframe_round_trips[Hello, World!] PASSED [ 64%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_frame_deframe_round_trips[~\x01~] PASSED [ 67%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_frame_deframe_round_trips[}\x02}] PASSED [ 71%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_frame_deframe_round_trips[\x00\x01\x02\x03\x04\x05\x06\x07\x08\t\n\x0b\x0c\r\x0e\x0f\x10\x11\x12\x13\x14\x15\x16\x17\x18\x19\x1a\x1b\x1c\x1d\x1e\x1f !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~\x7f\x80\x81\x82\x83\x84\x85\x86\x87\x88\x89\x8a\x8b\x8c\x8d\x8e\x8f\x90\x91\x92\x93\x94\x95\x96\x97\x98\x99\x9a\x9b\x9c\x9d\x9e\x9f\xa0\xa1\xa2\xa3\xa4\xa5\xa6\xa7\xa8\xa9\xaa\xab\xac\xad\xae\xaf\xb0\xb1\xb2\xb3\xb4\xb5\xb6\xb7\xb8\xb9\xba\xbb\xbc\xbd\xbe\xbf\xc0\xc1\xc2\xc3\xc4\xc5\xc6\xc7\xc8\xc9\xca\xcb\xcc\xcd\xce\xcf\xd0\xd1\xd2\xd3\xd4\xd5\xd6\xd7\xd8\xd9\xda\xdb\xdc\xdd\xde\xdf\xe0\xe1\xe2\xe3\xe4\xe5\xe6\xe7\xe8\xe9\xea\xeb\xec\xed\xee\xef\xf0\xf1\xf2\xf3\xf4\xf5\xf6\xf7\xf8\xf9\xfa\xfb\xfc\xfd\xfe\xff] PASSED [ 75%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_many_hdlc_round_trips PASSED [ 78%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_invalid_hdlc_escape_sequences_and_frames[] PASSED [ 82%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_invalid_hdlc_escape_sequences_and_frames[~] PASSED [ 85%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_invalid_hdlc_escape_sequences_and_frames[missing-flags] PASSED [ 89%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_invalid_hdlc_escape_sequences_and_frames[~\x01~~] PASSED [ 92%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_invalid_hdlc_escape_sequences_and_frames[~}~] PASSED [ 96%]
|
||||
migration/tests/test_fragmentation_cpp_equivalence.py::TestHDLCFramerCppEquivalence::test_non_bytes_errors PASSED [100%]
|
||||
|
||||
======================================================================================== 28 passed in 0.43s =========================================================================================
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $
|
||||
```
|
||||
|
||||
|
||||
## Interpretation
|
||||
|
||||
This establishes phase-1 behavioral equivalence for the protocol-core fragmentation/reassembly layer. The C++ code is not yet integrated into BLEInterface.py for live Reticulum/BLE traffic.
|
||||
41
migration/reticulum_info_20260516_1444.txt
Normal file
41
migration/reticulum_info_20260516_1444.txt
Normal file
|
|
@ -0,0 +1,41 @@
|
|||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ date
|
||||
Sat May 16 14:41:10 PDT 2026
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ python3 -c 'import RNS; print(RNS.__file__)'
|
||||
/home/jlpoole/rnsenv/lib/python3.13/site-packages/RNS/__init__.py
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ which rnsd
|
||||
/home/jlpoole/rnsenv/bin/rnsd
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ which rnstatus
|
||||
/home/jlpoole/rnsenv/bin/rnstatus
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ which rnpath
|
||||
/home/jlpoole/rnsenv/bin/rnpath
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ equery belongs "$(which rnsd)"
|
||||
* Searching for /home/jlpoole/rnsenv/bin/rnsd ...
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ python3 -m pip show rns
|
||||
python3 -m pip show reticulum
|
||||
Name: rns
|
||||
Version: 1.1.3
|
||||
Summary: Self-configuring, encrypted and resilient mesh networking stack for LoRa, packet radio, WiFi and everything in between
|
||||
Home-page: https://reticulum.network/
|
||||
Author: Mark Qvist
|
||||
Author-email: mark@unsigned.io
|
||||
License: Reticulum License
|
||||
Location: /home/jlpoole/rnsenv/lib/python3.13/site-packages
|
||||
Requires: cryptography, pyserial
|
||||
Required-by: lxmf, lxst, nomadnet, sbapp
|
||||
WARNING: Package(s) not found: reticulum
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $ date
|
||||
hostname
|
||||
which python3
|
||||
python3 -V
|
||||
python3 -c 'import RNS, os; print("RNS file:", RNS.__file__); print("RNS dir :", os.path.dirname(RNS.__file__))'
|
||||
which rnsd
|
||||
rnsd --version 2>/dev/null || true
|
||||
Sat May 16 14:43:28 PDT 2026
|
||||
jp
|
||||
/home/jlpoole/rnsenv/bin/python3
|
||||
Python 3.13.12
|
||||
RNS file: /home/jlpoole/rnsenv/lib/python3.13/site-packages/RNS/__init__.py
|
||||
RNS dir : /home/jlpoole/rnsenv/lib/python3.13/site-packages/RNS
|
||||
/home/jlpoole/rnsenv/bin/rnsd
|
||||
rnsd 1.1.3
|
||||
(rnsenv) jlpoole@jp /usr/local/src/ble-reticulum $
|
||||
460
migration/scripts/import_codex_review.pl
Normal file
460
migration/scripts/import_codex_review.pl
Normal file
|
|
@ -0,0 +1,460 @@
|
|||
#!/usr/bin/env perl
|
||||
#
|
||||
# Codex
|
||||
#
|
||||
|
||||
use strict;
|
||||
use warnings;
|
||||
use Getopt::Long qw(GetOptions);
|
||||
use File::Basename qw(dirname);
|
||||
use Cwd qw(abs_path);
|
||||
|
||||
my $script_dir = dirname(abs_path($0));
|
||||
my $default_db = "ble_migration.sqlite";
|
||||
my $default_md = "Codex_response_20260616_1514.md";
|
||||
my $default_inventory = "ble_symbols_20260516_1429.txt";
|
||||
|
||||
my $db_path = $default_db;
|
||||
my $markdown_path = $default_md;
|
||||
my $inventory_path = $default_inventory;
|
||||
my $reviewer = 'codex';
|
||||
my $dry_run = 0;
|
||||
my $help = 0;
|
||||
|
||||
GetOptions(
|
||||
'db=s' => \$db_path,
|
||||
'markdown=s' => \$markdown_path,
|
||||
'inventory=s' => \$inventory_path,
|
||||
'reviewer=s' => \$reviewer,
|
||||
'dry-run' => \$dry_run,
|
||||
'help' => \$help,
|
||||
) or usage(2);
|
||||
|
||||
usage(0) if $help;
|
||||
|
||||
die "Markdown file not found: $markdown_path\n" unless -f $markdown_path;
|
||||
die "SQLite database not found: $db_path\n" unless -f $db_path;
|
||||
die "Inventory file not found: $inventory_path\n" unless -f $inventory_path;
|
||||
|
||||
my $inventory = load_inventory($inventory_path);
|
||||
my @rows = parse_markdown_tables($markdown_path, $inventory);
|
||||
die "No symbol rows found in $markdown_path\n" unless @rows;
|
||||
|
||||
my $sql = build_sql(\@rows, $reviewer);
|
||||
|
||||
if ($dry_run) {
|
||||
print $sql;
|
||||
exit 0;
|
||||
}
|
||||
|
||||
open(my $sqlite, '|-', 'sqlite3', $db_path)
|
||||
or die "Unable to start sqlite3 for $db_path: $!\n";
|
||||
print {$sqlite} $sql;
|
||||
close($sqlite) or die "sqlite3 import failed for $db_path\n";
|
||||
|
||||
print "Imported " . scalar(@rows) . " symbol review rows into $db_path\n";
|
||||
|
||||
sub usage {
|
||||
my ($exit_code) = @_;
|
||||
print <<"USAGE";
|
||||
Usage:
|
||||
perl migration/import_codex_review.pl [options]
|
||||
|
||||
Options:
|
||||
--db PATH SQLite database path
|
||||
default: migration/ble_migration.sqlite
|
||||
--markdown PATH Codex Markdown review path
|
||||
default: migration/Codex_response_20260616_1514.md
|
||||
--inventory PATH symbol inventory used to infer class_name for unqualified
|
||||
method rows
|
||||
default: migration/ble_symbols_20260516_1429.txt
|
||||
--reviewer NAME reviewer value for reviews table
|
||||
default: codex
|
||||
--dry-run print SQL instead of applying it
|
||||
--help show this help
|
||||
|
||||
The importer is idempotent for symbols: it upserts by the schema's unique key
|
||||
(source_file, class_name, symbol_name, line_number). It appends one review row
|
||||
per imported Markdown table row each time it is run.
|
||||
USAGE
|
||||
exit $exit_code;
|
||||
}
|
||||
|
||||
sub load_inventory {
|
||||
my ($path) = @_;
|
||||
open(my $fh, '<', $path) or die "Unable to read $path: $!\n";
|
||||
|
||||
my %inventory;
|
||||
my $current_file;
|
||||
my $current_class = '';
|
||||
|
||||
while (my $line = <$fh>) {
|
||||
chomp $line;
|
||||
|
||||
if ($line =~ /^(\S+\.py)\s*$/) {
|
||||
$current_file = $1;
|
||||
$current_class = '';
|
||||
next;
|
||||
}
|
||||
|
||||
next unless defined $current_file;
|
||||
|
||||
if ($line =~ /^\s{4}class\s+([A-Za-z_][A-Za-z0-9_]*)\s+line\s+(\d+)/) {
|
||||
my ($class, $line_number) = ($1, $2);
|
||||
$current_class = $class;
|
||||
my $key = inventory_key($current_file, $class, $line_number);
|
||||
$inventory{$key} = {
|
||||
symbol_type => 'class',
|
||||
class_name => '',
|
||||
symbol_name => $class,
|
||||
};
|
||||
next;
|
||||
}
|
||||
|
||||
if ($line =~ /^\s{8}(?:async\s+)?def\s+([A-Za-z_][A-Za-z0-9_]*)\([^)]*\)\s+line\s+(\d+)/) {
|
||||
my ($method, $line_number) = ($1, $2);
|
||||
my $key = inventory_key($current_file, $method, $line_number);
|
||||
$inventory{$key} = {
|
||||
symbol_type => 'method',
|
||||
class_name => $current_class,
|
||||
symbol_name => $method,
|
||||
};
|
||||
next;
|
||||
}
|
||||
|
||||
if ($line =~ /^\s{4}def\s+([A-Za-z_][A-Za-z0-9_]*)\([^)]*\)\s+line\s+(\d+)/) {
|
||||
my ($function, $line_number) = ($1, $2);
|
||||
my $key = inventory_key($current_file, $function, $line_number);
|
||||
$inventory{$key} = {
|
||||
symbol_type => 'function',
|
||||
class_name => '',
|
||||
symbol_name => $function,
|
||||
};
|
||||
next;
|
||||
}
|
||||
}
|
||||
|
||||
close($fh);
|
||||
return \%inventory;
|
||||
}
|
||||
|
||||
sub inventory_key {
|
||||
my ($file, $symbol, $line_number) = @_;
|
||||
return join('|', basename_only($file), $symbol, defined($line_number) ? $line_number : '');
|
||||
}
|
||||
|
||||
sub parse_markdown_tables {
|
||||
my ($path, $inventory) = @_;
|
||||
open(my $fh, '<', $path) or die "Unable to read $path: $!\n";
|
||||
|
||||
my @rows;
|
||||
my $current_file;
|
||||
my $in_expected_table = 0;
|
||||
|
||||
while (my $line = <$fh>) {
|
||||
chomp $line;
|
||||
|
||||
if ($line =~ /^\|\s*file path\s*\|\s*symbol\s*\|\s*line\s*\|\s*tag\s*\|\s*phase-1 C\+\+ candidate\s*\|/i) {
|
||||
$in_expected_table = 1;
|
||||
next;
|
||||
}
|
||||
|
||||
if ($in_expected_table && $line =~ /^\|\s*-+/) {
|
||||
next;
|
||||
}
|
||||
|
||||
if ($in_expected_table && $line !~ /^\|/) {
|
||||
$in_expected_table = 0;
|
||||
next;
|
||||
}
|
||||
|
||||
next unless $in_expected_table;
|
||||
|
||||
my @cells = split_markdown_row($line);
|
||||
next unless @cells == 8;
|
||||
|
||||
my ($file, $symbol, $line_number, $tag, $candidate, $rationale, $callers, $callees) = @cells;
|
||||
next if lc($file) eq 'file path';
|
||||
|
||||
$file = clean_cell($file);
|
||||
if ($file eq 'same') {
|
||||
die "Encountered 'same' file before an explicit file path\n" unless defined $current_file;
|
||||
$file = $current_file;
|
||||
} else {
|
||||
$current_file = $file;
|
||||
}
|
||||
|
||||
$symbol = clean_cell($symbol);
|
||||
$tag = clean_tag(clean_cell($tag));
|
||||
$candidate = lc(clean_cell($candidate)) eq 'yes' ? 1 : 0;
|
||||
$line_number = clean_cell($line_number);
|
||||
$line_number = $line_number =~ /(\d+)/ ? $1 : undef;
|
||||
|
||||
$rationale = clean_cell($rationale);
|
||||
$callers = clean_cell($callers);
|
||||
$callees = clean_cell($callees);
|
||||
|
||||
my ($symbol_type, $class_name, $symbol_name);
|
||||
my $inventory_key = inventory_key($file, symbol_lookup_name($symbol), $line_number);
|
||||
if (exists $inventory->{$inventory_key}) {
|
||||
$symbol_type = $inventory->{$inventory_key}->{symbol_type};
|
||||
$class_name = $inventory->{$inventory_key}->{class_name};
|
||||
$symbol_name = $inventory->{$inventory_key}->{symbol_name};
|
||||
} else {
|
||||
($symbol_type, $class_name, $symbol_name) = infer_symbol_shape($symbol);
|
||||
$class_name = '' unless defined $class_name;
|
||||
}
|
||||
my $phase = $candidate ? '1_candidate' : '0_inventory';
|
||||
my $confidence = $symbol =~ /\+\s*all\s+methods/i ? 'medium' : 'high';
|
||||
my $notes = "Imported from " . basename_only($path) . "; original symbol cell: $symbol";
|
||||
|
||||
push @rows, {
|
||||
source_file => $file,
|
||||
symbol_name => $symbol_name,
|
||||
symbol_type => $symbol_type,
|
||||
class_name => $class_name,
|
||||
line_number => $line_number,
|
||||
tag => $tag,
|
||||
phase => $phase,
|
||||
status => 'REVIEWED',
|
||||
cpp_candidate => $candidate,
|
||||
confidence => $confidence,
|
||||
rationale => $rationale,
|
||||
callers => $callers,
|
||||
callees => $callees,
|
||||
notes => $notes,
|
||||
};
|
||||
}
|
||||
|
||||
close($fh);
|
||||
return @rows;
|
||||
}
|
||||
|
||||
sub split_markdown_row {
|
||||
my ($line) = @_;
|
||||
$line =~ s/^\|//;
|
||||
$line =~ s/\|$//;
|
||||
my @cells = split /\|/, $line, -1;
|
||||
for my $cell (@cells) {
|
||||
$cell =~ s/^\s+//;
|
||||
$cell =~ s/\s+$//;
|
||||
}
|
||||
return @cells;
|
||||
}
|
||||
|
||||
sub clean_cell {
|
||||
my ($value) = @_;
|
||||
$value = '' unless defined $value;
|
||||
$value =~ s/<br\s*\/?>/\n/gi;
|
||||
$value =~ s/`([^`]*)`/$1/g;
|
||||
$value =~ s/^\s+//;
|
||||
$value =~ s/\s+$//;
|
||||
return $value;
|
||||
}
|
||||
|
||||
sub clean_tag {
|
||||
my ($tag) = @_;
|
||||
$tag =~ s/^\[//;
|
||||
$tag =~ s/\]$//;
|
||||
$tag = uc($tag || 'UNKNOWN');
|
||||
return $tag =~ /^(CORE|GLUE|PLATFORM|TEST|UNKNOWN)$/ ? $tag : 'UNKNOWN';
|
||||
}
|
||||
|
||||
sub infer_symbol_shape {
|
||||
my ($symbol) = @_;
|
||||
|
||||
if ($symbol =~ /^([A-Za-z_][A-Za-z0-9_]*)\s*\+\s*all\s+methods\b/i) {
|
||||
return ('class_group', undef, $symbol);
|
||||
}
|
||||
|
||||
if ($symbol =~ /^([A-Za-z_][A-Za-z0-9_]*)\.([A-Za-z_][A-Za-z0-9_]*)$/) {
|
||||
return ('method', $1, $2);
|
||||
}
|
||||
|
||||
if ($symbol =~ /^_[A-Za-z_][A-Za-z0-9_]*$/ || $symbol =~ /^[a-z_][A-Za-z0-9_]*$/) {
|
||||
return ('function', undef, $symbol);
|
||||
}
|
||||
|
||||
if ($symbol =~ /^[A-Z][A-Za-z0-9_]*$/) {
|
||||
return ('class', undef, $symbol);
|
||||
}
|
||||
|
||||
return ('symbol', undef, $symbol);
|
||||
}
|
||||
|
||||
sub symbol_lookup_name {
|
||||
my ($symbol) = @_;
|
||||
if ($symbol =~ /^([A-Za-z_][A-Za-z0-9_]*)\.([A-Za-z_][A-Za-z0-9_]*)$/) {
|
||||
return $2;
|
||||
}
|
||||
if ($symbol =~ /^([A-Za-z_][A-Za-z0-9_]*)\s*\+\s*all\s+methods\b/i) {
|
||||
return $1;
|
||||
}
|
||||
return $symbol;
|
||||
}
|
||||
|
||||
sub build_sql {
|
||||
my ($rows, $reviewer_name) = @_;
|
||||
|
||||
my $sql = <<"SQL";
|
||||
.bail on
|
||||
PRAGMA foreign_keys = ON;
|
||||
BEGIN;
|
||||
|
||||
CREATE TEMP TABLE import_symbols (
|
||||
import_order INTEGER PRIMARY KEY,
|
||||
source_file TEXT NOT NULL,
|
||||
symbol_name TEXT NOT NULL,
|
||||
symbol_type TEXT,
|
||||
class_name TEXT,
|
||||
line_number INTEGER,
|
||||
tag TEXT NOT NULL,
|
||||
phase TEXT NOT NULL,
|
||||
status TEXT NOT NULL,
|
||||
cpp_candidate INTEGER NOT NULL,
|
||||
confidence TEXT,
|
||||
rationale TEXT,
|
||||
callers TEXT,
|
||||
callees TEXT,
|
||||
notes TEXT
|
||||
);
|
||||
|
||||
SQL
|
||||
|
||||
my $i = 0;
|
||||
for my $row (@{$rows}) {
|
||||
$i++;
|
||||
$sql .= "INSERT INTO import_symbols VALUES ("
|
||||
. join(', ',
|
||||
$i,
|
||||
sql_quote($row->{source_file}),
|
||||
sql_quote($row->{symbol_name}),
|
||||
sql_quote($row->{symbol_type}),
|
||||
sql_quote($row->{class_name}),
|
||||
sql_integer($row->{line_number}),
|
||||
sql_quote($row->{tag}),
|
||||
sql_quote($row->{phase}),
|
||||
sql_quote($row->{status}),
|
||||
$row->{cpp_candidate} ? 1 : 0,
|
||||
sql_quote($row->{confidence}),
|
||||
sql_quote($row->{rationale}),
|
||||
sql_quote($row->{callers}),
|
||||
sql_quote($row->{callees}),
|
||||
sql_quote($row->{notes}),
|
||||
)
|
||||
. ");\n";
|
||||
}
|
||||
|
||||
$sql .= <<"SQL";
|
||||
|
||||
CREATE TEMP TABLE import_existing AS
|
||||
SELECT
|
||||
i.import_order,
|
||||
s.symbol_id,
|
||||
s.tag AS old_tag,
|
||||
s.status AS old_status
|
||||
FROM import_symbols i
|
||||
JOIN symbols s
|
||||
ON s.source_file = i.source_file
|
||||
AND COALESCE(s.class_name, '') = COALESCE(i.class_name, '')
|
||||
AND s.symbol_name = i.symbol_name
|
||||
AND COALESCE(s.line_number, -1) = COALESCE(i.line_number, -1);
|
||||
|
||||
INSERT INTO symbols (
|
||||
source_file,
|
||||
symbol_name,
|
||||
symbol_type,
|
||||
class_name,
|
||||
line_number,
|
||||
tag,
|
||||
phase,
|
||||
status,
|
||||
cpp_candidate,
|
||||
confidence,
|
||||
rationale,
|
||||
callers,
|
||||
callees,
|
||||
notes,
|
||||
updated_at
|
||||
)
|
||||
SELECT
|
||||
source_file,
|
||||
symbol_name,
|
||||
symbol_type,
|
||||
class_name,
|
||||
line_number,
|
||||
tag,
|
||||
phase,
|
||||
status,
|
||||
cpp_candidate,
|
||||
confidence,
|
||||
rationale,
|
||||
callers,
|
||||
callees,
|
||||
notes,
|
||||
CURRENT_TIMESTAMP
|
||||
FROM import_symbols
|
||||
WHERE 1
|
||||
ON CONFLICT(source_file, class_name, symbol_name, line_number) DO UPDATE SET
|
||||
symbol_type = excluded.symbol_type,
|
||||
tag = excluded.tag,
|
||||
phase = excluded.phase,
|
||||
status = excluded.status,
|
||||
cpp_candidate = excluded.cpp_candidate,
|
||||
confidence = excluded.confidence,
|
||||
rationale = excluded.rationale,
|
||||
callers = excluded.callers,
|
||||
callees = excluded.callees,
|
||||
notes = excluded.notes,
|
||||
updated_at = CURRENT_TIMESTAMP;
|
||||
|
||||
INSERT INTO reviews (
|
||||
symbol_id,
|
||||
reviewer,
|
||||
old_tag,
|
||||
new_tag,
|
||||
old_status,
|
||||
new_status,
|
||||
note
|
||||
)
|
||||
SELECT
|
||||
s.symbol_id,
|
||||
@{[sql_quote($reviewer_name)]},
|
||||
e.old_tag,
|
||||
i.tag,
|
||||
e.old_status,
|
||||
i.status,
|
||||
'Imported from Codex Markdown review. Rationale: ' || COALESCE(i.rationale, '')
|
||||
FROM import_symbols i
|
||||
JOIN symbols s
|
||||
ON s.source_file = i.source_file
|
||||
AND COALESCE(s.class_name, '') = COALESCE(i.class_name, '')
|
||||
AND s.symbol_name = i.symbol_name
|
||||
AND COALESCE(s.line_number, -1) = COALESCE(i.line_number, -1)
|
||||
LEFT JOIN import_existing e
|
||||
ON e.import_order = i.import_order;
|
||||
|
||||
COMMIT;
|
||||
SQL
|
||||
|
||||
return $sql;
|
||||
}
|
||||
|
||||
sub sql_quote {
|
||||
my ($value) = @_;
|
||||
return 'NULL' unless defined $value;
|
||||
$value =~ s/'/''/g;
|
||||
return "'$value'";
|
||||
}
|
||||
|
||||
sub sql_integer {
|
||||
my ($value) = @_;
|
||||
return 'NULL' unless defined $value && $value =~ /^\d+$/;
|
||||
return $value;
|
||||
}
|
||||
|
||||
sub basename_only {
|
||||
my ($path) = @_;
|
||||
$path =~ s{.*/}{};
|
||||
return $path;
|
||||
}
|
||||
58
migration/sql/ble_migration_schema.sql
Normal file
58
migration/sql/ble_migration_schema.sql
Normal file
|
|
@ -0,0 +1,58 @@
|
|||
-- 20260516 ChatGPT
|
||||
-- $Header$
|
||||
--
|
||||
-- Example:
|
||||
-- cd /usr/local/src/ble-reticulum/migration
|
||||
-- sqlite3 ble_migration.sqlite ".read sql/ble_migration_schema.sql"
|
||||
-- Verify
|
||||
-- sqlite3 ble_migration.sqlite ".tables"
|
||||
-- sqlite3 ble_migration.sqlite ".schema symbols"
|
||||
--
|
||||
-- Purpose:
|
||||
-- Track symbol-by-symbol migration status for BLE Reticulum protocol extraction.
|
||||
|
||||
CREATE TABLE IF NOT EXISTS symbols (
|
||||
symbol_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
source_file TEXT NOT NULL,
|
||||
symbol_name TEXT NOT NULL,
|
||||
symbol_type TEXT,
|
||||
class_name TEXT,
|
||||
line_number INTEGER,
|
||||
tag TEXT DEFAULT 'UNKNOWN',
|
||||
phase TEXT DEFAULT '0_inventory',
|
||||
status TEXT DEFAULT 'DISCOVERED',
|
||||
cpp_candidate INTEGER DEFAULT 0,
|
||||
confidence TEXT DEFAULT 'unknown',
|
||||
rationale TEXT,
|
||||
callers TEXT,
|
||||
callees TEXT,
|
||||
notes TEXT,
|
||||
first_seen_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
|
||||
UNIQUE(source_file, class_name, symbol_name, line_number)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS reviews (
|
||||
review_id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
symbol_id INTEGER NOT NULL REFERENCES symbols(symbol_id),
|
||||
reviewed_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
reviewer TEXT,
|
||||
old_tag TEXT,
|
||||
new_tag TEXT,
|
||||
old_status TEXT,
|
||||
new_status TEXT,
|
||||
note TEXT
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS symbols_tag_idx
|
||||
ON symbols(tag);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS symbols_status_idx
|
||||
ON symbols(status);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS symbols_phase_idx
|
||||
ON symbols(phase);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS symbols_cpp_candidate_idx
|
||||
ON symbols(cpp_candidate);
|
||||
5
migration/sql/do_not_port_yet.sql
Normal file
5
migration/sql/do_not_port_yet.sql
Normal file
|
|
@ -0,0 +1,5 @@
|
|||
-- Do-not-port-yet list
|
||||
SELECT source_file, class_name, symbol_name, tag, rationale
|
||||
FROM symbols
|
||||
WHERE tag IN ('GLUE', 'PLATFORM')
|
||||
ORDER BY tag, source_file, line_number;
|
||||
6
migration/sql/need_review.sql
Normal file
6
migration/sql/need_review.sql
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
-- Things needing your review
|
||||
SELECT source_file, class_name, symbol_name, line_number, tag, rationale
|
||||
FROM symbols
|
||||
WHERE status = 'NEEDS_REVIEW'
|
||||
OR tag = 'UNKNOWN'
|
||||
ORDER BY source_file, line_number;
|
||||
6
migration/sql/phase_1_candidates.sql
Normal file
6
migration/sql/phase_1_candidates.sql
Normal file
|
|
@ -0,0 +1,6 @@
|
|||
-- Phase-1 C++ candidates
|
||||
SELECT source_file, class_name, symbol_name, line_number, rationale
|
||||
FROM symbols
|
||||
WHERE tag = 'CORE'
|
||||
AND cpp_candidate = 1
|
||||
ORDER BY source_file, line_number;
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
UPDATE symbols
|
||||
SET status = 'TESTED',
|
||||
phase = '1_protocol_core',
|
||||
notes = COALESCE(notes, '') || char(10) ||
|
||||
'2026-05-16: C++ equivalence tests passed: 28 tests in migration/tests/test_fragmentation_cpp_equivalence.py.'
|
||||
WHERE source_file = 'src/ble_reticulum/BLEFragmentation.py'
|
||||
AND symbol_name IN (
|
||||
'BLEFragmenter',
|
||||
'__init__',
|
||||
'fragment_packet',
|
||||
'get_fragment_overhead',
|
||||
'BLEReassembler',
|
||||
'receive_fragment',
|
||||
'_reassemble',
|
||||
'cleanup_stale_buffers',
|
||||
'get_statistics',
|
||||
'reset_statistics',
|
||||
'HDLCFramer',
|
||||
'frame_packet',
|
||||
'deframe_packet'
|
||||
);
|
||||
179
migration/tests/test_fragmentation_backend_shim.py
Normal file
179
migration/tests/test_fragmentation_backend_shim.py
Normal file
|
|
@ -0,0 +1,179 @@
|
|||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
SRC_DIR = os.path.join(REPO_ROOT, "src")
|
||||
CPP_BUILD_DIR = os.path.join(REPO_ROOT, "migration", "protocol_core")
|
||||
|
||||
|
||||
PROBE = r"""
|
||||
import json
|
||||
from ble_reticulum.BLEFragmentationBackend import (
|
||||
BACKEND,
|
||||
BLEFragmenter,
|
||||
BLEReassembler,
|
||||
HDLCFramer,
|
||||
)
|
||||
|
||||
fragmenter = BLEFragmenter(mtu=20)
|
||||
reassembler = BLEReassembler(timeout=0.1)
|
||||
packet = b"abc"
|
||||
fragments = fragmenter.fragment_packet(packet)
|
||||
result = reassembler.receive_fragment(fragments[0], "device1")
|
||||
framed = HDLCFramer.frame_packet(packet)
|
||||
deframed = HDLCFramer.deframe_packet(framed)
|
||||
|
||||
print(json.dumps({
|
||||
"backend": BACKEND,
|
||||
"fragmenter_module": BLEFragmenter.__module__,
|
||||
"reassembler_module": BLEReassembler.__module__,
|
||||
"hdlc_module": HDLCFramer.__module__,
|
||||
"fragment": fragments[0].hex(),
|
||||
"result": result.hex(),
|
||||
"deframed": deframed.hex(),
|
||||
"stats_keys": sorted(reassembler.get_statistics().keys()),
|
||||
"fragmenter_methods": sorted(
|
||||
name for name in dir(BLEFragmenter) if name in {
|
||||
"fragment_packet",
|
||||
"get_fragment_overhead",
|
||||
}
|
||||
),
|
||||
"reassembler_methods": sorted(
|
||||
name for name in dir(BLEReassembler) if name in {
|
||||
"receive_fragment",
|
||||
"_reassemble",
|
||||
"cleanup_stale_buffers",
|
||||
"get_statistics",
|
||||
"reset_statistics",
|
||||
}
|
||||
),
|
||||
"hdlc_methods": sorted(
|
||||
name for name in dir(HDLCFramer) if name in {
|
||||
"frame_packet",
|
||||
"deframe_packet",
|
||||
}
|
||||
),
|
||||
}))
|
||||
"""
|
||||
|
||||
|
||||
def run_probe(backend, include_cpp):
|
||||
env = os.environ.copy()
|
||||
env["BLE_RETICULUM_FRAGMENTATION_BACKEND"] = backend
|
||||
pythonpath = [SRC_DIR]
|
||||
if include_cpp:
|
||||
pythonpath.append(CPP_BUILD_DIR)
|
||||
if env.get("PYTHONPATH"):
|
||||
pythonpath.append(env["PYTHONPATH"])
|
||||
env["PYTHONPATH"] = os.pathsep.join(pythonpath)
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "-c", PROBE],
|
||||
cwd=REPO_ROOT,
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
check=True,
|
||||
)
|
||||
return json.loads(completed.stdout)
|
||||
|
||||
|
||||
def test_cpp_backend_loads_when_available():
|
||||
result = run_probe("cpp", include_cpp=True)
|
||||
|
||||
assert result["backend"] == "cpp"
|
||||
assert result["fragmenter_module"] == "ble_protocol_core_cpp"
|
||||
assert result["reassembler_module"] == "ble_protocol_core_cpp"
|
||||
assert result["hdlc_module"] == "ble_protocol_core_cpp"
|
||||
assert result["fragment"] == "0100000001616263"
|
||||
assert result["result"] == "616263"
|
||||
assert result["deframed"] == "616263"
|
||||
|
||||
|
||||
def test_python_backend_still_works_when_cpp_backend_is_unavailable():
|
||||
result = run_probe("auto", include_cpp=False)
|
||||
|
||||
assert result["backend"] == "python"
|
||||
assert result["fragmenter_module"] == "ble_reticulum.BLEFragmentation"
|
||||
assert result["fragment"] == "0100000001616263"
|
||||
assert result["result"] == "616263"
|
||||
assert result["deframed"] == "616263"
|
||||
|
||||
|
||||
def test_auto_backend_prefers_cpp_when_available():
|
||||
result = run_probe("auto", include_cpp=True)
|
||||
|
||||
assert result["backend"] == "cpp"
|
||||
assert result["fragmenter_module"] == "ble_protocol_core_cpp"
|
||||
|
||||
|
||||
def test_python_backend_can_be_forced_even_when_cpp_backend_is_available():
|
||||
result = run_probe("python", include_cpp=True)
|
||||
|
||||
assert result["backend"] == "python"
|
||||
assert result["fragmenter_module"] == "ble_reticulum.BLEFragmentation"
|
||||
|
||||
|
||||
def test_cpp_backend_request_fails_clearly_when_unavailable():
|
||||
env = os.environ.copy()
|
||||
env["BLE_RETICULUM_FRAGMENTATION_BACKEND"] = "cpp"
|
||||
env["PYTHONPATH"] = SRC_DIR
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "-c", "import ble_reticulum.BLEFragmentationBackend"],
|
||||
cwd=REPO_ROOT,
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
assert completed.returncode != 0
|
||||
assert "C++ BLE fragmentation backend is not available" in completed.stderr
|
||||
|
||||
|
||||
def test_selected_backend_exposes_methods_expected_by_current_python_code():
|
||||
result = run_probe("cpp", include_cpp=True)
|
||||
|
||||
assert result["fragmenter_methods"] == [
|
||||
"fragment_packet",
|
||||
"get_fragment_overhead",
|
||||
]
|
||||
assert result["reassembler_methods"] == [
|
||||
"_reassemble",
|
||||
"cleanup_stale_buffers",
|
||||
"get_statistics",
|
||||
"receive_fragment",
|
||||
"reset_statistics",
|
||||
]
|
||||
assert result["hdlc_methods"] == [
|
||||
"deframe_packet",
|
||||
"frame_packet",
|
||||
]
|
||||
assert result["stats_keys"] == [
|
||||
"fragments_received",
|
||||
"packets_reassembled",
|
||||
"packets_timeout",
|
||||
"pending_packets",
|
||||
]
|
||||
|
||||
|
||||
def test_invalid_backend_setting_fails_clearly():
|
||||
env = os.environ.copy()
|
||||
env["BLE_RETICULUM_FRAGMENTATION_BACKEND"] = "rust"
|
||||
env["PYTHONPATH"] = SRC_DIR
|
||||
|
||||
completed = subprocess.run(
|
||||
[sys.executable, "-c", "import ble_reticulum.BLEFragmentationBackend"],
|
||||
cwd=REPO_ROOT,
|
||||
env=env,
|
||||
text=True,
|
||||
capture_output=True,
|
||||
)
|
||||
|
||||
assert completed.returncode != 0
|
||||
assert "expected 'auto', 'cpp', or 'python'" in completed.stderr
|
||||
309
migration/tests/test_fragmentation_cpp_equivalence.py
Normal file
309
migration/tests/test_fragmentation_cpp_equivalence.py
Normal file
|
|
@ -0,0 +1,309 @@
|
|||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
|
||||
REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))
|
||||
SRC_DIR = os.path.join(REPO_ROOT, "src")
|
||||
CPP_BUILD_DIR = os.path.join(REPO_ROOT, "migration", "protocol_core")
|
||||
|
||||
sys.path.insert(0, SRC_DIR)
|
||||
sys.path.insert(0, CPP_BUILD_DIR)
|
||||
|
||||
from ble_reticulum.BLEFragmentation import ( # noqa: E402
|
||||
BLEFragmenter as PyBLEFragmenter,
|
||||
)
|
||||
from ble_reticulum.BLEFragmentation import ( # noqa: E402
|
||||
BLEReassembler as PyBLEReassembler,
|
||||
)
|
||||
from ble_reticulum.BLEFragmentation import HDLCFramer as PyHDLCFramer # noqa: E402
|
||||
|
||||
cpp = pytest.importorskip(
|
||||
"ble_protocol_core_cpp",
|
||||
reason=(
|
||||
"compiled pybind11 module missing; build with "
|
||||
"`python3 migration/protocol_core/setup.py build_ext --inplace`"
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
CppBLEFragmenter = cpp.BLEFragmenter
|
||||
CppBLEReassembler = cpp.BLEReassembler
|
||||
CppHDLCFramer = cpp.HDLCFramer
|
||||
|
||||
|
||||
def assert_same_exception(py_callable, cpp_callable):
|
||||
with pytest.raises(Exception) as py_exc:
|
||||
py_callable()
|
||||
with pytest.raises(Exception) as cpp_exc:
|
||||
cpp_callable()
|
||||
|
||||
assert type(cpp_exc.value) is type(py_exc.value)
|
||||
assert str(cpp_exc.value) == str(py_exc.value)
|
||||
|
||||
|
||||
def compare_fragmenter(mtu, packet):
|
||||
py_fragmenter = PyBLEFragmenter(mtu=mtu)
|
||||
cpp_fragmenter = CppBLEFragmenter(mtu=mtu)
|
||||
|
||||
py_fragments = py_fragmenter.fragment_packet(packet)
|
||||
cpp_fragments = cpp_fragmenter.fragment_packet(packet)
|
||||
|
||||
assert cpp_fragmenter.mtu == py_fragmenter.mtu
|
||||
assert cpp_fragmenter.payload_size == py_fragmenter.payload_size
|
||||
assert cpp_fragments == py_fragments
|
||||
assert cpp_fragmenter.get_fragment_overhead(len(packet)) == py_fragmenter.get_fragment_overhead(
|
||||
len(packet)
|
||||
)
|
||||
return py_fragments, cpp_fragments
|
||||
|
||||
|
||||
def compare_reassembly(mtu, packet, order=None, sender_id="device1"):
|
||||
py_fragments, cpp_fragments = compare_fragmenter(mtu, packet)
|
||||
py_reassembler = PyBLEReassembler()
|
||||
cpp_reassembler = CppBLEReassembler()
|
||||
|
||||
if order is None:
|
||||
order = list(range(len(py_fragments)))
|
||||
|
||||
py_result = None
|
||||
cpp_result = None
|
||||
for index in order:
|
||||
py_result = py_reassembler.receive_fragment(py_fragments[index], sender_id)
|
||||
cpp_result = cpp_reassembler.receive_fragment(cpp_fragments[index], sender_id)
|
||||
assert cpp_result == py_result
|
||||
|
||||
assert py_result == packet
|
||||
assert cpp_result == packet
|
||||
assert cpp_reassembler.get_statistics() == py_reassembler.get_statistics()
|
||||
|
||||
|
||||
class TestBLEFragmenterCppEquivalence:
|
||||
def test_single_fragment_packets(self):
|
||||
for mtu, packet in [
|
||||
(185, b"Hello, Reticulum!"),
|
||||
(20, b"A"),
|
||||
(50, bytes(range(10))),
|
||||
]:
|
||||
py_fragments, cpp_fragments = compare_fragmenter(mtu, packet)
|
||||
assert len(py_fragments) == 1
|
||||
assert len(cpp_fragments) == 1
|
||||
|
||||
def test_multi_fragment_packets(self):
|
||||
for mtu, packet in [
|
||||
(185, b"A" * 500),
|
||||
(100, b"B" * 300),
|
||||
(20, bytes(range(256))),
|
||||
]:
|
||||
py_fragments, cpp_fragments = compare_fragmenter(mtu, packet)
|
||||
assert len(py_fragments) > 1
|
||||
assert len(cpp_fragments) > 1
|
||||
compare_reassembly(mtu, packet)
|
||||
|
||||
@pytest.mark.parametrize("mtu", [20, 23, 50, 185])
|
||||
def test_mtu_boundary_sizes(self, mtu):
|
||||
payload_size = max(mtu, 20) - PyBLEFragmenter.HEADER_SIZE
|
||||
sizes = [
|
||||
1,
|
||||
payload_size - 1,
|
||||
payload_size,
|
||||
payload_size + 1,
|
||||
payload_size * 2,
|
||||
payload_size * 2 + 1,
|
||||
]
|
||||
|
||||
for size in sizes:
|
||||
if size <= 0:
|
||||
continue
|
||||
packet = bytes((i % 251 for i in range(size)))
|
||||
compare_reassembly(mtu, packet)
|
||||
|
||||
def test_empty_and_non_bytes_packet_errors(self):
|
||||
assert_same_exception(
|
||||
lambda: PyBLEFragmenter().fragment_packet(b""),
|
||||
lambda: CppBLEFragmenter().fragment_packet(b""),
|
||||
)
|
||||
with pytest.raises(TypeError):
|
||||
CppBLEFragmenter().fragment_packet("not bytes")
|
||||
with pytest.raises(TypeError):
|
||||
PyBLEFragmenter().fragment_packet("not bytes")
|
||||
|
||||
|
||||
class TestBLEReassemblerCppEquivalence:
|
||||
def test_single_fragment_reassembly(self):
|
||||
compare_reassembly(185, b"Short message")
|
||||
|
||||
def test_multi_fragment_reassembly(self):
|
||||
compare_reassembly(100, b"E" * 300)
|
||||
|
||||
def test_out_of_order_fragments_with_start_first(self):
|
||||
packet = b"F" * 150
|
||||
py_fragments, _ = compare_fragmenter(50, packet)
|
||||
assert len(py_fragments) == 4
|
||||
compare_reassembly(50, packet, order=[0, 2, 1, 3])
|
||||
|
||||
def test_malformed_fragments(self):
|
||||
cases = [
|
||||
b"\x01\x00",
|
||||
b"\xff\x00\x00\x00\x01payload",
|
||||
b"\x01\x00\x01\x00\x01payload",
|
||||
b"\x01\x00\x00\x00\x00payload",
|
||||
]
|
||||
|
||||
for fragment in cases:
|
||||
py_reassembler = PyBLEReassembler()
|
||||
cpp_reassembler = CppBLEReassembler()
|
||||
assert_same_exception(
|
||||
lambda fragment=fragment: py_reassembler.receive_fragment(fragment, "device1"),
|
||||
lambda fragment=fragment: cpp_reassembler.receive_fragment(fragment, "device1"),
|
||||
)
|
||||
|
||||
def test_duplicate_fragments_same_data(self):
|
||||
packet = b"D" * 160
|
||||
py_fragments, cpp_fragments = compare_fragmenter(50, packet)
|
||||
py_reassembler = PyBLEReassembler()
|
||||
cpp_reassembler = CppBLEReassembler()
|
||||
|
||||
assert py_reassembler.receive_fragment(py_fragments[0], "device1") is None
|
||||
assert cpp_reassembler.receive_fragment(cpp_fragments[0], "device1") is None
|
||||
|
||||
assert py_reassembler.receive_fragment(py_fragments[1], "device1") is None
|
||||
assert cpp_reassembler.receive_fragment(cpp_fragments[1], "device1") is None
|
||||
|
||||
assert py_reassembler.receive_fragment(py_fragments[1], "device1") is None
|
||||
assert cpp_reassembler.receive_fragment(cpp_fragments[1], "device1") is None
|
||||
|
||||
py_result = None
|
||||
cpp_result = None
|
||||
for index in range(2, len(py_fragments)):
|
||||
py_result = py_reassembler.receive_fragment(py_fragments[index], "device1")
|
||||
cpp_result = cpp_reassembler.receive_fragment(cpp_fragments[index], "device1")
|
||||
assert cpp_result == py_result
|
||||
|
||||
assert py_result == packet
|
||||
assert cpp_result == packet
|
||||
|
||||
def test_duplicate_fragments_different_data(self):
|
||||
packet = b"Q" * 160
|
||||
py_fragments, cpp_fragments = compare_fragmenter(50, packet)
|
||||
|
||||
py_reassembler = PyBLEReassembler()
|
||||
cpp_reassembler = CppBLEReassembler()
|
||||
py_reassembler.receive_fragment(py_fragments[0], "device1")
|
||||
cpp_reassembler.receive_fragment(cpp_fragments[0], "device1")
|
||||
py_reassembler.receive_fragment(py_fragments[1], "device1")
|
||||
cpp_reassembler.receive_fragment(cpp_fragments[1], "device1")
|
||||
|
||||
py_bad = bytearray(py_fragments[1])
|
||||
cpp_bad = bytearray(cpp_fragments[1])
|
||||
py_bad[-1] ^= 0x01
|
||||
cpp_bad[-1] ^= 0x01
|
||||
|
||||
assert_same_exception(
|
||||
lambda: py_reassembler.receive_fragment(bytes(py_bad), "device1"),
|
||||
lambda: cpp_reassembler.receive_fragment(bytes(cpp_bad), "device1"),
|
||||
)
|
||||
assert len(cpp_reassembler.reassembly_buffers) == len(py_reassembler.reassembly_buffers)
|
||||
|
||||
def test_stale_buffer_cleanup(self):
|
||||
packet = b"G" * 300
|
||||
py_fragments, cpp_fragments = compare_fragmenter(100, packet)
|
||||
py_reassembler = PyBLEReassembler(timeout=0.1)
|
||||
cpp_reassembler = CppBLEReassembler(timeout=0.1)
|
||||
|
||||
assert py_reassembler.receive_fragment(py_fragments[0], "device1") is None
|
||||
assert cpp_reassembler.receive_fragment(cpp_fragments[0], "device1") is None
|
||||
assert len(cpp_reassembler.reassembly_buffers) == len(py_reassembler.reassembly_buffers)
|
||||
|
||||
time.sleep(0.2)
|
||||
|
||||
assert cpp_reassembler.cleanup_stale_buffers() == py_reassembler.cleanup_stale_buffers()
|
||||
assert len(cpp_reassembler.reassembly_buffers) == len(py_reassembler.reassembly_buffers)
|
||||
assert cpp_reassembler.get_statistics() == py_reassembler.get_statistics()
|
||||
|
||||
def test_statistics_reset(self):
|
||||
packet = b"H" * 300
|
||||
py_fragments, cpp_fragments = compare_fragmenter(100, packet)
|
||||
py_reassembler = PyBLEReassembler()
|
||||
cpp_reassembler = CppBLEReassembler()
|
||||
|
||||
for py_fragment, cpp_fragment in zip(py_fragments, cpp_fragments):
|
||||
assert cpp_reassembler.receive_fragment(cpp_fragment, "device1") == py_reassembler.receive_fragment(
|
||||
py_fragment, "device1"
|
||||
)
|
||||
|
||||
assert cpp_reassembler.get_statistics() == py_reassembler.get_statistics()
|
||||
py_reassembler.reset_statistics()
|
||||
cpp_reassembler.reset_statistics()
|
||||
assert cpp_reassembler.get_statistics() == py_reassembler.get_statistics()
|
||||
|
||||
def test_internal_reassemble_method_matches_python(self):
|
||||
py_reassembler = PyBLEReassembler()
|
||||
cpp_reassembler = CppBLEReassembler()
|
||||
buffer = {"fragments": {0: b"abc", 1: b"def", 2: b"ghi"}, "total": 3}
|
||||
|
||||
assert cpp_reassembler._reassemble(buffer) == py_reassembler._reassemble(buffer)
|
||||
|
||||
malformed = {"fragments": {0: b"abc", 2: b"ghi"}, "total": 3}
|
||||
assert_same_exception(
|
||||
lambda: py_reassembler._reassemble(malformed),
|
||||
lambda: cpp_reassembler._reassemble(malformed),
|
||||
)
|
||||
|
||||
|
||||
class TestHDLCFramerCppEquivalence:
|
||||
@pytest.mark.parametrize(
|
||||
"packet",
|
||||
[
|
||||
b"",
|
||||
b"Hello, World!",
|
||||
bytes([0x7E, 0x01, 0x7E]),
|
||||
bytes([0x7D, 0x02, 0x7D]),
|
||||
bytes(range(256)),
|
||||
],
|
||||
)
|
||||
def test_frame_deframe_round_trips(self, packet):
|
||||
py_framed = PyHDLCFramer.frame_packet(packet)
|
||||
cpp_framed = CppHDLCFramer.frame_packet(packet)
|
||||
assert cpp_framed == py_framed
|
||||
|
||||
assert CppHDLCFramer.deframe_packet(cpp_framed) == PyHDLCFramer.deframe_packet(
|
||||
py_framed
|
||||
)
|
||||
assert CppHDLCFramer.deframe_packet(cpp_framed) == packet
|
||||
|
||||
def test_many_hdlc_round_trips(self):
|
||||
for value in range(256):
|
||||
packet = bytes([value] * 10)
|
||||
py_framed = PyHDLCFramer.frame_packet(packet)
|
||||
cpp_framed = CppHDLCFramer.frame_packet(packet)
|
||||
assert cpp_framed == py_framed
|
||||
assert CppHDLCFramer.deframe_packet(cpp_framed) == packet
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"frame",
|
||||
[
|
||||
b"",
|
||||
b"\x7e",
|
||||
b"missing-flags",
|
||||
bytes([PyHDLCFramer.FLAG, 0x01, PyHDLCFramer.FLAG, PyHDLCFramer.FLAG]),
|
||||
bytes([PyHDLCFramer.FLAG, PyHDLCFramer.ESCAPE, PyHDLCFramer.FLAG]),
|
||||
],
|
||||
)
|
||||
def test_invalid_hdlc_escape_sequences_and_frames(self, frame):
|
||||
assert_same_exception(
|
||||
lambda frame=frame: PyHDLCFramer.deframe_packet(frame),
|
||||
lambda frame=frame: CppHDLCFramer.deframe_packet(frame),
|
||||
)
|
||||
|
||||
def test_non_bytes_errors(self):
|
||||
with pytest.raises(TypeError):
|
||||
PyHDLCFramer.frame_packet("not bytes")
|
||||
with pytest.raises(TypeError):
|
||||
CppHDLCFramer.frame_packet("not bytes")
|
||||
with pytest.raises(TypeError):
|
||||
PyHDLCFramer.deframe_packet("not bytes")
|
||||
with pytest.raises(TypeError):
|
||||
CppHDLCFramer.deframe_packet("not bytes")
|
||||
|
|
@ -64,10 +64,14 @@ package-dir = {"" = "src"}
|
|||
[tool.setuptools.package-data]
|
||||
"ble_reticulum" = ["*.py"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
testpaths = ["tests"]
|
||||
python_files = ["test_*.py"]
|
||||
python_classes = ["Test*"]
|
||||
python_functions = ["test_*"]
|
||||
asyncio_mode = "auto"
|
||||
addopts = "-v --tb=short"
|
||||
#
|
||||
# 5/16/26 jlpoole: remming to stop warning
|
||||
# (WARNING: ignoring pytest config in pyproject.toml!)
|
||||
#
|
||||
#[tool.pytest.ini_options]
|
||||
#testpaths = ["tests"]
|
||||
#python_files = ["test_*.py"]
|
||||
#python_classes = ["Test*"]
|
||||
#python_functions = ["test_*"]
|
||||
#asyncio_mode = "auto"
|
||||
#addopts = "-v --tb=short"
|
||||
|
|
|
|||
|
|
@ -15,7 +15,10 @@ addopts =
|
|||
-v
|
||||
--strict-markers
|
||||
--tb=short
|
||||
--disable-warnings
|
||||
#
|
||||
# 5/16/26 jlpoole: we want the warnings for the C++ migration
|
||||
# so next line is rem'd out
|
||||
#--disable-warnings
|
||||
# Legacy tests with import issues - run explicitly if needed
|
||||
--ignore=tests/test_bleak_threading_hang.py
|
||||
--ignore=tests/test_bleak_with_exec_loading.py
|
||||
|
|
|
|||
35
samples/If_full.txt
Normal file
35
samples/If_full.txt
Normal file
|
|
@ -0,0 +1,35 @@
|
|||
If you can keep your head when all about you
|
||||
Are losing theirs and blaming it on you,
|
||||
If you can trust yourself when all men doubt you,
|
||||
But make allowance for their doubting too;
|
||||
If you can wait and not be tired by waiting,
|
||||
Or being lied about, don’t deal in lies,
|
||||
Or being hated, don’t give way to hating,
|
||||
And yet don’t look too good, nor talk too wise:
|
||||
|
||||
If you can dream—and not make dreams your master;
|
||||
If you can think—and not make thoughts your aim;
|
||||
If you can meet with Triumph and Disaster
|
||||
And treat those two impostors just the same;
|
||||
If you can bear to hear the truth you’ve spoken
|
||||
Twisted by knaves to make a trap for fools,
|
||||
Or watch the things you gave your life to, broken,
|
||||
And stoop and build ’em up with worn-out tools:
|
||||
|
||||
If you can make one heap of all your winnings
|
||||
And risk it on one turn of pitch-and-toss,
|
||||
And lose, and start again at your beginnings
|
||||
And never breathe a word about your loss;
|
||||
If you can force your heart and nerve and sinew
|
||||
To serve your turn long after they are gone,
|
||||
And so hold on when there is nothing in you
|
||||
Except the Will which says to them: ‘Hold on!’
|
||||
|
||||
If you can talk with crowds and keep your virtue,
|
||||
Or walk with Kings—nor lose the common touch,
|
||||
If neither foes nor loving friends can hurt you,
|
||||
If all men count with you, but none too much;
|
||||
If you can fill the unforgiving minute
|
||||
With sixty seconds’ worth of distance run,
|
||||
Yours is the Earth and everything that’s in it,
|
||||
And—which is more—you’ll be a Man, my son!
|
||||
370
samples/US_Constitution.txt
Normal file
370
samples/US_Constitution.txt
Normal file
|
|
@ -0,0 +1,370 @@
|
|||
The United States Constitution
|
||||
|
||||
We the People of the United States, in Order to form a more perfect Union, establish Justice, insure domestic Tranquility, provide for the common defence, promote the general Welfare, and secure the Blessings of Liberty to ourselves and our Posterity, do ordain and establish this Constitution for the United States of America.
|
||||
The Constitutional Convention
|
||||
Article I
|
||||
Section 1: Congress
|
||||
|
||||
All legislative Powers herein granted shall be vested in a Congress of the United States, which shall consist of a Senate and House of Representatives.
|
||||
Section 2: The House of Representatives
|
||||
|
||||
The House of Representatives shall be composed of Members chosen every second Year by the People of the several States, and the Electors in each State shall have the Qualifications requisite for Electors of the most numerous Branch of the State Legislature.
|
||||
|
||||
No Person shall be a Representative who shall not have attained to the Age of twenty five Years, and been seven Years a Citizen of the United States, and who shall not, when elected, be an Inhabitant of that State in which he shall be chosen.
|
||||
|
||||
Representatives and direct Taxes shall be apportioned among the several States which may be included within this Union, according to their respective Numbers, which shall be determined by adding to the whole Number of free Persons, including those bound to Service for a Term of Years, and excluding Indians not taxed, three fifths of all other Persons. The actual Enumeration shall be made within three Years after the first Meeting of the Congress of the United States, and within every subsequent Term of ten Years, in such Manner as they shall by Law direct.The Number of Representatives shall not exceed one for every thirty Thousand, but each State shall have at Least one Representative; and until such enumeration shall be made, the State of New Hampshire shall be entitled to chuse three, Massachusetts eight, Rhode-Island and Providence Plantations one, Connecticut five, New-York six, New Jersey four, Pennsylvania eight, Delaware one, Maryland six, Virginia ten, North Carolina five, South Carolina five, and Georgia three.
|
||||
|
||||
When vacancies happen in the Representation from any State, the Executive Authority thereof shall issue Writs of Election to fill such Vacancies.
|
||||
|
||||
The House of Representatives shall chuse their Speaker and other Officers;and shall have the sole Power of Impeachment.
|
||||
Section 3: The Senate
|
||||
|
||||
The Senate of the United States shall be composed of two Senators from each State, chosen by the Legislature thereof, for six Years; and each Senator shall have one Vote.
|
||||
|
||||
Immediately after they shall be assembled in Consequence of the first Election, they shall be divided as equally as may be into three Classes. The Seats of the Senators of the first Class shall be vacated at the Expiration of the second Year, of the second Class at the Expiration of the fourth Year, and of the third Class at the Expiration of the sixth Year, so that one third may be chosen every second Year; and if Vacancies happen by Resignation, or otherwise, during the Recess of the Legislature of any State, the Executive thereof may make temporary Appointments until the next Meeting of the Legislature, which shall then fill such Vacancies.
|
||||
|
||||
No Person shall be a Senator who shall not have attained to the Age of thirty Years, and been nine Years a Citizen of the United States, and who shall not, when elected, be an Inhabitant of that State for which he shall be chosen.
|
||||
|
||||
The Vice President of the United States shall be President of the Senate, but shall have no Vote, unless they be equally divided.
|
||||
|
||||
The Senate shall chuse their other Officers, and also a President pro tempore, in the Absence of the Vice President, or when he shall exercise the Office of President of the United States.
|
||||
|
||||
The Senate shall have the sole Power to try all Impeachments. When sitting for that Purpose, they shall be on Oath or Affirmation. When the President of the United States is tried, the Chief Justice shall preside: And no Person shall be convicted without the Concurrence of two thirds of the Members present.
|
||||
|
||||
Judgment in Cases of Impeachment shall not extend further than to removal from Office, and disqualification to hold and enjoy any Office of honor, Trust or Profit under the United States: but the Party convicted shall nevertheless be liable and subject to Indictment, Trial, Judgment and Punishment, according to Law.
|
||||
Section 4: Elections
|
||||
|
||||
The Times, Places and Manner of holding Elections for Senators and Representatives, shall be prescribed in each State by the Legislature thereof; but the Congress may at any time by Law make or alter such Regulations, except as to the Places of chusing Senators.
|
||||
|
||||
The Congress shall assemble at least once in every Year, and such Meeting shall be on the first Monday in December, unless they shall by Law appoint a different Day.
|
||||
Section 5: Powers and Duties of Congress
|
||||
|
||||
Each House shall be the Judge of the Elections, Returns and Qualifications of its own Members,and a Majority of each shall constitute a Quorum to do Business; but a smaller Number may adjourn from day to day, and may be authorized to compel the Attendance of absent Members, in such Manner, and under such Penalties as each House may provide.
|
||||
|
||||
Each House may determine the Rules of its Proceedings, punish its Members for disorderly Behaviour, and, with the Concurrence of two thirds, expel a Member.
|
||||
|
||||
Each House shall keep a Journal of its Proceedings, and from time to time publish the same, excepting such Parts as may in their Judgment require Secrecy; and the Yeas and Nays of the Members of either House on any question shall, at the Desire of one fifth of those Present, be entered on the Journal.
|
||||
|
||||
Neither House, during the Session of Congress, shall, without the Consent of the other, adjourn for more than three days, nor to any other Place than that in which the two Houses shall be sitting.
|
||||
Section 6: Rights and Disabilities of Members
|
||||
|
||||
The Senators and Representatives shall receive a Compensation for their Services, to be ascertained by Law, and paid out of the Treasury of the United States.They shall in all Cases, except Treason, Felony and Breach of the Peace, be privileged from Arrest during their Attendance at the Session of their respective Houses, and in going to and returning from the same; and for any Speech or Debate in either House, they shall not be questioned in any other Place.
|
||||
|
||||
No Senator or Representative shall, during the Time for which he was elected, be appointed to any civil Office under the Authority of the United States, which shall have been created, or the Emoluments whereof shall have been encreased during such time; and no Person holding any Office under the United States, shall be a Member of either House during his Continuance in Office.
|
||||
Section 7: Legislative Process
|
||||
|
||||
All Bills for raising Revenue shall originate in the House of Representatives; but the Senate may propose or concur with Amendments as on other Bills.
|
||||
|
||||
Every Bill which shall have passed the House of Representatives and the Senate, shall, before it become a Law, be presented to the President of the United States; If he approve he shall sign it, but if not he shall return it, with his Objections to that House in which it shall have originated, who shall enter the Objections at large on their Journal, and proceed to reconsider it. If after such Reconsideration two thirds of that House shall agree to pass the Bill, it shall be sent, together with the Objections, to the other House, by which it shall likewise be reconsidered, and if approved by two thirds of that House, it shall become a Law. But in all such Cases the Votes of both Houses shall be determined by yeas and Nays, and the Names of the Persons voting for and against the Bill shall be entered on the Journal of each House respectively. If any Bill shall not be returned by the President within ten Days (Sundays excepted) after it shall have been presented to him, the Same shall be a Law, in like Manner as if he had signed it, unless the Congress by their Adjournment prevent its Return, in which Case it shall not be a Law.
|
||||
|
||||
Every Order, Resolution, or Vote to which the Concurrence of the Senate and House of Representatives may be necessary (except on a question of Adjournment) shall be presented to the President of the United States; and before the Same shall take Effect, shall be approved by him, or being disapproved by him, shall be repassed by two thirds of the Senate and House of Representatives, according to the Rules and Limitations prescribed in the Case of a Bill.
|
||||
Section 8: Powers of Congress
|
||||
|
||||
The Congress shall have Power To lay and collect Taxes, Duties, Imposts and Excises, to pay the Debts and provide for the common Defence and general Welfare of the United States; but all Duties, Imposts and Excises shall be uniform throughout the United States;
|
||||
|
||||
To borrow Money on the credit of the United States;
|
||||
|
||||
To regulate Commerce with foreign Nations, and among the several States, and with the Indian Tribes;
|
||||
|
||||
To establish a uniform Rule of Naturalization, and uniform Laws on the subject of Bankruptcies throughout the United States;
|
||||
|
||||
To coin Money, regulate the Value thereof, and of foreign Coin, and fix the Standard of Weights and Measures;
|
||||
|
||||
To provide for the Punishment of counterfeiting the Securities and current Coin of the United States;
|
||||
|
||||
To establish Post Offices and post Roads;
|
||||
|
||||
To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries;
|
||||
|
||||
To constitute Tribunals inferior to the supreme Court;
|
||||
|
||||
To define and punish Piracies and Felonies committed on the high Seas, and Offences against the Law of Nations;
|
||||
|
||||
To declare War, grant Letters of Marque and Reprisal, and make Rules concerning Captures on Land and Water;
|
||||
|
||||
To raise and support Armies, but no Appropriation of Money to that Use shall be for a longer Term than two Years;
|
||||
|
||||
To provide and maintain a Navy;
|
||||
|
||||
To make Rules for the Government and Regulation of the land and naval Forces;
|
||||
|
||||
To provide for calling forth the Militia to execute the Laws of the Union, suppress Insurrections and repel Invasions;
|
||||
|
||||
To provide for organizing, arming, and disciplining, the Militia, and for governing such Part of them as may be employed in the Service of the United States, reserving to the States respectively, the Appointment of the Officers, and the Authority of training the Militia according to the discipline prescribed by Congress;
|
||||
|
||||
To exercise exclusive Legislation in all Cases whatsoever, over such District (not exceeding ten Miles square) as may, by Cession of particular States, and the Acceptance of Congress, become the Seat of the Government of the United States, and to exercise like Authority over all Places purchased by the Consent of the Legislature of the State in which the Same shall be, for the Erection of Forts, Magazines, Arsenals, dock-Yards and other needful Buildings;-And
|
||||
|
||||
To make all Laws which shall be necessary and proper for carrying into Execution the foregoing Powers, and all other Powers vested by this Constitution in the Government of the United States, or in any Department or Officer thereof.
|
||||
Section 9: Powers Denied Congress
|
||||
|
||||
The Migration or Importation of such Persons as any of the States now existing shall think proper to admit, shall not be prohibited by the Congress prior to the Year one thousand eight hundred and eight, but a Tax or duty may be imposed on such Importation, not exceeding ten dollars for each Person.
|
||||
|
||||
The Privilege of the Writ of Habeas Corpus shall not be suspended, unless when in Cases of Rebellion or Invasion the public Safety may require it.
|
||||
|
||||
No Bill of Attainder or ex post facto Law shall be passed.
|
||||
|
||||
No Capitation, or other direct, Tax shall be laid, unless in Proportion to the Census or enumeration herein before directed to be taken.
|
||||
|
||||
No Tax or Duty shall be laid on Articles exported from any State.
|
||||
|
||||
No Preference shall be given by any Regulation of Commerce or Revenue to the Ports of one State over those of another: nor shall Vessels bound to, or from, one State, be obliged to enter, clear, or pay Duties in another.
|
||||
|
||||
No Money shall be drawn from the Treasury, but in Consequence of Appropriations made by Law; and a regular Statement and Account of the Receipts and Expenditures of all public Money shall be published from time to time.
|
||||
|
||||
No Title of Nobility shall be granted by the United States: And no Person holding any Office of Profit or Trust under them, shall, without the Consent of the Congress, accept of any present, Emolument, Office, or Title, of any kind whatever, from any King, Prince, or foreign State.
|
||||
Section 10: Powers Denied to the States
|
||||
|
||||
No State shall enter into any Treaty, Alliance, or Confederation; grant Letters of Marque and Reprisal; coin Money; emit Bills of Credit; make any Thing but gold and silver Coin a Tender in Payment of Debts; pass any Bill of Attainder, ex post facto Law, or Law impairing the Obligation of Contracts, or grant any Title of Nobility.
|
||||
|
||||
No State shall, without the Consent of the Congress, lay any Imposts or Duties on Imports or Exports, except what may be absolutely necessary for executing it's inspection Laws: and the net Produce of all Duties and Imposts, laid by any State on Imports or Exports, shall be for the Use of the Treasury of the United States; and all such Laws shall be subject to the Revision and Controul of the Congress.
|
||||
|
||||
No State shall, without the Consent of Congress, lay any Duty of Tonnage, keep Troops, or Ships of War in time of Peace, enter into any Agreement or Compact with another State, or with a foreign Power, or engage in War, unless actually invaded, or in such imminent Danger as will not admit of delay.
|
||||
Article II
|
||||
Section 1
|
||||
|
||||
The executive Power shall be vested in a President of the United States of America.
|
||||
|
||||
He shall hold his Office during the Term of four Years, and, together with the Vice President, chosen for the same Term, be elected, as follows:
|
||||
|
||||
Each State shall appoint, in such Manner as the Legislature thereof may direct, a Number of Electors, equal to the whole Number of Senators and Representatives to which the State may be entitled in the Congress: but no Senator or Representative, or Person holding an Office of Trust or Profit under the United States, shall be appointed an Elector.
|
||||
|
||||
The Electors shall meet in their respective States, and vote by Ballot for two Persons, of whom one at least shall not be an Inhabitant of the same State with themselves. And they shall make a List of all the Persons voted for, and of the Number of Votes for each; which List they shall sign and certify, and transmit sealed to the Seat of the Government of the United States, directed to the President of the Senate. The President of the Senate shall, in the Presence of the Senate and House of Representatives, open all the Certificates, and the Votes shall then be counted. The Person having the greatest Number of Votes shall be the President, if such Number be a Majority of the whole Number of Electors appointed; and if there be more than one who have such Majority, and have an equal Number of Votes, then the House of Representatives shall immediately chuse by Ballot one of them for President; and if no Person have a Majority, then from the five highest on the List the said House shall in like Manner chuse the President. But in chusing the President, the Votes shall be taken by States, the Representation from each State having one Vote; A quorum for this Purpose shall consist of a Member or Members from two thirds of the States, and a Majority of all the States shall be necessary to a Choice. In every Case, after the Choice of the President, the Person having the greatest Number of Votes of the Electors shall be the Vice President. But if there should remain two or more who have equal Votes, the Senate shall chuse from them by Ballot the Vice President.
|
||||
|
||||
The Congress may determine the Time of chusing the Electors, and the Day on which they shall give their Votes; which Day shall be the same throughout the United States.
|
||||
|
||||
No Person except a natural born Citizen, or a Citizen of the United States, at the time of the Adoption of this Constitution, shall be eligible to the Office of President; neither shall any Person be eligible to that Office who shall not have attained to the Age of thirty five Years, and been fourteen Years a Resident within the United States.
|
||||
|
||||
In Case of the Removal of the President from Office, or of his Death, Resignation, or Inability to discharge the Powers and Duties of the said Office, the Same shall devolve on the Vice President, and the Congress may by Law provide for the Case of Removal, Death, Resignation or Inability, both of the President and Vice President, declaring what Officer shall then act as President, and such Officer shall act accordingly, until the Disability be removed, or a President shall be elected.
|
||||
|
||||
The President shall, at stated Times, receive for his Services, a Compensation, which shall neither be encreased nor diminished during the Period for which he shall have been elected, and he shall not receive within that Period any other Emolument from the United States, or any of them.
|
||||
|
||||
Before he enter on the Execution of his Office, he shall take the following Oath or Affirmation:--"I do solemnly swear (or affirm) that I will faithfully execute the Office of President of the United States, and will to the best of my Ability, preserve, protect and defend the Constitution of the United States."
|
||||
Section 2
|
||||
|
||||
The President shall be Commander in Chief of the Army and Navy of the United States, and of the Militia of the several States, when called into the actual Service of the United States; he may require the Opinion, in writing, of the principal Officer in each of the executive Departments, upon any Subject relating to the Duties of their respective Offices, and he shall have Power to grant Reprieves and Pardons for Offences against the United States, except in Cases of Impeachment.
|
||||
|
||||
He shall have Power, by and with the Advice and Consent of the Senate, to make Treaties, provided two thirds of the Senators present concur; and he shall nominate, and by and with the Advice and Consent of the Senate, shall appoint Ambassadors, other public Ministers and Consuls, Judges of the supreme Court, and all other Officers of the United States, whose Appointments are not herein otherwise provided for, and which shall be established by Law: but the Congress may by Law vest the Appointment of such inferior Officers, as they think proper, in the President alone, in the Courts of Law, or in the Heads of Departments.
|
||||
|
||||
The President shall have Power to fill up all Vacancies that may happen during the Recess of the Senate, by granting Commissions which shall expire at the End of their next Session.
|
||||
Section 3
|
||||
|
||||
He shall from time to time give to the Congress Information of the State of the Union, and recommend to their Consideration such Measures as he shall judge necessary and expedient; he may, on extraordinary Occasions, convene both Houses, or either of them, and in Case of Disagreement between them, with Respect to the Time of Adjournment, he may adjourn them to such Time as he shall think proper; he shall receive Ambassadors and other public Ministers; he shall take Care that the Laws be faithfully executed, and shall Commission all the Officers of the United States.
|
||||
Section 4
|
||||
|
||||
The President, Vice President and all civil Officers of the United States, shall be removed from Office on Impeachment for, and Conviction of, Treason, Bribery, or other high Crimes and Misdemeanors.
|
||||
Article III
|
||||
Section 1
|
||||
|
||||
The judicial Power of the United States, shall be vested in one supreme Court, and in such inferior Courts as the Congress may from time to time ordain and establish. The Judges, both of the supreme and inferior Courts, shall hold their Offices during good Behaviour, and shall, at stated Times, receive for their Services, a Compensation, which shall not be diminished during their Continuance in Office.
|
||||
Section 2
|
||||
|
||||
The judicial Power shall extend to all Cases, in Law and Equity, arising under this Constitution, the Laws of the United States, and Treaties made, or which shall be made, under their Authority;--to all Cases affecting Ambassadors, other public Ministers and Consuls;--to all Cases of admiralty and maritime Jurisdiction;--to Controversies to which the United States shall be a Party;--to Controversies between two or more States;--between a State and Citizens of another State;--between Citizens of different States;--between Citizens of the same State claiming Lands under Grants of different States, and between a State, or the Citizens thereof, and foreign States, Citizens or Subjects.
|
||||
|
||||
In all Cases affecting Ambassadors, other public Ministers and Consuls, and those in which a State shall be Party, the supreme Court shall have original Jurisdiction. In all the other Cases before mentioned, the supreme Court shall have appellate Jurisdiction, both as to Law and Fact, with such Exceptions, and under such Regulations as the Congress shall make.
|
||||
|
||||
The Trial of all Crimes, except in Cases of Impeachment; shall be by Jury; and such Trial shall be held in the State where the said Crimes shall have been committed; but when not committed within any State, the Trial shall be at such Place or Places as the Congress may by Law have directed.
|
||||
Section 3
|
||||
|
||||
Treason against the United States, shall consist only in levying War against them, or in adhering to their Enemies, giving them Aid and Comfort. No Person shall be convicted of Treason unless on the Testimony of two Witnesses to the same overt Act, or on Confession in open Court.
|
||||
|
||||
The Congress shall have Power to declare the Punishment of Treason, but no Attainder of Treason shall work Corruption of Blood, or Forfeiture except during the Life of the Person attainted.
|
||||
Article IV
|
||||
Section 1
|
||||
|
||||
Full Faith and Credit shall be given in each State to the public Acts, Records, and judicial Proceedings of every other State. And the Congress may by general Laws prescribe the Manner in which such Acts, Records and Proceedings shall be proved, and the Effect thereof.
|
||||
Section 2
|
||||
|
||||
The Citizens of each State shall be entitled to all Privileges and Immunities of Citizens in the several States.
|
||||
|
||||
A Person charged in any State with Treason, Felony, or other Crime, who shall flee from Justice, and be found in another State, shall on Demand of the executive Authority of the State from which he fled, be delivered up, to be removed to the State having Jurisdiction of the Crime.
|
||||
|
||||
No Person held to Service or Labour in one State, under the Laws thereof, escaping into another, shall, in Consequence of any Law or Regulation therein, be discharged from such Service or Labour, but shall be delivered up on Claim of the Party to whom such Service or Labour may be due.
|
||||
Section 3
|
||||
|
||||
New States may be admitted by the Congress into this Union; but no new State shall be formed or erected within the Jurisdiction of any other State; nor any State be formed by the Junction of two or more States, or Parts of States, without the Consent of the Legislatures of the States concerned as well as of the Congress.
|
||||
|
||||
The Congress shall have Power to dispose of and make all needful Rules and Regulations respecting the Territory or other Property belonging to the United States; and nothing in this Constitution shall be so construed as to Prejudice any Claims of the United States, or of any particular State.
|
||||
Section 4
|
||||
|
||||
The United States shall guarantee to every State in this Union a Republican Form of Government, and shall protect each of them against Invasion; and on Application of the Legislature, or of the Executive (when the Legislature cannot be convened) against domestic Violence.
|
||||
Article V
|
||||
|
||||
The Congress, whenever two thirds of both Houses shall deem it necessary, shall propose Amendments to this Constitution, or, on the Application of the Legislatures of two thirds of the several States, shall call a Convention for proposing Amendments, which, in either Case, shall be valid to all Intents and Purposes, as Part of this Constitution, when ratified by the Legislatures of three fourths of the several States, or by Conventions in three fourths thereof, as the one or the other Mode of Ratification may be proposed by the Congress; Provided that no Amendment which may be made prior to the Year One thousand eight hundred and eight shall in any Manner affect the first and fourth Clauses in the Ninth Section of the first Article; and that no State, without its Consent, shall be deprived of its equal Suffrage in the Senate.
|
||||
Article VI
|
||||
|
||||
All Debts contracted and Engagements entered into, before the Adoption of this Constitution, shall be as valid against the United States under this Constitution, as under the Confederation.
|
||||
|
||||
This Constitution, and the Laws of the United States which shall be made in Pursuance thereof; and all Treaties made, or which shall be made, under the Authority of the United States, shall be the supreme Law of the Land; and the Judges in every State shall be bound thereby, any Thing in the Constitution or Laws of any State to the Contrary notwithstanding.
|
||||
|
||||
The Senators and Representatives before mentioned, and the Members of the several State Legislatures, and all executive and judicial Officers, both of the United States and of the several States, shall be bound by Oath or Affirmation, to support this Constitution; but no religious Test shall ever be required as a Qualification to any Office or public Trust under the United States.
|
||||
Article VII
|
||||
|
||||
The Ratification of the Conventions of nine States, shall be sufficient for the Establishment of this Constitution between the States so ratifying the Same.
|
||||
First Amendment
|
||||
|
||||
Congress shall make no law respecting an establishment of religion, or prohibiting the free exercise thereof; or abridging the freedom of speech, or of the press; or the right of the people peaceably to assemble, and to petition the Government for a redress of grievances.
|
||||
Second Amendment
|
||||
|
||||
A well regulated Militia, being necessary to the security of a free State, the right of the people to keep and bear Arms, shall not be infringed.
|
||||
Third Amendment
|
||||
|
||||
No Soldier shall, in time of peace be quartered in any house, without the consent of the Owner, nor in time of war, but in a manner to be prescribed by law.
|
||||
Fourth Amendment
|
||||
|
||||
The right of the people to be secure in their persons, houses, papers, and effects, against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probable cause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons or things to be seized.
|
||||
Fifth Amendment
|
||||
|
||||
No person shall be held to answer for a capital, or otherwise infamous crime, unless on a presentment or indictment of a Grand Jury, except in cases arising in the land or naval forces, or in the Militia, when in actual service in time of War or public danger; nor shall any person be subject for the same offence to be twice put in jeopardy of life or limb; nor shall be compelled in any criminal case to be a witness against himself, nor be deprived of life, liberty, or property, without due process of law; nor shall private property be taken for public use, without just compensation.
|
||||
Sixth Amendment
|
||||
|
||||
In all criminal prosecutions, the accused shall enjoy the right to a speedy and public trial, by an impartial jury of the State and district wherein the crime shall have been committed, which district shall have been previously ascertained by law, and to be informed of the nature and cause of the accusation; to be confronted with the witnesses against him; to have compulsory process for obtaining witnesses in his favor, and to have the Assistance of Counsel for his defence.
|
||||
Seventh Amendment
|
||||
|
||||
In Suits at common law, where the value in controversy shall exceed twenty dollars, the right of trial by jury shall be preserved, and no fact tried by a jury, shall be otherwise re-examined in any Court of the United States, than according to the rules of the common law.
|
||||
Eighth Amendment
|
||||
|
||||
Excessive bail shall not be required, nor excessive fines imposed, nor cruel and unusual punishments inflicted.
|
||||
Ninth Amendment
|
||||
|
||||
The enumeration in the Constitution, of certain rights, shall not be construed to deny or disparage others retained by the people.
|
||||
10th Amendment
|
||||
|
||||
The powers not delegated to the United States by the Constitution, nor prohibited by it to the States, are reserved to the States respectively, or to the people.
|
||||
11th Amendment
|
||||
|
||||
The Judicial power of the United States shall not be construed to extend to any suit in law or equity, commenced or prosecuted against one of the United States by Citizens of another State, or by Citizens or Subjects of any Foreign State.
|
||||
12th Amendment
|
||||
|
||||
The Electors shall meet in their respective states and vote by ballot for President and Vice-President, one of whom, at least, shall not be an inhabitant of the same state with themselves; they shall name in their ballots the person voted for as President, and in distinct ballots the person voted for as Vice-President, and they shall make distinct lists of all persons voted for as President, and of all persons voted for as Vice-President, and of the number of votes for each, which lists they shall sign and certify, and transmit sealed to the seat of the government of the United States, directed to the President of the Senate; -- the President of the Senate shall, in the presence of the Senate and House of Representatives, open all the certificates and the votes shall then be counted; -- The person having the greatest number of votes for President, shall be the President, if such number be a majority of the whole number of Electors appointed; and if no person have such majority, then from the persons having the highest numbers not exceeding three on the list of those voted for as President, the House of Representatives shall choose immediately, by ballot, the President. But in choosing the President, the votes shall be taken by states, the representation from each state having one vote; a quorum for this purpose shall consist of a member or members from two-thirds of the states, and a majority of all the states shall be necessary to a choice. And if the House of Representatives shall not choose a President whenever the right of choice shall devolve upon them, before the fourth day of March next following, then the Vice-President shall act as President, as in the case of the death or other constitutional disability of the President.-- The person having the greatest number of votes as Vice-President, shall be the Vice-President, if such number be a majority of the whole number of Electors appointed, and if no person have a majority, then from the two highest numbers on the list, the Senate shall choose the Vice-President; a quorum for the purpose shall consist of two-thirds of the whole number of Senators, and a majority of the whole number shall be necessary to a choice. But no person constitutionally ineligible to the office of President shall be eligible to that of Vice-President of the United States.
|
||||
13th Amendment
|
||||
Section 1
|
||||
|
||||
Neither slavery nor involuntary servitude, except as a punishment for crime whereof the party shall have been duly convicted, shall exist within the United States, or any place subject to their jurisdiction.
|
||||
Section 2
|
||||
|
||||
Congress shall have power to enforce this article by appropriate legislation.
|
||||
14th Amendment
|
||||
Section 1
|
||||
|
||||
All persons born or naturalized in the United States, and subject to the jurisdiction thereof, are citizens of the United States and of the State wherein they reside. No State shall make or enforce any law which shall abridge the privileges or immunities of citizens of the United States; nor shall any State deprive any person of life, liberty, or property, without due process of law; nor deny to any person within its jurisdiction the equal protection of the laws.
|
||||
Section 2
|
||||
|
||||
Representatives shall be apportioned among the several States according to their respective numbers, counting the whole number of persons in each State, excluding Indians not taxed. But when the right to vote at any election for the choice of electors for President and Vice-President of the United States, Representatives in Congress, the Executive and Judicial officers of a State, or the members of the Legislature thereof, is denied to any of the male inhabitants of such State, being twenty-one years of age, and citizens of the United States, or in any way abridged, except for participation in rebellion, or other crime, the basis of representation therein shall be reduced in the proportion which the number of such male citizens shall bear to the whole number of male citizens twenty-one years of age in such State.
|
||||
Section 3
|
||||
|
||||
No person shall be a Senator or Representative in Congress, or elector of President and Vice-President, or hold any office, civil or military, under the United States, or under any State, who, having previously taken an oath, as a member of Congress, or as an officer of the United States, or as a member of any State legislature, or as an executive or judicial officer of any State, to support the Constitution of the United States, shall have engaged in insurrection or rebellion against the same, or given aid or comfort to the enemies thereof. But Congress may by a vote of two-thirds of each House, remove such disability.
|
||||
Section 4
|
||||
|
||||
The validity of the public debt of the United States, authorized by law, including debts incurred for payment of pensions and bounties for services in suppressing insurrection or rebellion, shall not be questioned. But neither the United States nor any State shall assume or pay any debt or obligation incurred in aid of insurrection or rebellion against the United States, or any claim for the loss or emancipation of any slave; but all such debts, obligations and claims shall be held illegal and void.
|
||||
Section 5
|
||||
|
||||
The Congress shall have power to enforce, by appropriate legislation, the provisions of this article.
|
||||
15th Amendment
|
||||
Section 1
|
||||
|
||||
The right of citizens of the United States to vote shall not be denied or abridged by the United States or by any State on account of race, color, or previous condition of servitude.
|
||||
Section 2
|
||||
|
||||
The Congress shall have power to enforce this article by appropriate legislation.
|
||||
16th Amendment
|
||||
|
||||
The Congress shall have power to lay and collect taxes on incomes, from whatever source derived, without apportionment among the several States, and without regard to any census or enumeration.
|
||||
17th Amendment
|
||||
|
||||
The Senate of the United States shall be composed of two Senators from each State, elected by the people thereof, for six years; and each Senator shall have one vote. The electors in each State shall have the qualifications requisite for electors of the most numerous branch of the State legislatures.
|
||||
|
||||
When vacancies happen in the representation of any State in the Senate, the executive authority of such State shall issue writs of election to fill such vacancies: Provided, That the legislature of any State may empower the executive thereof to make temporary appointments until the people fill the vacancies by election as the legislature may direct.
|
||||
|
||||
This amendment shall not be so construed as to affect the election or term of any Senator chosen before it becomes valid as part of the Constitution.
|
||||
18th Amendment
|
||||
Section 1
|
||||
|
||||
After one year from the ratification of this article the manufacture, sale, or transportation of intoxicating liquors within, the importation thereof into, or the exportation thereof from the United States and all territory subject to the jurisdiction thereof for beverage purposes is hereby prohibited.
|
||||
Section 2
|
||||
|
||||
The Congress and the several States shall have concurrent power to enforce this article by appropriate legislation.
|
||||
Section 3
|
||||
|
||||
This article shall be inoperative unless it shall have been ratified as an amendment to the Constitution by the legislatures of the several States, as provided in the Constitution, within seven years from the date of the submission hereof to the States by the Congress.
|
||||
19th Amendment
|
||||
|
||||
The right of citizens of the United States to vote shall not be denied or abridged by the United States or by any State on account of sex.
|
||||
|
||||
Congress shall have power to enforce this article by appropriate legislation.
|
||||
20th Amendment
|
||||
Section 1
|
||||
|
||||
The terms of the President and the Vice President shall end at noon on the 20th day of January, and the terms of Senators and Representatives at noon on the 3d day of January, of the years in which such terms would have ended if this article had not been ratified; and the terms of their successors shall then begin.
|
||||
Section 2
|
||||
|
||||
The Congress shall assemble at least once in every year, and such meeting shall begin at noon on the 3d day of January, unless they shall by law appoint a different day.
|
||||
Section 3
|
||||
|
||||
If, at the time fixed for the beginning of the term of the President, the President elect shall have died, the Vice President elect shall become President. If a President shall not have been chosen before the time fixed for the beginning of his term, or if the President elect shall have failed to qualify, then the Vice President elect shall act as President until a President shall have qualified; and the Congress may by law provide for the case wherein neither a President elect nor a Vice President elect shall have qualified, declaring who shall then act as President, or the manner in which one who is to act shall be selected, and such person shall act accordingly until a President or Vice President shall have qualified.
|
||||
Section 4
|
||||
|
||||
The Congress may by law provide for the case of the death of any of the persons from whom the House of Representatives may choose a President whenever the right of choice shall have devolved upon them, and for the case of the death of any of the persons from whom the Senate may choose a Vice President whenever the right of choice shall have devolved upon them.
|
||||
Section 5
|
||||
|
||||
Sections 1 and 2 shall take effect on the 15th day of October following the ratification of this article.
|
||||
Section 6
|
||||
|
||||
This article shall be inoperative unless it shall have been ratified as an amendment to the Constitution by the legislatures of three-fourths of the several States within seven years from the date of its submission.
|
||||
21st Amendment
|
||||
Section 1
|
||||
|
||||
The eighteenth article of amendment to the Constitution of the United States is hereby repealed.
|
||||
Section 2
|
||||
|
||||
The transportation or importation into any State, Territory, or possession of the United States for delivery or use therein of intoxicating liquors, in violation of the laws thereof, is hereby prohibited.
|
||||
Section 3
|
||||
|
||||
This article shall be inoperative unless it shall have been ratified as an amendment to the Constitution by conventions in the several States, as provided in the Constitution, within seven years from the date of the submission hereof to the States by the Congress.
|
||||
22nd Amendment
|
||||
Section 1
|
||||
|
||||
No person shall be elected to the office of the President more than twice, and no person who has held the office of President, or acted as President, for more than two years of a term to which some other person was elected President shall be elected to the office of the President more than once. But this Article shall not apply to any person holding the office of President when this Article was proposed by the Congress, and shall not prevent any person who may be holding the office of President, or acting as President, during the term within which this Article becomes operative from holding the office of President or acting as President during the remainder of such term.
|
||||
Section 2
|
||||
|
||||
This article shall be inoperative unless it shall have been ratified as an amendment to the Constitution by the legislatures of three-fourths of the several States within seven years from the date of its submission to the States by the Congress.
|
||||
23rd Amendment
|
||||
Section 1
|
||||
|
||||
The District constituting the seat of Government of the United States shall appoint in such manner as Congress may direct:
|
||||
|
||||
A number of electors of President and Vice President equal to the whole number of Senators and Representatives in Congress to which the District would be entitled if it were a State, but in no event more than the least populous State; they shall be in addition to those appointed by the States, but they shall be considered, for the purposes of the election of President and Vice President, to be electors appointed by a State; and they shall meet in the District and perform such duties as provided by the twelfth article of amendment.
|
||||
Section 2
|
||||
|
||||
The Congress shall have power to enforce this article by appropriate legislation.
|
||||
24th Amendment
|
||||
Section 1
|
||||
|
||||
The right of citizens of the United States to vote in any primary or other election for President or Vice President, for electors for President or Vice President, or for Senator or Representative in Congress, shall not be denied or abridged by the United States or any State by reason of failure to pay poll tax or other tax.
|
||||
Section 2
|
||||
|
||||
The Congress shall have power to enforce this article by appropriate legislation.
|
||||
25th Amendment
|
||||
Section 1
|
||||
|
||||
In case of the removal of the President from office or of his death or resignation, the Vice President shall become President.
|
||||
Section 2
|
||||
|
||||
Whenever there is a vacancy in the office of the Vice President, the President shall nominate a Vice President who shall take office upon confirmation by a majority vote of both Houses of Congress.
|
||||
Section 3
|
||||
|
||||
Whenever the President transmits to the President pro tempore of the Senate and the Speaker of the House of Representatives his written declaration that he is unable to discharge the powers and duties of his office, and until he transmits to them a written declaration to the contrary, such powers and duties shall be discharged by the Vice President as Acting President.
|
||||
Section 4
|
||||
|
||||
Whenever the Vice President and a majority of either the principal officers of the executive departments or of such other body as Congress may by law provide, transmit to the President pro tempore of the Senate and the Speaker of the House of Representatives their written declaration that the President is unable to discharge the powers and duties of his office, the Vice President shall immediately assume the powers and duties of the office as Acting President.
|
||||
|
||||
Thereafter, when the President transmits to the President pro tempore of the Senate and the Speaker of the House of Representatives his written declaration that no inability exists, he shall resume the powers and duties of his office unless the Vice President and a majority of either the principal officers of the executive department or of such other body as Congress may by law provide, transmit within four days to the President pro tempore of the Senate and the Speaker of the House of Representatives their written declaration that the President is unable to discharge the powers and duties of his office. Thereupon Congress shall decide the issue, assembling within forty-eight hours for that purpose if not in session. If the Congress, within twenty-one days after receipt of the latter written declaration, or, if Congress is not in session, within twenty-one days after Congress is required to assemble, determines by two-thirds vote of both Houses that the President is unable to discharge the powers and duties of his office, the Vice President shall continue to discharge the same as Acting President; otherwise, the President shall resume the powers and duties of his office.
|
||||
26th Amendment
|
||||
Section 1
|
||||
|
||||
The right of citizens of the United States, who are eighteen years of age or older, to vote shall not be denied or abridged by the United States or by any State on account of age.
|
||||
Section 2
|
||||
|
||||
The Congress shall have power to enforce this article by appropriate legislation.
|
||||
27th Amendment
|
||||
|
||||
No law, varying the compensation for the services of the Senators and Representatives, shall take effect, until an election of Representatives shall have intervened.
|
||||
508
scripts/analyze_reticulum_file_transfer_20260516_1130.pl
Normal file
508
scripts/analyze_reticulum_file_transfer_20260516_1130.pl
Normal file
|
|
@ -0,0 +1,508 @@
|
|||
#!/usr/bin/env perl
|
||||
# ./analyze_reticulum_file_transfer_20260516_1130.pl 20250516_1115_zerodev1.txt 20250516_1115_zerodev2.txt
|
||||
# ./analyze_reticulum_file_transfer_20260516_1130.pl --detail 20250516_1115_zerodev1.txt 20250516_1115_zerodev2.txt
|
||||
# ./analyze_reticulum_file_transfer_20260516_1130.pl --csv chunks_$(date +%Y%m%d_%H%M).csv 20250516_1115_zerodev1.txt 20250516_1115_zerodev2.txt
|
||||
# chmod 755 analyze_reticulum_file_transfer_20260516_1130.pl
|
||||
# 2026-05-16 ChatGPT
|
||||
# $Header$
|
||||
# $HeadURL$
|
||||
|
||||
use strict;
|
||||
use warnings;
|
||||
use Getopt::Long qw(GetOptions);
|
||||
use POSIX qw(strftime);
|
||||
use Time::Local qw(timegm);
|
||||
use List::Util qw(min max sum);
|
||||
|
||||
my $detail = 0;
|
||||
my $csv_file = '';
|
||||
my $show_provenance = 0;
|
||||
|
||||
GetOptions(
|
||||
'detail!' => \$detail,
|
||||
'csv=s' => \$csv_file,
|
||||
'provenance!' => \$show_provenance,
|
||||
) or die usage();
|
||||
|
||||
@ARGV >= 1 or die usage();
|
||||
|
||||
my %mon = (
|
||||
Jan => 0, Feb => 1, Mar => 2, Apr => 3, May => 4, Jun => 5,
|
||||
Jul => 6, Aug => 7, Sep => 8, Oct => 9, Nov => 10, Dec => 11,
|
||||
);
|
||||
|
||||
my %tz_offset = (
|
||||
UTC => 0,
|
||||
GMT => 0,
|
||||
PST => -8 * 3600,
|
||||
PDT => -7 * 3600,
|
||||
);
|
||||
|
||||
my @chunks;
|
||||
my @hellos;
|
||||
my @log_reports;
|
||||
my %sent_declared;
|
||||
|
||||
for my $file (@ARGV) {
|
||||
my $report = parse_file($file);
|
||||
push @log_reports, $report;
|
||||
|
||||
push @chunks, @{ $report->{chunks} };
|
||||
push @hellos, @{ $report->{hellos} };
|
||||
|
||||
for my $k (keys %{ $report->{sent_declared} }) {
|
||||
$sent_declared{$k} = $report->{sent_declared}{$k};
|
||||
}
|
||||
}
|
||||
|
||||
@chunks or die "No RX file_chunk records with send_epoch= were found.\n";
|
||||
|
||||
print "Reticulum BLE file transfer analysis\n";
|
||||
print "Generated: ", strftime('%Y-%m-%d %H:%M:%S %Z', localtime), "\n";
|
||||
print "Input files:\n";
|
||||
print " $_\n" for @ARGV;
|
||||
print "\n";
|
||||
|
||||
print "Log provenance summary:\n";
|
||||
for my $r (@log_reports) {
|
||||
printf " %-28s receiver=%-10s date='%s' command_lines=%d post_marker_lines=%d\n",
|
||||
basename($r->{file}), $r->{receiver}, ($r->{date_line} // 'UNKNOWN'),
|
||||
scalar(@{ $r->{commands} }), $r->{post_marker_lines};
|
||||
}
|
||||
print "\n";
|
||||
|
||||
if ($show_provenance) {
|
||||
print "Captured command clumps:\n";
|
||||
for my $r (@log_reports) {
|
||||
print "--- ", $r->{file}, " ---\n";
|
||||
print $_ for @{ $r->{commands} };
|
||||
print "\n";
|
||||
}
|
||||
}
|
||||
|
||||
print "Chrony clock notes from logs:\n";
|
||||
for my $r (@log_reports) {
|
||||
printf " %s\n", basename($r->{file});
|
||||
if (@{ $r->{chrony_system_time} }) {
|
||||
for my $line (@{ $r->{chrony_system_time} }) {
|
||||
print " $line\n";
|
||||
}
|
||||
}
|
||||
else {
|
||||
print " no 'System time' lines found\n";
|
||||
}
|
||||
}
|
||||
print "\n";
|
||||
|
||||
print "Declared outbound sends observed in logs:\n";
|
||||
if (%sent_declared) {
|
||||
for my $k (sort keys %sent_declared) {
|
||||
my $s = $sent_declared{$k};
|
||||
printf " sender=%-10s file=%-24s chunks=%4d bytes=%7d chunk_data_bytes=%s\n",
|
||||
$s->{sender}, $s->{message_file}, $s->{chunks}, $s->{bytes}, ($s->{chunk_data_bytes} // 'n/a');
|
||||
}
|
||||
}
|
||||
else {
|
||||
print " none found\n";
|
||||
}
|
||||
print "\n";
|
||||
|
||||
my %by_dir;
|
||||
for my $c (@chunks) {
|
||||
push @{ $by_dir{ $c->{sender} . '->' . $c->{receiver} } }, $c;
|
||||
}
|
||||
|
||||
for my $dir (sort keys %by_dir) {
|
||||
my @r = sort { $a->{chunk_no} <=> $b->{chunk_no} } @{ $by_dir{$dir} };
|
||||
my @lat = map { $_->{latency} } @r;
|
||||
my @recv_gap = consecutive_deltas(map { $_->{recv_epoch} } @r);
|
||||
my @send_gap = consecutive_deltas(map { $_->{send_epoch} } @r);
|
||||
|
||||
my $n = scalar @r;
|
||||
my $first = $r[0];
|
||||
my $last = $r[-1];
|
||||
my $expected = max(map { $_->{chunk_total} } @r);
|
||||
my $bytes = sum(map { $_->{bytes} } @r);
|
||||
my $missing = missing_ranges([ map { $_->{chunk_no} } @r ], $expected);
|
||||
my $dupes = duplicate_list(map { $_->{chunk_no} } @r);
|
||||
my $recv_span = $last->{recv_epoch} - $first->{recv_epoch};
|
||||
my $send_span = $last->{send_epoch} - $first->{send_epoch};
|
||||
my $recv_payload_rate = $recv_span > 0 ? $bytes / $recv_span : 0;
|
||||
my $send_payload_rate = $send_span > 0 ? $bytes / $send_span : 0;
|
||||
|
||||
printf "Direction: %s\n", $dir;
|
||||
printf " file : %s\n", $first->{message_file};
|
||||
printf " chunks received : %d of %d\n", $n, $expected;
|
||||
printf " completeness : %.2f%%\n", 100.0 * $n / $expected if $expected;
|
||||
printf " missing chunks : %s\n", @$missing ? join(',', @$missing) : 'none';
|
||||
printf " duplicate chunks : %s\n", @$dupes ? join(',', @$dupes) : 'none';
|
||||
printf " payload bytes RX : %d\n", $bytes;
|
||||
printf " first chunk RX : %s\n", $first->{recv_clock};
|
||||
printf " last chunk RX : %s\n", $last->{recv_clock};
|
||||
printf " receiver span : %.3f s\n", $recv_span;
|
||||
printf " sender span : %.3f s\n", $send_span;
|
||||
printf " payload rate RX span : %.1f B/s %.1f bit/s\n", $recv_payload_rate, $recv_payload_rate * 8.0;
|
||||
printf " payload rate TX span : %.1f B/s %.1f bit/s\n", $send_payload_rate, $send_payload_rate * 8.0;
|
||||
print_latency_block('one-way latency', @lat);
|
||||
print_gap_block('receiver inter-chunk gap', @recv_gap);
|
||||
print_gap_block('sender inter-chunk gap', @send_gap);
|
||||
print "\n";
|
||||
}
|
||||
|
||||
if (@hellos) {
|
||||
print "Hello/handshake RX records:\n";
|
||||
for my $h (sort { $a->{recv_epoch} <=> $b->{recv_epoch} } @hellos) {
|
||||
printf " %-10s -> %-10s recv=%s latency=%8.3f ms message='%s'\n",
|
||||
$h->{sender}, $h->{receiver}, $h->{recv_clock},
|
||||
$h->{latency} * 1000.0, $h->{message};
|
||||
}
|
||||
print "\n";
|
||||
}
|
||||
|
||||
if ($detail) {
|
||||
print "Per-chunk detail:\n";
|
||||
printf "%12s %-10s %-10s %8s %8s %8s %10s %10s %10s\n",
|
||||
'recv_time', 'sender', 'receiver', 'chunk', 'total', 'bytes', 'lat_ms', 'recv_gap', 'send_gap';
|
||||
|
||||
for my $dir (sort keys %by_dir) {
|
||||
my @r = sort { $a->{chunk_no} <=> $b->{chunk_no} } @{ $by_dir{$dir} };
|
||||
my ($prev_recv, $prev_send);
|
||||
for my $c (@r) {
|
||||
my $rg = defined $prev_recv ? $c->{recv_epoch} - $prev_recv : undef;
|
||||
my $sg = defined $prev_send ? $c->{send_epoch} - $prev_send : undef;
|
||||
printf "%12s %-10s %-10s %8d %8d %8d %10.3f %10s %10s\n",
|
||||
$c->{recv_clock}, $c->{sender}, $c->{receiver},
|
||||
$c->{chunk_no}, $c->{chunk_total}, $c->{bytes}, $c->{latency} * 1000.0,
|
||||
defined $rg ? sprintf('%.3f', $rg) : '',
|
||||
defined $sg ? sprintf('%.3f', $sg) : '';
|
||||
$prev_recv = $c->{recv_epoch};
|
||||
$prev_send = $c->{send_epoch};
|
||||
}
|
||||
}
|
||||
print "\n";
|
||||
}
|
||||
|
||||
if ($csv_file ne '') {
|
||||
open my $out, '>', $csv_file or die "Cannot write $csv_file: $!\n";
|
||||
print $out join(',', qw(receiver sender file chunk_no chunk_total bytes send_epoch recv_epoch latency_s latency_ms recv_time source_log)), "\n";
|
||||
for my $c (sort { $a->{recv_epoch} <=> $b->{recv_epoch} } @chunks) {
|
||||
print $out join(',',
|
||||
csvq($c->{receiver}), csvq($c->{sender}), csvq($c->{message_file}),
|
||||
$c->{chunk_no}, $c->{chunk_total}, $c->{bytes},
|
||||
sprintf('%.6f', $c->{send_epoch}), sprintf('%.6f', $c->{recv_epoch}),
|
||||
sprintf('%.6f', $c->{latency}), sprintf('%.3f', $c->{latency} * 1000.0),
|
||||
csvq($c->{recv_clock}), csvq($c->{source_log})
|
||||
), "\n";
|
||||
}
|
||||
close $out;
|
||||
print "CSV written: $csv_file\n\n";
|
||||
}
|
||||
|
||||
print "Caution: one-way latency assumes sender and receiver clocks are synchronized.\n";
|
||||
print "Your chronyc tracking output helps bound this error, but it is not a substitute for ACK/round-trip timing.\n";
|
||||
|
||||
sub parse_file {
|
||||
my ($file) = @_;
|
||||
open my $fh, '<', $file or die "Cannot open $file: $!\n";
|
||||
|
||||
my $receiver = receiver_from_filename($file);
|
||||
my @commands;
|
||||
my @chunks;
|
||||
my @hellos;
|
||||
my @chrony_system_time;
|
||||
my %sent_declared;
|
||||
my $pending_chunk_data_bytes;
|
||||
my $in_command_clump = 1;
|
||||
my $post_marker_lines = 0;
|
||||
my ($year, $month, $day, $tz, $date_line);
|
||||
my $current_record = '';
|
||||
my $current_recv_parts;
|
||||
my $current_line_no = 0;
|
||||
my $line_no = 0;
|
||||
|
||||
while (my $line = <$fh>) {
|
||||
++$line_no;
|
||||
if ($in_command_clump) {
|
||||
push @commands, $line;
|
||||
if ($line =~ /^#\s*End of command clump\s*$/) {
|
||||
$in_command_clump = 0;
|
||||
}
|
||||
next;
|
||||
}
|
||||
|
||||
++$post_marker_lines;
|
||||
chomp(my $line_chomp = $line);
|
||||
|
||||
if (!defined $date_line && $line_chomp =~ /^\S+\s+\S+\s+\d{1,2}\s+\d\d:\d\d:\d\d(?:\s+(?:AM|PM))?\s+\S+\s+\d{4}$/i) {
|
||||
$date_line = $line_chomp;
|
||||
($year, $month, $day, $tz) = parse_log_date($line_chomp);
|
||||
}
|
||||
|
||||
if ($line_chomp =~ /^System time\s+:/) {
|
||||
push @chrony_system_time, $line_chomp;
|
||||
}
|
||||
|
||||
if ($line_chomp =~ /^\[(\d\d):(\d\d):(\d\d)\.(\d{3})\]\s+Starting node\s+(\S+)/) {
|
||||
$receiver = $1 if 0; # placeholder to keep pattern variables local-looking
|
||||
}
|
||||
|
||||
if ($line_chomp =~ /^\[(\d\d):(\d\d):(\d\d)\.(\d{3})\]\s+Sending file\s+(\S+)\s+as\s+(\d+)\s+chunk\(s\),\s+(\d+)\s+bytes/) {
|
||||
next unless defined $date_line;
|
||||
my ($hh, $mi, $ss, $ms, $path, $chunks, $bytes) = ($1, $2, $3, $4, $5, $6, $7);
|
||||
my $key = $receiver . '|' . basename($path);
|
||||
$sent_declared{$key} = {
|
||||
sender => $receiver,
|
||||
path => $path,
|
||||
message_file => basename($path),
|
||||
chunks => $chunks + 0,
|
||||
bytes => $bytes + 0,
|
||||
time_epoch => epoch_from_local_parts($year, $month, $day, $hh, $mi, $ss, $ms, $tz),
|
||||
chunk_data_bytes => $pending_chunk_data_bytes,
|
||||
};
|
||||
}
|
||||
|
||||
if ($line_chomp =~ /Requested message chunk size\s+(\d+)\s+exceeds Reticulum link budget; using\s+(\d+)\s+data bytes per chunk/) {
|
||||
$pending_chunk_data_bytes = $2;
|
||||
for my $k (keys %sent_declared) {
|
||||
$sent_declared{$k}{chunk_data_bytes} = $2;
|
||||
}
|
||||
}
|
||||
|
||||
if ($line =~ /^\[(\d\d):(\d\d):(\d\d)\.(\d{3})\]\s+RX\s+/) {
|
||||
flush_record(\$current_record, $current_recv_parts, $current_line_no, \@chunks, \@hellos,
|
||||
$file, $receiver, $year, $month, $day, $tz) if $current_record ne '';
|
||||
$current_record = $line;
|
||||
$current_recv_parts = [$1, $2, $3, $4];
|
||||
$current_line_no = $line_no;
|
||||
if ($line =~ /send_epoch=/) {
|
||||
flush_record(\$current_record, $current_recv_parts, $current_line_no, \@chunks, \@hellos,
|
||||
$file, $receiver, $year, $month, $day, $tz);
|
||||
}
|
||||
}
|
||||
elsif ($current_record ne '') {
|
||||
$current_record .= $line;
|
||||
if ($line =~ /send_epoch=/) {
|
||||
flush_record(\$current_record, $current_recv_parts, $current_line_no, \@chunks, \@hellos,
|
||||
$file, $receiver, $year, $month, $day, $tz);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
flush_record(\$current_record, $current_recv_parts, $current_line_no, \@chunks, \@hellos,
|
||||
$file, $receiver, $year, $month, $day, $tz) if $current_record ne '';
|
||||
|
||||
close $fh;
|
||||
|
||||
return {
|
||||
file => $file,
|
||||
receiver => $receiver,
|
||||
commands => \@commands,
|
||||
date_line => $date_line,
|
||||
post_marker_lines => $post_marker_lines,
|
||||
chunks => \@chunks,
|
||||
hellos => \@hellos,
|
||||
chrony_system_time => \@chrony_system_time,
|
||||
sent_declared => \%sent_declared,
|
||||
};
|
||||
}
|
||||
|
||||
sub flush_record {
|
||||
my ($record_ref, $recv_parts, $line_no, $chunks_ref, $hellos_ref, $file, $receiver, $year, $month, $day, $tz) = @_;
|
||||
return if $$record_ref eq '';
|
||||
my $record = $$record_ref;
|
||||
$$record_ref = '';
|
||||
|
||||
return unless defined $year;
|
||||
return unless $record =~ /send_epoch=([0-9]+(?:\.[0-9]+)?)/s;
|
||||
my $send_epoch = $1 + 0.0;
|
||||
my ($hh, $mi, $ss, $ms) = @$recv_parts;
|
||||
my $recv_epoch = epoch_from_local_parts($year, $month, $day, $hh, $mi, $ss, $ms, $tz);
|
||||
my $latency = $recv_epoch - $send_epoch;
|
||||
my $recv_clock = sprintf('%02d:%02d:%02d.%03d', $hh, $mi, $ss, $ms);
|
||||
|
||||
if ($record =~ /file_chunk\s+(\d+)\/(\d+)\s+from\s+(\S+)\s+bytes=(\d+)\s+file=(\S+)/s) {
|
||||
push @$chunks_ref, {
|
||||
source_log => $file,
|
||||
source_line => $line_no,
|
||||
receiver => $receiver,
|
||||
sender => $3,
|
||||
chunk_no => $1 + 0,
|
||||
chunk_total => $2 + 0,
|
||||
bytes => $4 + 0,
|
||||
message_file => $5,
|
||||
send_epoch => $send_epoch,
|
||||
recv_epoch => $recv_epoch,
|
||||
latency => $latency,
|
||||
recv_clock => $recv_clock,
|
||||
};
|
||||
}
|
||||
elsif ($record =~ /RX\s+link=<[^>]+>:\s+(.+?)\s+from\s+(\S+)\s+send_epoch=/s) {
|
||||
my ($message, $sender) = ($1, $2);
|
||||
$message =~ s/\s+/ /g;
|
||||
push @$hellos_ref, {
|
||||
source_log => $file,
|
||||
source_line => $line_no,
|
||||
receiver => $receiver,
|
||||
sender => $sender,
|
||||
message => $message,
|
||||
send_epoch => $send_epoch,
|
||||
recv_epoch => $recv_epoch,
|
||||
latency => $latency,
|
||||
recv_clock => $recv_clock,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
sub print_latency_block {
|
||||
my ($label, @values) = @_;
|
||||
return unless @values;
|
||||
printf " %-21s min/median/mean/p95/max/stddev: %.3f / %.3f / %.3f / %.3f / %.3f / %.3f ms\n",
|
||||
$label,
|
||||
min(@values) * 1000.0,
|
||||
percentile(50, @values) * 1000.0,
|
||||
(sum(@values) / @values) * 1000.0,
|
||||
percentile(95, @values) * 1000.0,
|
||||
max(@values) * 1000.0,
|
||||
stddev(@values) * 1000.0;
|
||||
}
|
||||
|
||||
sub print_gap_block {
|
||||
my ($label, @values) = @_;
|
||||
return unless @values;
|
||||
printf " %-21s min/median/mean/p95/max/stddev: %.3f / %.3f / %.3f / %.3f / %.3f / %.3f ms\n",
|
||||
$label,
|
||||
min(@values) * 1000.0,
|
||||
percentile(50, @values) * 1000.0,
|
||||
(sum(@values) / @values) * 1000.0,
|
||||
percentile(95, @values) * 1000.0,
|
||||
max(@values) * 1000.0,
|
||||
stddev(@values) * 1000.0;
|
||||
}
|
||||
|
||||
sub parse_log_date {
|
||||
my ($line) = @_;
|
||||
|
||||
# Supports:
|
||||
# Sat May 16 11:12:57 PDT 2026
|
||||
# Sat May 16 11:12:58 AM PDT 2026
|
||||
if ($line =~ /^\S+\s+(\S+)\s+(\d{1,2})\s+(\d\d):(\d\d):(\d\d)(?:\s+(AM|PM))?\s+(\S+)\s+(\d{4})/i) {
|
||||
my ($mon_name, $day, $ampm, $tz, $year) = ($1, $2, $6, uc($7), $8);
|
||||
exists $mon{$mon_name} or die "Cannot parse month '$mon_name' in: $line\n";
|
||||
exists $tz_offset{$tz} or die "Unknown timezone '$tz' in: $line\nAdd it to %tz_offset.\n";
|
||||
return ($year, $mon{$mon_name}, $day, $tz);
|
||||
}
|
||||
|
||||
die "Cannot parse log date from: $line\n";
|
||||
}
|
||||
|
||||
sub epoch_from_local_parts {
|
||||
my ($year, $month, $day, $hh, $mi, $ss, $ms, $tz) = @_;
|
||||
my $epoch_as_if_utc = timegm($ss, $mi, $hh, $day, $month, $year - 1900);
|
||||
return $epoch_as_if_utc - $tz_offset{$tz} + ($ms / 1000.0);
|
||||
}
|
||||
|
||||
sub receiver_from_filename {
|
||||
my ($file) = @_;
|
||||
return $1 if $file =~ /(zerodev\d+)/;
|
||||
return basename($file);
|
||||
}
|
||||
|
||||
sub consecutive_deltas {
|
||||
my (@v) = @_;
|
||||
my @d;
|
||||
for (my $i = 1; $i < @v; ++$i) {
|
||||
push @d, $v[$i] - $v[$i - 1];
|
||||
}
|
||||
return @d;
|
||||
}
|
||||
|
||||
sub missing_ranges {
|
||||
my ($seen_ref, $expected) = @_;
|
||||
my %seen = map { $_ => 1 } @$seen_ref;
|
||||
my @missing = grep { !$seen{$_} } 1 .. $expected;
|
||||
return [] unless @missing;
|
||||
|
||||
my @ranges;
|
||||
my ($start, $prev) = ($missing[0], $missing[0]);
|
||||
for my $m (@missing[1 .. $#missing]) {
|
||||
if ($m == $prev + 1) {
|
||||
$prev = $m;
|
||||
}
|
||||
else {
|
||||
push @ranges, $start == $prev ? $start : "$start-$prev";
|
||||
($start, $prev) = ($m, $m);
|
||||
}
|
||||
}
|
||||
push @ranges, $start == $prev ? $start : "$start-$prev";
|
||||
return \@ranges;
|
||||
}
|
||||
|
||||
sub duplicate_list {
|
||||
my (@v) = @_;
|
||||
my %count;
|
||||
++$count{$_} for @v;
|
||||
return [ sort { $a <=> $b } grep { $count{$_} > 1 } keys %count ];
|
||||
}
|
||||
|
||||
sub percentile {
|
||||
my ($p, @values) = @_;
|
||||
@values = sort { $a <=> $b } @values;
|
||||
return undef unless @values;
|
||||
return $values[0] if @values == 1;
|
||||
|
||||
my $rank = ($p / 100.0) * (@values - 1);
|
||||
my $lo = int($rank);
|
||||
my $hi = $lo + 1;
|
||||
return $values[$lo] if $hi > $#values;
|
||||
|
||||
my $frac = $rank - $lo;
|
||||
return $values[$lo] + (($values[$hi] - $values[$lo]) * $frac);
|
||||
}
|
||||
|
||||
sub stddev {
|
||||
my (@values) = @_;
|
||||
return 0 if @values < 2;
|
||||
my $mean = sum(@values) / @values;
|
||||
my $ss = 0;
|
||||
$ss += ($_ - $mean) ** 2 for @values;
|
||||
return sqrt($ss / @values);
|
||||
}
|
||||
|
||||
sub format_hms_epoch {
|
||||
my ($epoch) = @_;
|
||||
my $whole = int($epoch);
|
||||
my $ms = int(($epoch - $whole) * 1000.0 + 0.5);
|
||||
if ($ms >= 1000) {
|
||||
++$whole;
|
||||
$ms -= 1000;
|
||||
}
|
||||
return strftime('%H:%M:%S', localtime($whole)) . sprintf('.%03d', $ms);
|
||||
}
|
||||
|
||||
sub basename {
|
||||
my ($path) = @_;
|
||||
$path =~ s{.*/}{};
|
||||
return $path;
|
||||
}
|
||||
|
||||
sub csvq {
|
||||
my ($v) = @_;
|
||||
$v = '' unless defined $v;
|
||||
$v =~ s/"/""/g;
|
||||
return '"' . $v . '"';
|
||||
}
|
||||
|
||||
sub usage {
|
||||
return <<'USAGE';
|
||||
Usage: analyze_reticulum_file_transfer_20260516_1130.pl [options] node1.log node2.log [more.log ...]
|
||||
|
||||
Options:
|
||||
--detail print one row per received file chunk
|
||||
--csv file.csv write per-chunk data to CSV
|
||||
--provenance print captured command clumps from each log
|
||||
|
||||
The script ignores the command provenance at the top of each captured terminal log
|
||||
until it sees '# End of command clump'. It then parses the date, chronyc notes,
|
||||
file send declarations, hello RX messages, and multiline RX file_chunk records.
|
||||
USAGE
|
||||
}
|
||||
62
scripts/reticulum_file_transfer_analysis_20260516_1130.txt
Normal file
62
scripts/reticulum_file_transfer_analysis_20260516_1130.txt
Normal file
|
|
@ -0,0 +1,62 @@
|
|||
Reticulum BLE file transfer analysis
|
||||
Generated: 2026-05-16 18:28:23 UTC
|
||||
Input files:
|
||||
/mnt/data/20250516_1115_zerodev1.txt
|
||||
/mnt/data/20250516_1115_zerodev2.txt
|
||||
|
||||
Log provenance summary:
|
||||
20250516_1115_zerodev1.txt receiver=zerodev1 date='Sat May 16 11:12:57 PDT 2026' command_lines=15 post_marker_lines=589
|
||||
20250516_1115_zerodev2.txt receiver=zerodev2 date='Sat May 16 11:12:58 AM PDT 2026' command_lines=14 post_marker_lines=584
|
||||
|
||||
Chrony clock notes from logs:
|
||||
20250516_1115_zerodev1.txt
|
||||
System time : 0.000000009 seconds fast of NTP time
|
||||
System time : 0.000000008 seconds fast of NTP time
|
||||
20250516_1115_zerodev2.txt
|
||||
System time : 0.000114634 seconds slow of NTP time
|
||||
System time : 0.000105892 seconds slow of NTP time
|
||||
|
||||
Declared outbound sends observed in logs:
|
||||
sender=zerodev1 file=US_Constitution.txt chunks= 140 bytes= 44225 chunk_data_bytes=316
|
||||
sender=zerodev2 file=US_Constitution.txt chunks= 148 bytes= 44225 chunk_data_bytes=n/a
|
||||
|
||||
Direction: zerodev1->zerodev2
|
||||
file : US_Constitution.txt
|
||||
chunks received : 140 of 140
|
||||
completeness : 100.00%
|
||||
missing chunks : none
|
||||
duplicate chunks : none
|
||||
payload bytes RX : 44225
|
||||
first chunk RX : 11:13:18.364
|
||||
last chunk RX : 11:13:55.318
|
||||
receiver span : 36.954 s
|
||||
sender span : 14.300 s
|
||||
payload rate RX span : 1196.8 B/s 9574.1 bit/s
|
||||
payload rate TX span : 3092.6 B/s 24741.1 bit/s
|
||||
one-way latency min/median/mean/p95/max/stddev: 244.725 / 12236.420 / 11924.860 / 21922.870 / 22898.637 / 6711.984 ms
|
||||
receiver inter-chunk gap min/median/mean/p95/max/stddev: 149.000 / 243.000 / 265.856 / 340.100 / 732.000 / 62.665 ms
|
||||
sender inter-chunk gap min/median/mean/p95/max/stddev: 101.926 / 102.941 / 102.878 / 104.257 / 107.490 / 0.874 ms
|
||||
|
||||
Direction: zerodev2->zerodev1
|
||||
file : US_Constitution.txt
|
||||
chunks received : 148 of 148
|
||||
completeness : 100.00%
|
||||
missing chunks : none
|
||||
duplicate chunks : none
|
||||
payload bytes RX : 44225
|
||||
first chunk RX : 11:13:18.267
|
||||
last chunk RX : 11:13:53.760
|
||||
receiver span : 35.493 s
|
||||
sender span : 15.482 s
|
||||
payload rate RX span : 1246.0 B/s 9968.2 bit/s
|
||||
payload rate TX span : 2856.6 B/s 22852.9 bit/s
|
||||
one-way latency min/median/mean/p95/max/stddev: 212.914 / 10820.779 / 10497.937 / 19392.748 / 20224.292 / 5938.272 ms
|
||||
receiver inter-chunk gap min/median/mean/p95/max/stddev: 145.000 / 243.000 / 241.449 / 291.700 / 729.000 / 57.537 ms
|
||||
sender inter-chunk gap min/median/mean/p95/max/stddev: 103.946 / 105.794 / 105.317 / 106.740 / 111.933 / 1.353 ms
|
||||
|
||||
Hello/handshake RX records:
|
||||
zerodev2 -> zerodev1 recv=11:13:18.032 latency= 163.721 ms message='hello'
|
||||
zerodev1 -> zerodev2 recv=11:13:18.034 latency= 103.598 ms message='hello back'
|
||||
|
||||
Caution: one-way latency assumes sender and receiver clocks are synchronized.
|
||||
Your chronyc tracking output helps bound this error, but it is not a substitute for ACK/round-trip timing.
|
||||
70
src/ble_reticulum/BLEFragmentationBackend.py
Normal file
70
src/ble_reticulum/BLEFragmentationBackend.py
Normal file
|
|
@ -0,0 +1,70 @@
|
|||
"""
|
||||
Selectable backend for BLE fragmentation protocol classes.
|
||||
|
||||
This module is intentionally small. It gives integration code one import location
|
||||
while preserving a Python fallback until the C++ backend is proven in live BLE
|
||||
use.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
|
||||
BACKEND_ENV_VAR = "BLE_RETICULUM_FRAGMENTATION_BACKEND"
|
||||
DEFAULT_BACKEND = "auto"
|
||||
|
||||
|
||||
def _requested_backend() -> str:
|
||||
value = os.environ.get(BACKEND_ENV_VAR, DEFAULT_BACKEND).strip().lower()
|
||||
if value not in ("auto", "cpp", "python"):
|
||||
raise ValueError(
|
||||
f"Invalid {BACKEND_ENV_VAR}={value!r}; expected 'auto', 'cpp', or 'python'"
|
||||
)
|
||||
return value
|
||||
|
||||
|
||||
def _load_python_backend():
|
||||
from .BLEFragmentation import BLEFragmenter, BLEReassembler, HDLCFramer
|
||||
|
||||
return "python", BLEFragmenter, BLEReassembler, HDLCFramer
|
||||
|
||||
|
||||
def _load_cpp_backend():
|
||||
# Future packaging should prefer a package-local extension module. The
|
||||
# top-level name is used by the current migration/protocol_core build.
|
||||
try:
|
||||
from ._ble_protocol_core import BLEFragmenter, BLEReassembler, HDLCFramer
|
||||
except ImportError as package_error:
|
||||
try:
|
||||
from ble_protocol_core_cpp import BLEFragmenter, BLEReassembler, HDLCFramer
|
||||
except ImportError as top_level_error:
|
||||
raise ImportError(
|
||||
"C++ BLE fragmentation backend is not available. Build or install "
|
||||
"the pybind11 extension, or set "
|
||||
f"{BACKEND_ENV_VAR}=python."
|
||||
) from top_level_error
|
||||
|
||||
return "cpp", BLEFragmenter, BLEReassembler, HDLCFramer
|
||||
|
||||
|
||||
_backend = _requested_backend()
|
||||
|
||||
if _backend == "python":
|
||||
BACKEND, BLEFragmenter, BLEReassembler, HDLCFramer = _load_python_backend()
|
||||
elif _backend == "cpp":
|
||||
BACKEND, BLEFragmenter, BLEReassembler, HDLCFramer = _load_cpp_backend()
|
||||
else:
|
||||
try:
|
||||
BACKEND, BLEFragmenter, BLEReassembler, HDLCFramer = _load_cpp_backend()
|
||||
except ImportError:
|
||||
BACKEND, BLEFragmenter, BLEReassembler, HDLCFramer = _load_python_backend()
|
||||
|
||||
|
||||
__all__ = [
|
||||
"BACKEND",
|
||||
"BACKEND_ENV_VAR",
|
||||
"BLEFragmenter",
|
||||
"BLEReassembler",
|
||||
"HDLCFramer",
|
||||
]
|
||||
Loading…
Add table
Add a link
Reference in a new issue