microReticulumTbeam/exercises/305_microReticulum_ble_file_transfer

Exercise 305: microReticulum BLE file transfer

This exercise builds on Exercise 304's equal-peer BLE transport. Both boards run the same dual-role BLE interface, form a Reticulum Link, and then send the selected text file across that Link at the same time.

The file transfer protocol is intentionally small and visible on the serial console:

FTB -> file begin, with file name, byte count, chunk count, and checksum
FTD -> numbered file data chunk
FTE -> file end, with verification metadata repeated

Each receiver checks byte count, chunk count, and FNV-1a checksum. After a sender completes a transfer, it rests for 10 seconds and then starts the same selected file again.

Sample Set

Exercise 305 uses the same payload files as the Pi Zero BLE Reticulum tests:

texts/If.txt                 195 bytes
texts/If_full.txt           1583 bytes
texts/US_Constitution.txt  44225 bytes

The selected file is compiled into the firmware. The transfer code does not care which file is selected; platformio.ini chooses the source text through custom_text_source, and scripts/embed_text.py generates SelectedText.h in the build directory before compilation.

Transfer Profiles

The transfer pressure is selected in platformio.ini with build flags:

-D FILE_TRANSFER_CHUNK_SIZE=32
-D FILE_TRANSFER_CHUNK_INTERVAL_MS=500

FILE_TRANSFER_CHUNK_SIZE is the number of text bytes placed in each application-level FTD message before microReticulum wraps and encrypts it as a Link packet. Larger chunks reduce the number of packets needed for a file, but each encrypted packet becomes larger. If it grows beyond what the ESP32 BLE transport can reliably carry under simultaneous two-way traffic, Reticulum may log Link decrypt/HMAC failures because ciphertext arrived damaged or incomplete.

FILE_TRANSFER_CHUNK_INTERVAL_MS is the delay between application-level file chunks. Smaller intervals increase throughput, but also increase BLE write/notify pressure. With both nodes transmitting at the same time, too short a cadence can overflow buffers or expose ordering/loss issues in the current ESP32 BLE transport.

The conservative bring-up profile uses:

32 byte chunks, 500 ms between chunks

The Pi-Zero-comparison profile uses:

300 byte chunks, 100 ms between chunks

That is the apples-to-apples starting point for the previous Zero-to-Zero tests. Those commands requested --message-chunk-size 900, but the Python sender intentionally applied an internal board/Link-budget cap before sending. The run17 report for the Constitution transfer shows effective chunk data around 300 to 316 bytes, not 900 bytes, with roughly 100 ms sender pacing.

VERIFY_FAIL means the file protocol received an incomplete or corrupted transfer. A Reticulum Link HMAC/decryption error means corruption happened earlier, before the file protocol could parse the packet.

Priority

See Exercise 304_microReticulum_ble_dual_role_ping_pong README.md for explanation of "deterministic tie-breaker" of the role of client and server based on the ESP32 MAC.

Environments

Conservative ESP32 bring-up environments:

tbeam_if
tbeam_if_full
tbeam_constitution

Pi-Zero-comparison environments:

tbeam_if_pi_zero_profile
tbeam_if_full_pi_zero_profile
tbeam_constitution_pi_zero_profile

Build Once, Upload Twice

Each selected text environment produces one firmware image. Build it once, then upload that same image to both boards.

Build the short If sample:

source /home/jlpoole/rnsenv/bin/activate
cd /usr/local/src/microreticulum/microReticulumTbeam
pio run -d exercises/305_microReticulum_ble_file_transfer -e tbeam_if

Build the Pi-Zero-profile Constitution sample:

source /home/jlpoole/rnsenv/bin/activate
cd /usr/local/src/microreticulum/microReticulumTbeam
pio run -d exercises/305_microReticulum_ble_file_transfer -e tbeam_constitution_pi_zero_profile

After the build succeeds, upload the same environment to both boards. These commands may be run one after the other:

pio run -d exercises/305_microReticulum_ble_file_transfer -e tbeam_if -t upload --upload-port /dev/ttytAMY
pio run -d exercises/305_microReticulum_ble_file_transfer -e tbeam_if -t upload --upload-port /dev/ttytBOB

Use the same -e value in upload commands that you used for the build.

For strict parallel uploads, use esptool.py directly against the already-built artifacts. This avoids two concurrent pio run processes touching the same .pio build directory:

cd /usr/local/src/microreticulum/microReticulumTbeam/exercises/305_microReticulum_ble_file_transfer
esptool.py --chip esp32s3 --port /dev/ttytAMY --baud 460800 write_flash -z \
  0x0000 .pio/build/tbeam_if/bootloader.bin \
  0x8000 .pio/build/tbeam_if/partitions.bin \
  0xe000 /home/jlpoole/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin \
  0x10000 .pio/build/tbeam_if/firmware.bin &
esptool.py --chip esp32s3 --port /dev/ttytBOB --baud 460800 write_flash -z \
  0x0000 .pio/build/tbeam_if/bootloader.bin \
  0x8000 .pio/build/tbeam_if/partitions.bin \
  0xe000 /home/jlpoole/.platformio/packages/framework-arduinoespressif32/tools/partitions/boot_app0.bin \
  0x10000 .pio/build/tbeam_if/firmware.bin &
wait

For another environment, replace each .pio/build/tbeam_if/ path with that environment's build directory.

Monitor:

pio device monitor -p /dev/ttytAMY -b 115200
pio device monitor -p /dev/ttytBOB -b 115200

Expected Output

Once the Link is active, both nodes start sending:

Selected file=If.txt bytes=195 chunk=32 interval_ms=500 repeat_rest_ms=10000
TX FILE BEGIN: round=1 file=If.txt bytes=195 chunks=7 crc=...
TX FILE DATA: round=1 seq=1/7 bytes=32 preview="If you can keep your head..."
TX FILE END: round=1 file=If.txt bytes=195 chunks=7 crc=... next_round_in_ms=10000

The receiver verifies the transfer:

RX FILE BEGIN: from=Node-... file=If.txt bytes=195 chunks=7 crc=...
RX FILE DATA: from=Node-... seq=1/7 bytes=32 preview="If you can keep your head..."
RX FILE END: from=Node-... file=If.txt received=195/195 chunks=7/7 crc=... status=OK

Ten seconds after TX FILE END, the same selected file starts again. This rest interval is measured after transfer completion, so large files get the same 10-second pause before the next round.

Debug Lines

Exercise 305 includes machine-parseable debug records for the role-dependent BLE/Link failure investigation. Each record is one line of key=value fields.

RNSLINK   Link/announce events, peer hashes, Link ids, and Link object ids.
RNSTX     Application plaintext sends and encrypted Reticulum packets handed to BLE.
RNSRX     Reassembled BLE packets immediately before Reticulum receives them.
RNSDEC    Link encrypt/decrypt attempts and failures from microReticulum Link.cpp.
RNSBLE    BLE connect, identity, fragment TX/RX, and packet assembly events.
RNSQUEUE  BLE RX queue depth, pushes, pops, drops, and high-water marks.
RNSMEM    Heap, largest block, PSRAM, and current task stack high-water mark.
RNSERR    Classified adapter errors: reassembly gaps, short fragments, queue overflow, allocation failure.

Normal packet logging is rate-limited: first 25 packets, then every 25th packet, then all packets for two seconds after the first failure. Define RNS_DEBUG_VERBOSE=1 in platformio.ini to print every packet fingerprint.

Debug hooks are inserted at these points:

src/TBeamSupremeBleInterface.cpp  BLE callback boundary, fragment TX/RX, reassembly, queue, packet handoff.
src/main.cpp                     board-name mapping, announce/link events, application Link send.
/usr/local/src/microreticulum/microReticulum/src/Link.cpp
                                 Link encrypt/decrypt token fingerprints and classified decrypt failures.

Use the CRC fields to split the failure:

Same RNSTX token crc and RNSRX/RNSDEC token crc, but HMAC_INVALID -> likely wrong key/session/Link context.
Different RNSTX token crc and RNSRX/RNSDEC token crc -> BLE fragmentation, reassembly, queue, or buffer corruption.
RNSERR rx_reassembly_gap/timeout or queue_overflow -> mechanical adapter failure before Reticulum decrypt.
RNSMEM largest_block or min_heap collapse before failures -> heap pressure or fragmentation.