{"id":1041,"date":"2026-07-04T17:16:41","date_gmt":"2026-07-05T01:16:41","guid":{"rendered":"https:\/\/salemdata.net\/johnpress\/?p=1041"},"modified":"2026-07-04T17:16:41","modified_gmt":"2026-07-05T01:16:41","slug":"reticulum-a-precision-trace-among-7-nodes","status":"publish","type":"post","link":"https:\/\/salemdata.net\/johnpress\/?p=1041","title":{"rendered":"Reticulum: A Precision Trace Among 7 Nodes"},"content":{"rendered":"<p class=\"PDq2pG_selectionAnchorContainer\" data-start=\"4966\" data-end=\"5077\">What actually happens inside a Reticulum mesh running over LoRa when two endpoints cannot communicate directly?<\/p>\n<p data-start=\"5082\" data-end=\"5347\">I built a seven-node testbed using LilyGo T-Beam SUPREME units named AMY, BOB, CY, DAN, ED, FLO, and GUY. AMY and GUY exchange one encrypted \u201cHi\u201d message per minute, while software-defined radio blocks force their traffic through the other five nodes as transports.<\/p>\n<p data-start=\"5352\" data-end=\"5738\">All seven units discipline their clocks from GNSS satellites. In a roughly 17.5-minute synchronization run, the span among their median end-to-end observed clock offsets was about 0.166 ms, or 166 microseconds, and the minute-by-minute spread remained below about 0.633 ms. That common time base makes it possible to correlate events across seven independent logs with useful precision.<\/p>\n<h2>Background<\/h2>\n<p>I have seven LilyGo T-Beam SUPREME units named AMY, BOB, CY, DAN, ED, FLO &amp; GUY.\u00a0 The T-Beam is a small electrical assembly of components including a very powerful ESP32-S microprocessor which features wifi, bluetooth. There are also modules GPS<span id='easy-footnote-1-1041' class='easy-footnote-margin-adjust'><\/span><span class='easy-footnote'><a href='https:\/\/salemdata.net\/johnpress\/?p=1041#easy-footnote-bottom-1-1041' title='The LK76 or u-Blox versions are available.'><sup>1<\/sup><\/a><\/span> and LoRa radio.\u00a0 Reticulum\u2019s reference implementation is written in Python. For these constrained ESP32-based devices, I chose Chad Attermann\u2019s C++ microReticulum implementation. <span id='easy-footnote-2-1041' class='easy-footnote-margin-adjust'><\/span><span class='easy-footnote'><a href='https:\/\/salemdata.net\/johnpress\/?p=1041#easy-footnote-bottom-2-1041' title='There is a Rust based version of Reticulum being developed by Lew Palm.'><sup>2<\/sup><\/a><\/span><\/p>\n<figure id=\"attachment_925\" aria-describedby=\"caption-attachment-925\" style=\"width: 499px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-925\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/HopeStreet_FireBrigade_demo.jpg\" alt=\"Firefighters demonstrating a bucket brigade\" width=\"499\" height=\"333\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/HopeStreet_FireBrigade_demo.jpg 499w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/HopeStreet_FireBrigade_demo-300x200.jpg 300w\" sizes=\"auto, (max-width: 499px) 100vw, 499px\" \/><figcaption id=\"caption-attachment-925\" class=\"wp-caption-text\">Los Angeles County Firefighers Demonstrating A Bucket Brigade (2019)<\/figcaption><\/figure>\n<p>The T-Beams running microReticulum can provide a mesh network which essentially means some units can act as forwarders or middlemen between two radio units that are out of range from each other. I like to analogize a mesh network as akin to a bucket brigade.<\/p>\n<p>&nbsp;<\/p>\n<h2>What I have Accomplished as of July 4, 2026<\/h2>\n<p>I have modified microreticulum by riddling it with debug statements so I can understand what is going on at every step.\u00a0 Since my 7 units are all in proximity to one another, I have used blocks written into the software to simulate a field test where all units are dispersed and no more than two units are within range of each other.<\/p>\n<figure id=\"attachment_967\" aria-describedby=\"caption-attachment-967\" style=\"width: 784px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-967\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/seven_units-1.png\" alt=\"\" width=\"784\" height=\"386\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/seven_units-1.png 784w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/seven_units-1-300x148.png 300w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/seven_units-1-768x378.png 768w\" sizes=\"auto, (max-width: 784px) 100vw, 784px\" \/><figcaption id=\"caption-attachment-967\" class=\"wp-caption-text\">Seven Nodes<\/figcaption><\/figure>\n<p>Here is a diagram of the 7 units showing the possible paths of radio communication when they are all in the same room.\u00a0 What I have done is create artificial &#8220;blocks&#8221; in the software among various pairs so that Reticulum packets between AMY and GUY are forced to go through the other five units which act as transports.\u00a0 Since each unit has a custom block list, each unit has to be individually compiled.\u00a0 During my development I found that I could compile a 7 different firmwares on my high end server and save turn-around time by doing in 2 minutes what would take 10 minutes.<\/p>\n<p>A major hurdle I had to overcome was disciplining the clocks on each unit so everyone had the same time.\u00a0 That way when they produce logs, all the logs are using the same time and small differences such as one thousandth of a second, or more, can be accepted as real and not an aberration of different clocks.\u00a0 Learning to synchronize time to satellites took several weeks, and yesterday, to my surprise, I found Internet celebrity Engineer Jeff Geerling wrestling with the same issue with his pal <a href=\"https:\/\/www.youtube.com\/@SeanHodgins\">Sean Hodgins<\/a> in their collaboration and whimsical: <a href=\"https:\/\/www.youtube.com\/watch?v=AHSqyuOPpf0\">The Most Accurate Watch That Doesn&#8217;t Tell Time<\/a>\u00a0 \u00a0In a roughly 17.5-minute run, the span among the seven units\u2019 median end-to-end observed clock offsets was about 0.166 ms (166 \u00b5s &#8212; yes, that is 166 <strong>millionths<\/strong> of a second), with the minute-by-minute spread remaining below about 0.633 ms.\u00a0 The clock-synchronization analysis covered roughly 17.5 minutes. The detailed chronology and CAD plots below are from a separate 10-minute message run.<\/p>\n<p>My test run consists of the following:<\/p>\n<ul>\n<li>Only AMY and GUY talk to each, i.e. send &#8220;Hi&#8221; messages back and forth<\/li>\n<li>BOB, CY, DAN, ED &amp; FLO are active on the network, but they are not sending &#8220;Hi&#8221; messages, though they will forward packets on when requested.<\/li>\n<li>AMY and GUY are programmed to send a &#8220;Hi&#8221; message once a minute<\/li>\n<li>All units have the blocking rules illustrated above, so the only path for a message between AMY &amp; GUY is via the five other units.<\/li>\n<li>All units must discipline\/synchronize their internal clocks to 4 or more satellites, so the experiment must be conducted where they can readily see satellites.<\/li>\n<li>A successful run of 6 minutes should show in their logs 5 or more receptions of &#8220;Hi&#8221; messages from the other.<\/li>\n<\/ul>\n<p>Upshot of all of this is a high precision chronology table that serves as a time line.\u00a0 During the 10-minute run, AMY and GUY each logged eight successfully decrypted \u201cHi\u201d messages from the other endpoint. BOB, CY, DAN, ED, and FLO participated as transport nodes, forwarding traffic along the forced path.<\/p>\n<pre><span style=\"color: #00cd00;\"><b>jlpoole@jp<\/b><\/span><span style=\"color: #9d9df7;\"><b> ~\/logs\/20260704_1041 $<\/b><\/span> grep -n \"Hi\" *.log |nl\r\n     1\tAMY_raw_20260704_104100.log:323:20260704_104307.670 RNSLINKRX: ms=126733 board=library role=unknown event=decrypt_ok link_id=3ac84ff362419439d51b3606d57b431d plaintext_len=25 plaintext_crc32=658D6C47 text_encoding=percent text=GUY%20says%20Hi%20to%20AMY%20iter%3D0 link_obj={Link:3ac84ff362419439d51b3606d57b431d}\r\n     ...\r\n     9\tGUY_raw_20260704_104106.log:309:20260704_104309.001 RNSLINKRX: ms=122100 board=library role=unknown event=decrypt_ok link_id=3ac84ff362419439d51b3606d57b431d plaintext_len=25 plaintext_crc32=B936D34C text_encoding=percent text=AMY%20says%20Hi%20to%20GUY%20iter%3D0 link_obj={Link:3ac84ff362419439d51b3606d57b431d}\r\n     ...\r\n    16\tGUY_raw_20260704_104106.log:780:20260704_105009.033 RNSLINKRX: ms=542119 board=library role=unknown event=decrypt_ok link_id=1ecfef05be2b3d986e649549090bd38e plaintext_len=25 plaintext_crc32=275246EF text_encoding=percent text=AMY%20says%20Hi%20to%20GUY%20iter%3D7 link_obj={Link:1ecfef05be2b3d986e649549090bd38e}\r\n<span style=\"color: #00cd00;\"><b>jlpoole@jp<\/b><\/span><span style=\"color: #9d9df7;\"><b> ~\/logs\/20260704_1041 $<\/b><\/span> \r\n<\/pre>\n<p>The raw logs can be difficult to read, so I developed a database and parse all values of all seven logs.\u00a0 From the database, I can then generate an HTML table:<\/p>\n<figure id=\"attachment_1043\" aria-describedby=\"caption-attachment-1043\" style=\"width: 919px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-1043\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_172010_Sat.png\" alt=\"Screenshot of HTML table\" width=\"919\" height=\"676\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_172010_Sat.png 919w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_172010_Sat-300x221.png 300w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_172010_Sat-768x565.png 768w\" sizes=\"auto, (max-width: 919px) 100vw, 919px\" \/><figcaption id=\"caption-attachment-1043\" class=\"wp-caption-text\">Chronology Table<\/figcaption><\/figure>\n<p>The full table, with its 800+ entries for the 10 minute run is available here: <a href=\"https:\/\/salemdata.net\/dev\/reticulum\/microreticulum\/20260704_1041\/EX_205_20260704_1041_chronology.html\">microReticulum LINK Chronological Event Table<\/a><\/p>\n<p>One of the issues I had not given much thought about is the sharing of the radio wave.\u00a0 All units use the same LoRa RF channel, so they must share airtime much like callers sharing an old telephone party line.\u00a0 Before transmitting, a node performs Channel Activity Detection (aka &#8220;CAD&#8221;). If the channel is busy, it waits and retries rather than transmitting immediately. With all the activity of forwarding packets to and from, there were contentions.\u00a0 Using a radio wave waterfall tool, OpenWebrx, I saw mild usage, but in several experiments (a run-away forwarding chain with 255 hops) I encountered saturation which I fixed later by placing guards within the software<\/p>\n<figure id=\"attachment_869\" aria-describedby=\"caption-attachment-869\" style=\"width: 447px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-869\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/20260611_195639_Thu.png\" alt=\"Openwebrx screen showing radio signal waterfall\" width=\"447\" height=\"296\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/20260611_195639_Thu.png 1376w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/20260611_195639_Thu-300x199.png 300w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/06\/20260611_195639_Thu-768x510.png 768w\" sizes=\"auto, (max-width: 447px) 100vw, 447px\" \/><figcaption id=\"caption-attachment-869\" class=\"wp-caption-text\">Openwebrx Waterfall with Time Correlation<\/figcaption><\/figure>\n<figure id=\"attachment_1044\" aria-describedby=\"caption-attachment-1044\" style=\"width: 375px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-1044\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260627_180607_Sat.png\" alt=\"Screenshot of Openwebrx showing heavy usage of the 915 Mhz frequency\" width=\"375\" height=\"294\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260627_180607_Sat.png 1292w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260627_180607_Sat-300x235.png 300w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260627_180607_Sat-768x602.png 768w\" sizes=\"auto, (max-width: 375px) 100vw, 375px\" \/><figcaption id=\"caption-attachment-1044\" class=\"wp-caption-text\">Intensive Usage Of Bandwidth<\/figcaption><\/figure>\n<p>&nbsp;<\/p>\n<p>I therefore plotted out the times the units were having to delay their transmissions because the airwave was currently being used.<\/p>\n<p>Below are screenshots of the Plot.<\/p>\n<figure id=\"attachment_1046\" aria-describedby=\"caption-attachment-1046\" style=\"width: 796px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-1046\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173233_Sat.png\" alt=\"\" width=\"796\" height=\"527\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173233_Sat.png 1228w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173233_Sat-300x199.png 300w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173233_Sat-768x508.png 768w\" sizes=\"auto, (max-width: 796px) 100vw, 796px\" \/><figcaption id=\"caption-attachment-1046\" class=\"wp-caption-text\">Channel Activity Detection Delays &#8211; Normal View<\/figcaption><\/figure>\n<p>In the plot, you can zoom in and out and pan about.\u00a0 Here is a zoomed-in portion for the first cluster.<\/p>\n<figure id=\"attachment_1045\" aria-describedby=\"caption-attachment-1045\" style=\"width: 777px\" class=\"wp-caption aligncenter\"><img loading=\"lazy\" decoding=\"async\" class=\" wp-image-1045\" src=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173316_Sat.png\" alt=\"\" width=\"777\" height=\"414\" srcset=\"https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173316_Sat.png 1511w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173316_Sat-300x160.png 300w, https:\/\/salemdata.net\/johnpress\/wp-content\/uploads\/2026\/07\/20260704_173316_Sat-768x409.png 768w\" sizes=\"auto, (max-width: 777px) 100vw, 777px\" \/><figcaption id=\"caption-attachment-1045\" class=\"wp-caption-text\">Channel Activity Detection Delays &#8211; Zoomed in for details<\/figcaption><\/figure>\n<p>The interactive plot is available at: <a href=\"https:\/\/salemdata.net\/dev\/reticulum\/microreticulum\/20260704_1041\/EX_205_20260704_1041_CAD_delays_20260704_1423.html\">Channel Activity Detection Delay<\/a><\/p>\n<p>All log files and the SQLite database are available <a href=\"https:\/\/salemdata.net\/dev\/reticulum\/microreticulum\/20260704_1041\/\">here<\/a>. Access to my <a href=\"https:\/\/salemdata.net\/repo\">Forgejo source code<\/a> is available on request, I had my Forgejo instance publicly accessible, but soon bots starts hitting my sight bout 10 times a second, so I have restricted Forgejo access to authenticated accounts.<\/p>\n<p>I could not have possibly accomplished this alone and I relied heavily upon ChatGPT and Codex using my $20\/month subscription.\u00a0 I&#8217;ve used ChatGPT as an architect collaborator and Codex as the software engineer.\u00a0 I chose not to pursue the management path at my former employer (24 years), so never had to &#8220;manage.&#8221;\u00a0 My wife quips now I am playing that role between the two AIs.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>What actually happens inside a Reticulum mesh running over LoRa when two endpoints cannot communicate directly? I built a seven-node testbed using LilyGo T-Beam SUPREME units named AMY, BOB, CY, DAN, ED, FLO, and GUY. AMY and GUY exchange one encrypted \u201cHi\u201d message per minute, while software-defined radio blocks force their traffic through the other [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":1025,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_gfmr_meta_descriptions":[],"_gfmr_multilingual_taxonomy_terms":[],"footnotes":""},"categories":[127,126,26,133,80,131,130,121],"tags":[],"class_list":["post-1041","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-chatgpt","category-codex","category-electronics","category-mesh-network","category-meshtastic","category-microreticulum","category-reticulum","category-t-beam"],"_links":{"self":[{"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/posts\/1041","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1041"}],"version-history":[{"count":8,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/posts\/1041\/revisions"}],"predecessor-version":[{"id":1053,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/posts\/1041\/revisions\/1053"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=\/wp\/v2\/media\/1025"}],"wp:attachment":[{"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1041"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1041"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/salemdata.net\/johnpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1041"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}