Herding T-Beam Cats

Highly technical, but written for those who are blessed with curiosity

Introduction

I’m testing microReticulum, a C++ implementation of the Reticulum protocol developed by Chad Attermann, on 7 LilyGo T-Beam SUPREMEs.

The T-Beams use the ESP32-S, a game-changing small processor.  I’ve built an elaborate testing bench which I wanted to document should someone in the same arena want to streamline their software workbench and leverage the efficiencies I have developed.  Below is a description of a practical, console-driven embedded-systems workflow for testing seven T-Beam nodes with PlatformIO, synchronized clocks, structured logs, and AI-assisted debugging.

Development Environment

First off, I do not use the windowed based Arduino integrated development environment (IDE 2).  I found the interface too complicated with options I’ll never use and you have to move around with a mouse and click which really slows you down.  Instead, I prefer a command line interface, e.g. a console, where I can script and tweak commands that I repeatedly use.

While the Arduino IDE is very popular and an excellent beginner’s tool, I’ve moved on to the Platform IO ¹.  PlatformIO appears to be an open source project with a for-profit company, PlatformIO Labs, aligned to it.  PlatformIO Labs is headquartered in Estonia, and notes with pride that its R&D unit is located in Ukraine ².  PlatformIO touts:

A place where Developers and Teams have true Freedom! No more vendor lock-in!

PlatformIO’s philosophy really resonates with me, I who worked for the world’s largest database software company for 24 years.  Moreover, the ability to “get things done” ³, or “Git-R-Done!” ⁴,  in a command console and with scripts is especially efficient and to me, less complicated to understand.

I created a virtual Python environment since Gentoo Linux ⁵

is my preferred platform.  You do not want to pollute or corrupt Gentoo’s installation of Python since Gentoo Linux heavily relies upon Python to manage its environment.  Instead, you create a micro universe with a copy of Python so you can install all kinds of risky packages and if you break your environment, then you just dispose of it and build another and you do not compromise your operating system.

PlatformIO is easily installed within a [virtual] Python environment. When you are working with compiling C++ code, say in the microReticulum project, PlatformIO creates a directory “.pio” and stages and builds everything thereunder.  You can control your PlatformIO in the file platformiol.ini.  In mine, I have entries for each of my T-Beam devices identified by name: AMY, BOB, CY, ED, FLO & GUY.⁷  The ability to differentiate the T-Beams is helpful, for instance, because GUY is a T-Beam with a different and more expensive GPS chip and circuit, so it needs to have different libraries to successfully compile.

T-Beam development

When I stage a project, I do so under the path: /usr/local/src.  This is an old-time practice I learned early on when I learned Linux.  On any given server or workstation I control, I can always know what customized software I have installed outside of the Gentoo package system by seeing what is staged under /usr/local/src.  Think of it as a sort of a mental quarantine area.  We used this protocol, for the most part, for the many servers my team controlled.

This T-Beam world was all new to me.  So I decided to creates studies, or “exercises”, to learn about the many functions of the T-Beam as well as to test its stated features. ⁶  I did not want to take the fast track and develop something only to learn later that a certain hardware capability existed that could improve the experience.

Building Firmware

Compiling the code and building the “firmware” — a binary blob that is the operating system for the T-Beam —  for each T-Beam takes over 1 minute on my 2012 Intel-based workstation.  So if I want to test a simple change in the code with all 7 T-Beams, I’m looking at 8 – 10 minutes to build the 7 firmwares.  When Codex starts making tiny incremental changes, this amount of time adds up real quickly making progress about 4 increments per hour.  I determined a solution is to have an identical workbench on my more powerful server, ryzdesk (chip: AMD Ryzen 7950, tons of memory), and build the firmware for each unit there.  On ryzdesk, I can build all 7 firmwares in 1 minute 4 seconds.  That is a time saving of about 8 minutes+.  So my method of work is to edit and interact with Codex on my slower “jp” workstation, check-in the changes and push [transfer] the revised code to my Forgejo server (running on a Raspberry Pi 4B), pull the changes down to ryzdesk and compile, then copy the build tree back to my workstation and load the firmwares into the respective T-Beams.  Much of this is facilitated using scripts.

Trial Runs

With the T-Beam inoculated with new firmware, I then run them.  I have a requirement that each unit must have its clock “disciplined” to satellite time every 24 hours.  This is to assure the time on each T-Beam is very close so when events are time-stamped, I know all the units are referencing the same time.  My procedure is that time synchronization must occur every 24 hours, if at start-up the unit sees that it has been over 24 hours since it last synchronized with satellites, it forces me to take the unit to a window or outside so it can obtain satellite signals and adjust its internal clock.  I maintain a stratum 1 time server (Raspberry Pi5) which my workstation, jp, is synchronized to.  So time codes on the T-Beams and my workstation should clearly be within 1/1,000th of a second.  It is imperative to have synchronized time to fully analyze log files.

I want the trial runs to start relatively at the same time and end at the same time, so I have designed Bash and Perl scripts which launch a new console and connects via a serial port to a given T-beam.  What is printed out in the console has filters highlighting entries of interest in color and prepends high precision time codes to the entries as they are printed out in the console.

In addition to displaying in the console, the serial port stream coming from the T-Beam is routed to a log file with a date and time stamp in the file name.  With my Perl script (clump_tbeam_logs.pl) I combine the log files generated from the recent trial into a subdirectory so I can just reference the subdirectory, e.g. ~/logs/20260531_1046, to Codex who can then find them and focus only that run.

Post Trial Analysis

The log files are a wealth of information, and using grep, I can filter out entries of interest such as what each unit received decrypted into readable text.  For instance, I have the firmwares configured so that BOB and CY are supposed to communicate via the Reticulum “transport” through DAN, but the results below show that neither BOB nor CY are receiving their LINK Transmission (“RX LINK”) messages, only the ANNOUNCE messages. (This is, therefore, a problem/bug either in my code or in the microreticulum code yet to be solved/fixed.)

This organization makes my interaction with Codex very efficient.  Codex will tweak the code, I’ll compile and run a trial using the modified code, and then I have log files in an isolated directory which Codex can analyze.  I actually had Codex connecting directly to my T-Beams: Codex could compile the code, upload it to the T-Beams, and then open parallel serial monitors and monitor in real time what was happening.  While that is all amazing, it is a very expensive use of Codex and I found my 5 hour allotments or AI time skyrocketing.  In the interest of husbandry and permanence, I opted for this method of running trials and then having log files so both Codex and I have the same results in a permanent location which can be referenced later on.  This workflow also makes Codex more useful and less wasteful. Instead of letting Codex burn time directly controlling hardware, I run the trial myself and preserve the evidence in a timestamped log directory. Codex can then inspect the same files I inspected, reason from the same run, and propose the next small change. That keeps the AI bounded by evidence rather than wandering through guesses. It also forces me to have to read stuff and learn and not hand over the keys to the kingdom to AI like many unsophisticated managers and CEOs of companies do thinking they can replace their software developers with AI and save money [in the short term].

During my testing, I brought to the attention of the manufacturer, LilyGO, that they published the T-Beam SUPREME’s specification touting a microphone.  When I set up an exercise to test the microphone… I learned there was no built-in microphone and contacted LilyGO. They admitted the error and have since updated their specifications removing mention of a microphone.

1. https://docs.platformio.org/en/latest/what-is-platformio.html?utm_source=chatgpt.com#philosophy Some “PlatformIO Labs is headquartered in Estonia, with representation in the USA and an R&D center in Ukraine.”    familiar companies use PlatformIO:

   

   ↩

2.  Ukraine — Commentary: it seems that Ukraine is emerging as a pre-eminent leader in drone technology which necessarily involves single board computers, aka “SBC”.  The T-Beam is an SBC with steroids, so it seems to be a natural fit that PlatformIO Lab’s R&D unit would be in Ukraine.  I can only speculate that the expertise needed to advance drone technology is in the same universe as work with the Espressif ESP32 series, so working with a tool set such as PlatformIO makes me feel like I’m on the cutting edge.  It certainly is meeting my expectations of what a tool set should do.↩
3.   “getting things done” – a trite, to being the point of amusing, phrase usually broadcasted by incumbents during their re-election campaigns.↩
4.   Git-R-Done! Catch phrase from Larry the Cable Guy ↩
5.  https://en.wikipedia.org/wiki/Gentoo_Linux ↩
6.  I had my Forgejo repository (https://salemdata.net/repo)available and was getting barraged by AI bots, so I configured it to registered-users-only access.  If you would like access, submit your email, preferred username, and real name in the form below (which will not be published) and I’ll contact you by email to arrange for an account.  No charges, I just do not want my servers and Internet connection flooded by AI bots. ↩
7. The case design is the Lilygo T-Beam SUPREME TS2 Case by Alley Cat.  Most were printed on a Sovol SV06 ACE 3D Printer (5/31/26 $259 Amazon).  Most of the cases in the photo are PLA [polylactide] filament, the red DAN was ASA [Acrylonitrile styrene acrylate]. The ASA requires high constant ambient temperature, so I have the SV06 inside the YOOPAI 3D Printer Enclosure (5/31/26 Amazon).↩