From ca88c6b4c97fa013128efa8d20d35132ca1c96c5 Mon Sep 17 00:00:00 2001 From: torlando-tech Date: Mon, 29 Dec 2025 23:38:21 -0500 Subject: [PATCH] fix: restore RNS.Interfaces.Interface import for base class MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The sed replacement was too aggressive - it replaced the import for the base Interface class from the Reticulum package itself. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 --- .claude/commands/fix-ci.md | 10 + .claude/commands/fix-issue.md | 8 + COLUMBA_REFACTORING_GUIDE.md | 148 ++++++++++ comprehensive_refactor.py | 476 ++++++++++++++++++++++++++++++ perform_refactor.py | 109 +++++++ refactor_ble_interface.py | 105 +++++++ refactor_helper.py | 43 +++ refactor_pass2.py | 310 +++++++++++++++++++ src/ble_reticulum/BLEInterface.py | 9 +- 9 files changed, 1212 insertions(+), 6 deletions(-) create mode 100644 .claude/commands/fix-ci.md create mode 100644 .claude/commands/fix-issue.md create mode 100644 COLUMBA_REFACTORING_GUIDE.md create mode 100644 comprehensive_refactor.py create mode 100644 perform_refactor.py create mode 100644 refactor_ble_interface.py create mode 100644 refactor_helper.py create mode 100644 refactor_pass2.py diff --git a/.claude/commands/fix-ci.md b/.claude/commands/fix-ci.md new file mode 100644 index 0000000..2af0629 --- /dev/null +++ b/.claude/commands/fix-ci.md @@ -0,0 +1,10 @@ +Check the GitHub Actions CI status and fix any failures: + +1. Use `gh run list --limit 1` to get the latest run +2. Use `gh run view --log` to see what failed +3. Analyze the error logs +4. Fix the issues in the code +5. Run tests locally to verify +6. Commit and push the fix +7. Monitor the new CI run with `gh run watch` +8. If it fails again, iterate until it passes diff --git a/.claude/commands/fix-issue.md b/.claude/commands/fix-issue.md new file mode 100644 index 0000000..9dc77a1 --- /dev/null +++ b/.claude/commands/fix-issue.md @@ -0,0 +1,8 @@ +Please analyze and fix the GitHub issue: $ARGUMENTS. Follow these steps: +1. Use `gh issue view` to get the issue details +2. Understand the problem described in the issue +3. Search the codebase for relevant files +4. Implement the necessary changes to fix the issue +5. Run tests to verify the fix works +6. Create a PR with `gh pr create` with a clear description +7. Link the PR to the issue diff --git a/COLUMBA_REFACTORING_GUIDE.md b/COLUMBA_REFACTORING_GUIDE.md new file mode 100644 index 0000000..8ed4603 --- /dev/null +++ b/COLUMBA_REFACTORING_GUIDE.md @@ -0,0 +1,148 @@ + +# Refactoring Columba's BLE Layer to a Driver-Based Architecture + +## 1. Goal + +This guide outlines the process of refactoring the existing BLE implementation in the Columba Android project to align with the new driver-based architecture of the `ble-reticulum` project. + +The goal is to: +- Reuse the battle-tested `BLEInterface.py` from `ble-reticulum` as the main Reticulum logic for BLE in Columba. +- Create a new Android-specific BLE driver in Python (`AndroidBLEDriver.py`) that implements the `BLEDriverInterface`. +- Bridge this new Python driver to a dedicated Kotlin class (`KotlinBLEBridge.kt`) that handles all native Android BLE operations. +- Isolate the Kotlin BLE logic from the rest of the Columba UI application, with the `KotlinBLEBridge` acting as the sole entry point for the Python layer. + +This will result in a more modular, maintainable, and testable system, and will allow Columba to easily stay up-to-date with the latest improvements in the `ble-reticulum` project. + +## 2. Current State Analysis + +Based on the file structure of the Columba project, the current BLE implementation is a monolithic Kotlin implementation with a Python bridge. + +- **Kotlin:** The core BLE logic is in the `com.lxmf.messenger.reticulum.ble` package, with classes like `BleConnectionManager`, `BleGattClient`, `BleGattServer`, `BleScanner`, and `BleAdvertiser`. +- **Python:** The `rn_ble_interface.py` script acts as the Chaquopy bridge, importing and using the Kotlin classes to create a Reticulum interface. + +This architecture is tightly coupled, making it difficult to update and maintain. The new driver-based architecture will address these issues. + +## 3. Proposed Architecture + +The new architecture will consist of three main components: + +1. **`BLEInterface.py`:** The high-level, platform-agnostic Reticulum interface logic from the `ble-reticulum` project. +2. **`AndroidBLEDriver.py`:** A new Python class that implements the `BLEDriverInterface` and acts as a bridge to the Kotlin layer. +3. **`KotlinBLEBridge.kt`:** A new, isolated Kotlin class that exposes a clean API for the `AndroidBLEDriver.py` to interact with the native Android BLE stack. + +This architecture will allow us to reuse the `BLEInterface.py` and only implement the platform-specific BLE operations in the `AndroidBLEDriver.py` and `KotlinBLEBridge.kt`. + +## 4. Step-by-Step Refactoring Guide + +### Step 1: Create the `KotlinBLEBridge.kt` + +Create a new Kotlin class, `KotlinBLEBridge.kt`, in the `com.lxmf.messenger.reticulum.ble.service` package. This class will be the single entry point for all BLE operations from the Python layer. It should be a singleton and should not have any dependencies on the Columba UI. + +The `KotlinBLEBridge.kt` class should expose methods that correspond to the `BLEDriverInterface` in Python. For example: + +```kotlin +class KotlinBLEBridge(private val context: Context) { + + fun start(serviceUuid: String, rxCharUuid: String, txCharUuid: String, identityCharUuid: String) { + // Initialize the BLE stack + } + + fun stop() { + // Stop all BLE activity + } + + fun setIdentity(identityBytes: ByteArray) { + // Set the identity for the GATT server + } + + fun startScanning() { + // Start scanning for devices + } + + fun stopScanning() { + // Stop scanning + } + + fun startAdvertising(deviceName: String) { + // Start advertising + } + + fun stopAdvertising() { + // Stop advertising + } + + fun connect(address: String) { + // Connect to a device + } + + fun disconnect(address: String) { + // Disconnect from a device + } + + fun send(address: String, data: ByteArray) { + // Send data to a device + } + + // ... other methods as needed +} +``` + +This class will also be responsible for invoking the callbacks on the Python driver. You can use a listener interface to achieve this. + +### Step 2: Create the `AndroidBLEDriver.py` + +Create a new Python file, `AndroidBLEDriver.py`, in the `columba/app/src/main/python` directory. This class will implement the `BLEDriverInterface` and will use Chaquopy to call the methods of the `KotlinBLEBridge`. + +```python +from com.chaquo.python import Python +from RNS.Interfaces.bluetooth_driver import BLEDriverInterface, BLEDevice, DriverState + +class AndroidBLEDriver(BLEDriverInterface): + def __init__(self): + self.kotlin_ble_bridge = Python.getPlatform().getApplication().getKotlinBLEBridge() + # Set up callbacks from Kotlin to Python + # ... + + def start(self, service_uuid, rx_char_uuid, tx_char_uuid, identity_char_uuid): + self.kotlin_ble_bridge.start(service_uuid, rx_char_uuid, tx_char_uuid, identity_char_uuid) + + def stop(self): + self.kotlin_ble_bridge.stop() + + # ... implement all other methods of the BLEDriverInterface + +``` + +### Step 3: Refactor `rn_ble_interface.py` + +Modify the existing `rn_ble_interface.py` to use the new `BLEInterface` and `AndroidBLEDriver`. + +```python +# rn_ble_interface.py + +from RNS.Interfaces.BLEInterface import BLEInterface +from AndroidBLEDriver import AndroidBLEDriver + +# ... other imports + +class RNBLEInterface(BLEInterface): + def __init__(self, owner, config): + driver = AndroidBLEDriver() + super().__init__(owner, config, driver=driver) + +# ... rest of the file +``` + +### Step 4: Replace the old BLE implementation + +Once the new driver-based architecture is in place, you can start removing the old BLE implementation in the Columba project. This includes classes like `BleConnectionManager`, `BleGattClient`, `BleGattServer`, etc. The new `KotlinBLEBridge` should encapsulate all the necessary BLE logic. + +## 5. Testing + +Thorough testing is crucial for this refactoring. + +- **Unit Tests:** Write unit tests for the `KotlinBLEBridge` to ensure that it correctly interacts with the Android BLE stack. +- **Integration Tests:** Write integration tests that verify the communication between the `AndroidBLEDriver.py` and the `KotlinBLEBridge.kt`. +- **End-to-End Tests:** Run the full Columba application and test the BLE functionality to ensure that everything works as expected. + +By following this guide, you can refactor the Columba BLE layer to a more modern, modular, and maintainable architecture, while at the same time reusing the battle-tested `BLEInterface.py` from the `ble-reticulum` project. diff --git a/comprehensive_refactor.py b/comprehensive_refactor.py new file mode 100644 index 0000000..13f1b73 --- /dev/null +++ b/comprehensive_refactor.py @@ -0,0 +1,476 @@ +#!/usr/bin/env python3 +""" +Comprehensive refactoring script for BLEInterface.py to use driver abstraction. + +This script: +1. Removes platform-specific imports (bleak, bluezero, dbus_fast, monkey patch) +2. Adds driver abstraction imports +3. Refactors __init__ to create and configure driver +4. Removes async methods moved to driver +5. Adds driver callback implementations +6. Updates BLE operations to use driver calls +""" + +import re + +def read_file(path): + with open(path, 'r') as f: + return f.read() + +def write_file(path, content): + with open(path, 'w') as f: + f.write(content) + +def remove_imports_and_add_driver_imports(content): + """Remove bleak/bluezero/monkey patch, add driver imports.""" + + # Find the section to replace (from "# Check for bleak" to end of monkey patch) + pattern = r'# Check for bleak dependency.*?(?=class DiscoveredPeer)' + + replacement = '''# Import driver abstraction +try: + from bluetooth_driver import BLEDriverInterface, BLEDevice +except ImportError: + try: + from RNS.Interfaces.bluetooth_driver import BLEDriverInterface, BLEDevice + except ImportError: + # Fallback to root directory + import sys + import os + sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../"))) + from bluetooth_driver import BLEDriverInterface, BLEDevice + +# Import platform-specific driver +try: + from linux_bluetooth_driver import LinuxBluetoothDriver +except ImportError: + try: + from RNS.Interfaces.linux_bluetooth_driver import LinuxBluetoothDriver + except ImportError: + # Fallback to root directory + import sys + import os + sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../"))) + from linux_bluetooth_driver import LinuxBluetoothDriver + +HAS_DRIVER = True + +''' + + content = re.sub(pattern, replacement, content, flags=re.DOTALL) + return content + +def remove_method(content, method_name): + """Remove a method definition entirely.""" + # Pattern to match method definition and its body + # Match from "def method_name" or "async def method_name" until the next method/class definition + pattern = rf'^( )(async )?def {method_name}\(.*?\n((?:(?!\1(?:def|async def|class)\b).*\n)*)' + content = re.sub(pattern, '', content, flags=re.MULTILINE) + return content + +def refactor_init_method(content): + """Refactor __init__ to use driver abstraction.""" + + # Replace HAS_BLEAK check with HAS_DRIVER + content = content.replace( + 'if not HAS_BLEAK:\n raise ImportError(\n "BLEInterface requires the \'bleak\' library. "\n "Install with: pip install bleak==1.1.1"\n )', + 'if not HAS_DRIVER:\n raise ImportError(\n "BLEInterface requires the driver abstraction. "\n "Ensure bluetooth_driver.py and linux_bluetooth_driver.py are available."\n )' + ) + + # Remove GATT server creation section (lines starting with "# GATT server for peripheral mode" until "# Fragmentation") + pattern = r' # GATT server for peripheral mode.*?(?= # Fragmentation)' + content = re.sub(pattern, '', content, flags=re.DOTALL) + + # Remove async loop setup (lines starting with "# Async event loop" until "# Discovery state") + pattern = r' # Async event loop.*?(?= # Discovery state)' + content = re.sub(pattern, '', content, flags=re.DOTALL) + + # Remove BlueZ version detection + content = content.replace( + ' # BlueZ version and capabilities (for LE-specific connection support)\n self.bluez_version = self._detect_bluez_version()\n self.has_connect_device = False # Set to True if ConnectDevice() available\n', + '' + ) + + # Add driver creation after fragmentation section + driver_init = ''' + # Initialize BLE driver + self.driver = LinuxBluetoothDriver( + discovery_interval=self.discovery_interval, + connection_timeout=self.connection_timeout, + min_rssi=self.min_rssi, + service_discovery_delay=self.service_discovery_delay, + max_peers=self.max_peers, + adapter_index=0 # TODO: Make configurable + ) + + # Set driver callbacks + self.driver.on_device_discovered = self._device_discovered_callback + self.driver.on_device_connected = self._device_connected_callback + self.driver.on_mtu_negotiated = self._mtu_negotiated_callback + self.driver.on_data_received = self._data_received_callback + self.driver.on_device_disconnected = self._device_disconnected_callback + self.driver.on_error = self._error_callback + + # Set driver power mode + self.driver.set_power_mode(self.power_mode) +''' + + # Insert after "# Discovery state with prioritization" line + content = content.replace( + ' # Discovery state with prioritization\n', + ' # Discovery state with prioritization\n' + driver_init + '\n' + ) + + return content + +def add_driver_callbacks(content): + """Add driver callback implementations after _periodic_cleanup method.""" + + callbacks = ''' + def _device_discovered_callback(self, device: BLEDevice): + """ + Driver callback: Handle discovered BLE device. + + This callback is invoked by the driver when a device is discovered during scanning. + We use peer scoring and connection logic to decide whether to connect. + """ + # Update or create discovered peer entry + if device.address not in self.discovered_peers: + self.discovered_peers[device.address] = DiscoveredPeer( + address=device.address, + name=device.name, + rssi=device.rssi + ) + else: + self.discovered_peers[device.address].update_rssi(device.rssi) + + # Prune discovery cache if needed (HIGH #4) + if len(self.discovered_peers) > self.max_discovered_peers: + # Remove oldest entries by last_seen timestamp + sorted_peers = sorted( + self.discovered_peers.items(), + key=lambda x: x[1].last_seen + ) + to_remove = sorted_peers[:-self.max_discovered_peers] + for addr, _ in to_remove: + del self.discovered_peers[addr] + + # Decide whether to connect based on peer scoring + peers_to_connect = self._select_peers_to_connect() + if device.address in [p.address for p in peers_to_connect]: + # Initiate connection via driver + try: + self.driver.connect(device.address) + except Exception as e: + RNS.log(f"{self} failed to initiate connection to {device.name}: {e}", RNS.LOG_ERROR) + + def _device_connected_callback(self, address: str): + """ + Driver callback: Handle successful device connection. + + Called when driver has established a connection. We read the identity + characteristic and prepare to receive data. + """ + RNS.log(f"{self} connected to {address}, reading identity...", RNS.LOG_INFO) + + # Read identity characteristic + try: + identity_bytes = self.driver.read_characteristic( + address, + BLEInterface.CHARACTERISTIC_IDENTITY_UUID + ) + + if identity_bytes and len(identity_bytes) == 16: + peer_identity = bytes(identity_bytes) + identity_hash = self._compute_identity_hash(peer_identity) + + # Store identity mappings + self.address_to_identity[address] = peer_identity + self.identity_to_address[identity_hash] = address + + RNS.log(f"{self} received peer identity from {address}: {identity_hash}", RNS.LOG_INFO) + + # Record successful connection + self._record_connection_success(address) + + else: + RNS.log(f"{self} invalid identity from {address}, disconnecting", RNS.LOG_WARNING) + self.driver.disconnect(address) + self._record_connection_failure(address) + + except Exception as e: + RNS.log(f"{self} failed to read identity from {address}: {e}", RNS.LOG_ERROR) + self.driver.disconnect(address) + self._record_connection_failure(address) + + def _mtu_negotiated_callback(self, address: str, mtu: int): + """ + Driver callback: Handle MTU negotiation completion. + + Creates or updates the fragmenter for this peer with the negotiated MTU. + """ + RNS.log(f"{self} MTU negotiated with {address}: {mtu} bytes", RNS.LOG_INFO) + + # Get peer identity + peer_identity = self.address_to_identity.get(address) + if not peer_identity: + RNS.log(f"{self} no identity for {address}, cannot create fragmenter", RNS.LOG_WARNING) + return + + # Create or update fragmenter + frag_key = self._get_fragmenter_key(peer_identity, address) + + with self.frag_lock: + # Create fragmenter with MTU + self.fragmenters[frag_key] = BLEFragmenter(mtu=mtu) + + # Create reassembler if not exists + if frag_key not in self.reassemblers: + self.reassemblers[frag_key] = BLEReassembler() + + # Spawn peer interface if not exists + identity_hash = self._compute_identity_hash(peer_identity) + if identity_hash not in self.spawned_interfaces: + # Get peer name from discovered peers + peer_name = None + if address in self.discovered_peers: + peer_name = self.discovered_peers[address].name + else: + peer_name = f"BLE-{address[-8:]}" + + # Determine connection type based on MAC sorting + connection_type = "central" + if self.driver.get_local_address(): + local_mac = self.driver.get_local_address().lower() + peer_mac = address.lower() + if local_mac > peer_mac: + connection_type = "peripheral" + + self._spawn_peer_interface( + address=address, + name=peer_name, + peer_identity=peer_identity, + mtu=mtu, + connection_type=connection_type + ) + + def _data_received_callback(self, address: str, data: bytes): + """ + Driver callback: Handle received data from peer. + + Passes data to reassembly and routing logic. + """ + self._handle_ble_data(address, data) + + def _device_disconnected_callback(self, address: str): + """ + Driver callback: Handle device disconnection. + + Cleans up peer state, interfaces, and fragmentation buffers. + """ + RNS.log(f"{self} disconnected from {address}", RNS.LOG_INFO) + + # Clean up peer connection state + with self.peer_lock: + if address in self.peers: + del self.peers[address] + + # Detach interface + peer_identity = self.address_to_identity.get(address) + if peer_identity: + identity_hash = self._compute_identity_hash(peer_identity) + if identity_hash in self.spawned_interfaces: + peer_if = self.spawned_interfaces[identity_hash] + peer_if.detach() + del self.spawned_interfaces[identity_hash] + RNS.log(f"{self} detached interface for {address}", RNS.LOG_DEBUG) + + # Clean up fragmenter/reassembler + if peer_identity: + frag_key = self._get_fragmenter_key(peer_identity, address) + with self.frag_lock: + if frag_key in self.fragmenters: + del self.fragmenters[frag_key] + if frag_key in self.reassemblers: + del self.reassemblers[frag_key] + + def _error_callback(self, severity: str, message: str, exc: Exception = None): + """ + Driver callback: Handle driver errors. + + Logs errors with appropriate severity level. + """ + if severity == "critical": + log_level = RNS.LOG_CRITICAL + elif severity == "error": + log_level = RNS.LOG_ERROR + elif severity == "warning": + log_level = RNS.LOG_WARNING + else: + log_level = RNS.LOG_DEBUG + + if exc: + RNS.log(f"{self} driver {severity}: {message} - {type(exc).__name__}: {exc}", log_level) + else: + RNS.log(f"{self} driver {severity}: {message}", log_level) +''' + + # Insert callbacks after _periodic_cleanup method + # Find the end of _periodic_cleanup (next method definition) + pattern = r'( async def _periodic_cleanup\(self\):.*?(?=\n def ))' + match = re.search(pattern, content, re.DOTALL) + if match: + insert_pos = match.end() + content = content[:insert_pos] + '\n' + callbacks + content[insert_pos:] + + return content + +def refactor_start_method(content): + """Refactor start() method to use driver.""" + + # Replace loop thread creation with driver start + old_start = r' # Create and start async event loop in separate thread\s+self\.loop_thread = threading\.Thread\(target=self\._run_async_loop, daemon=True\)\s+self\.loop_thread\.start\(\)\s+# Wait for loop to initialize.*?return' + + new_start = ''' # Start the BLE driver + try: + self.driver.start( + service_uuid=self.service_uuid, + rx_char_uuid=BLEInterface.CHARACTERISTIC_RX_UUID, + tx_char_uuid=BLEInterface.CHARACTERISTIC_TX_UUID, + identity_char_uuid=BLEInterface.CHARACTERISTIC_IDENTITY_UUID + ) + RNS.log(f"{self} driver started successfully", RNS.LOG_INFO) + except Exception as e: + RNS.log(f"{self} failed to start driver: {e}", RNS.LOG_ERROR) + return''' + + content = re.sub(old_start, new_start, content, flags=re.DOTALL) + + # Remove discovery and cleanup task scheduling + content = content.replace( + ' # Schedule discovery to start (if central mode enabled)\n if self.enable_central:\n asyncio.run_coroutine_threadsafe(self._start_discovery(), self.loop)\n else:\n RNS.log(f"{self} central mode disabled, skipping peer discovery", RNS.LOG_INFO)\n\n # Start periodic cleanup task (CRITICAL #2: prevent unbounded reassembly buffer growth)\n asyncio.run_coroutine_threadsafe(self._periodic_cleanup(), self.loop)\n', + '' + ) + + return content + +def refactor_final_init(content): + """Refactor final_init() to set identity on driver and start advertising.""" + + old_final_init = r' def final_init\(self\):.*?(?=\n def _start_gatt_when_identity_ready)' + + new_final_init = ''' def final_init(self): + """ + Interface lifecycle hook called AFTER interface is added to Transport.interfaces + but BEFORE Transport.start() loads Transport.identity. + + Use this to start a background thread that waits for Transport.identity to be + loaded, then sets it on the driver and starts advertising. + """ + if self.enable_peripheral: + RNS.log(f"{self} Launching driver advertising startup thread (will wait for Transport.identity)", RNS.LOG_DEBUG) + startup_thread = threading.Thread(target=self._start_advertising_when_identity_ready, daemon=True, name="BLE-Advertising-Startup") + startup_thread.start() + + def _start_advertising_when_identity_ready(self): + """ + Background thread that waits for Transport.identity, sets it on driver, + then starts advertising. Times out after 60 seconds if identity doesn't load. + """ + import RNS.Transport as Transport + + attempt = 0 + start_time = time.time() + timeout = 60.0 # 60 second timeout + + RNS.log(f"{self} Waiting for Transport.identity to be loaded...", RNS.LOG_DEBUG) + + # Poll until Transport.identity is available (with 60s timeout) + while time.time() - start_time < timeout: + attempt += 1 + + try: + if hasattr(Transport, 'identity') and Transport.identity: + identity_hash = Transport.identity.hash + if identity_hash and len(identity_hash) == 16: + elapsed = time.time() - start_time + RNS.log(f"{self} Transport.identity available after {elapsed:.1f}s", RNS.LOG_INFO) + + # Generate identity-based device name if not configured + if self.device_name is None: + identity_str = identity_hash.hex() # Full 16 bytes as 32 hex chars + self.device_name = f"RNS-{identity_str}" + RNS.log(f"{self} Auto-generated identity-based device name: {self.device_name}", RNS.LOG_INFO) + + # Set identity on driver + self.driver.set_identity(identity_hash) + + # Start advertising + try: + self.driver.start_advertising(self.device_name, identity_hash) + RNS.log(f"{self} Started advertising as {self.device_name}", RNS.LOG_INFO) + except Exception as e: + RNS.log(f"{self} Failed to start advertising: {e}", RNS.LOG_ERROR) + + return + + except Exception as e: + RNS.log(f"{self} Error waiting for identity: {e}", RNS.LOG_DEBUG) + + time.sleep(0.5) + + RNS.log(f"{self} Timeout waiting for Transport.identity after {timeout}s", RNS.LOG_ERROR) +''' + + content = re.sub(old_final_init, new_final_init, content, flags=re.DOTALL) + + return content + +def main(): + input_file = 'src/RNS/Interfaces/BLEInterface.py' + + print("Reading file...") + content = read_file(input_file) + + print("Step 1: Removing imports and adding driver imports...") + content = remove_imports_and_add_driver_imports(content) + + print("Step 2: Removing async methods moved to driver...") + methods_to_remove = [ + '_run_async_loop', + '_detect_bluez_version', + '_log_bluez_config', + '_connect_via_dbus_le', + '_get_local_adapter_address', + '_start_discovery', + '_start_server', + '_discover_peers', + '_connect_to_peer' + ] + for method in methods_to_remove: + print(f" Removing {method}...") + content = remove_method(content, method) + + print("Step 3: Refactoring __init__ method...") + content = refactor_init_method(content) + + print("Step 4: Refactoring start() method...") + content = refactor_start_method(content) + + print("Step 5: Refactoring final_init() method...") + content = refactor_final_init(content) + + print("Step 6: Adding driver callbacks...") + content = add_driver_callbacks(content) + + print("Writing refactored file...") + write_file(input_file, content) + + print("Done! Refactoring complete.") + print("\nManual review needed for:") + print(" - BLEPeerInterface._send_via_central() and _send_via_peripheral()") + print(" - Any remaining bleak/bluezero references") + print(" - Local address retrieval (now driver.get_local_address())") + +if __name__ == '__main__': + main() diff --git a/perform_refactor.py b/perform_refactor.py new file mode 100644 index 0000000..c15ebac --- /dev/null +++ b/perform_refactor.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +""" +Comprehensively refactor BLEInterface.py to use driver abstraction. +""" + +def main(): + input_file = '/home/tyler/repos/public/ble-reticulum/src/RNS/Interfaces/BLEInterface.py' + output_file = input_file # Overwrite + + with open(input_file, 'r') as f: + lines = f.readlines() + + new_lines = [] + skip_until = -1 # Line number to skip until + in_method_to_remove = False + method_indent = 0 + + i = 0 + while i < len(lines): + line = lines[i] + line_no = i + 1 + + # Skip lines we've marked for deletion + if i < skip_until: + i += 1 + continue + + # Remove bleak/bluezero imports (lines 99-172 approximately) + if line_no == 99 and '# Check for bleak dependency' in line: + # Skip until we find the end of monkey patch section (line 172) + while i < len(lines) and not (i > 172 or 'class DiscoveredPeer' in lines[i]): + i += 1 + # Add driver imports instead + new_lines.append('# Import driver abstraction\n') + new_lines.append('try:\n') + new_lines.append(' from bluetooth_driver import BLEDriverInterface, BLEDevice\n') + new_lines.append('except ImportError:\n') + new_lines.append(' try:\n') + new_lines.append(' from RNS.Interfaces.bluetooth_driver import BLEDriverInterface, BLEDevice\n') + new_lines.append(' except ImportError:\n') + new_lines.append(' # Fallback to root directory\n') + new_lines.append(' import sys\n') + new_lines.append(' import os\n') + new_lines.append(' sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../")))\n') + new_lines.append(' from bluetooth_driver import BLEDriverInterface, BLEDevice\n') + new_lines.append('\n') + new_lines.append('# Import platform-specific driver\n') + new_lines.append('try:\n') + new_lines.append(' from linux_bluetooth_driver import LinuxBluetoothDriver\n') + new_lines.append('except ImportError:\n') + new_lines.append(' try:\n') + new_lines.append(' from RNS.Interfaces.linux_bluetooth_driver import LinuxBluetoothDriver\n') + new_lines.append(' except ImportError:\n') + new_lines.append(' # Fallback to root directory\n') + new_lines.append(' import sys\n') + new_lines.append(' import os\n') + new_lines.append(' sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), "../../../")))\n') + new_lines.append(' from linux_bluetooth_driver import LinuxBluetoothDriver\n') + new_lines.append('\n') + new_lines.append('HAS_DRIVER = True\n') + new_lines.append('\n') + continue + + # Detect methods to remove + methods_to_remove = [ + '_run_async_loop', + '_detect_bluez_version', + '_log_bluez_config', + '_connect_via_dbus_le', + '_get_local_adapter_address', + '_start_discovery', + '_start_server', + '_discover_peers', + '_connect_to_peer' + ] + + # Check if we're entering a method to remove + if any(f'def {method}' in line for method in methods_to_remove): + # Get the indent level of this method + method_indent = len(line) - len(line.lstrip()) + in_method_to_remove = True + RNS.log(f"Removing method at line {line_no}: {line.strip()[:50]}") + i += 1 + continue + + # If we're in a method to remove, skip until we find the next method or class + if in_method_to_remove: + current_indent = len(line) - len(line.lstrip()) + # If we find a line at the same or less indent (and it's not blank), we've exited the method + if line.strip() and current_indent <= method_indent: + in_method_to_remove = False + # Don't skip this line, process it normally + else: + i += 1 + continue + + # Add the line + new_lines.append(line) + i += 1 + + # Write output + with open(output_file, 'w') as f: + f.writelines(new_lines) + + print(f"Refactored {len(lines)} lines to {len(new_lines)} lines") + print(f"Removed {len(lines) - len(new_lines)} lines") + +if __name__ == '__main__': + main() diff --git a/refactor_ble_interface.py b/refactor_ble_interface.py new file mode 100644 index 0000000..9a8daac --- /dev/null +++ b/refactor_ble_interface.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +""" +Script to refactor BLEInterface.py to use the driver abstraction. + +This script performs automated transformations to remove platform-specific +code and replace it with driver abstraction calls. +""" + +import re + +def read_file(path): + with open(path, 'r') as f: + return f.read() + +def write_file(path, content): + with open(path, 'w') as f: + f.write(content) + +def refactor_imports(content): + """Remove platform-specific imports and add driver imports.""" + # Remove bleak imports + content = re.sub(r'# Check for bleak dependency.*?HAS_BLEAK = False\n', + '', content, flags=re.DOTALL) + + # Remove monkey patch code (lines 107-172 approximately) + content = re.sub(r'# ={70,}\n# Monkey patch.*?RNS\.log\(f"Failed to apply.*?\n', + '', content, flags=re.DOTALL) + + # Add driver imports after BLEFragmentation imports + driver_imports = ''' +# Import driver abstraction +try: + from bluetooth_driver import BLEDriverInterface, BLEDevice +except ImportError: + try: + from RNS.Interfaces.bluetooth_driver import BLEDriverInterface, BLEDevice + except ImportError: + # Fallback to root directory + import sys + import os + sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../../../'))) + from bluetooth_driver import BLEDriverInterface, BLEDevice + +# Import platform-specific driver +try: + from linux_bluetooth_driver import LinuxBluetoothDriver +except ImportError: + try: + from RNS.Interfaces.linux_bluetooth_driver import LinuxBluetoothDriver + except ImportError: + # Fallback to root directory + import sys + import os + sys.path.insert(0, os.path.abspath(os.path.join(os.path.dirname(__file__), '../../../'))) + from linux_bluetooth_driver import LinuxBluetoothDriver + +HAS_DRIVER = True +''' + + # Find BLEGATTServer import section and add driver imports after + content = re.sub( + r'(except ImportError:\s+HAS_GATT_SERVER = False)\n', + r'\1\n' + driver_imports + '\n', + content + ) + + return content + +def refactor_init(content): + """Refactor __init__ method to use driver.""" + # This is complex, will need manual editing + # For now, just remove the dependency check for bleak + content = re.sub( + r' # Check dependencies\s+if not HAS_BLEAK:.*?pip install bleak==1\.1\.1"\s+\)', + ' # Check dependencies\n if not HAS_DRIVER:\n raise ImportError(\n "BLEInterface requires the driver abstraction. "\n "Ensure bluetooth_driver.py and linux_bluetooth_driver.py are available."\n )', + content, + flags=re.DOTALL + ) + + return content + +def main(): + input_path = '/home/tyler/repos/public/ble-reticulum/src/RNS/Interfaces/BLEInterface.py' + output_path = input_path # Overwrite in place + + print(f"Reading {input_path}...") + content = read_file(input_path) + + print("Refactoring imports...") + content = refactor_imports(content) + + print("Refactoring __init__...") + content = refactor_init(content) + + print(f"Writing {output_path}...") + write_file(output_path, content) + + print("Done! Manual edits still required for:") + print(" - __init__ method (driver creation, callbacks)") + print(" - Remove async methods (_discover_peers, _connect_to_peer, etc.)") + print(" - Replace BLE operations with driver calls") + print(" - Add driver callback implementations") + +if __name__ == '__main__': + main() diff --git a/refactor_helper.py b/refactor_helper.py new file mode 100644 index 0000000..baa4012 --- /dev/null +++ b/refactor_helper.py @@ -0,0 +1,43 @@ +#!/usr/bin/env python3 +""" +Helper script to identify sections of BLEInterface.py that need refactoring. +""" + +with open('/home/tyler/repos/public/ble-reticulum/src/RNS/Interfaces/BLEInterface.py', 'r') as f: + lines = f.readlines() + +# Find methods to remove +methods_to_remove = [ + '_run_async_loop', + '_detect_bluez_version', + '_log_bluez_config', + '_connect_via_dbus_le', + '_get_local_adapter_address', + '_start_discovery', + '_start_server', + '_discover_peers', + '_connect_to_peer' +] + +print("Methods to remove:") +for method in methods_to_remove: + for i, line in enumerate(lines): + if f'def {method}' in line or f'async def {method}' in line: + print(f" Line {i+1}: {line.strip()}") + break + +# Find key sections +print("\nKey sections:") +for i, line in enumerate(lines): + if 'class DiscoveredPeer' in line: + print(f" DiscoveredPeer class: line {i+1}") + elif 'class BLEInterface' in line: + print(f" BLEInterface class: line {i+1}") + elif 'class BLEPeerInterface' in line: + print(f" BLEPeerInterface class: line {i+1}") + elif line.strip().startswith('def __init__(self, owner, configuration)'): + print(f" BLEInterface.__init__: line {i+1}") + elif '_score_peer' in line and 'def' in line: + print(f" _score_peer: line {i+1}") + elif '_handle_ble_data' in line and 'def' in line: + print(f" _handle_ble_data: line {i+1}") diff --git a/refactor_pass2.py b/refactor_pass2.py new file mode 100644 index 0000000..c58f6ad --- /dev/null +++ b/refactor_pass2.py @@ -0,0 +1,310 @@ +#!/usr/bin/env python3 +""" +Second pass refactoring: Replace remaining BLE operations with driver calls. +""" + +import re + +def read_file(path): + with open(path, 'r') as f: + return f.read() + +def write_file(path, content): + with open(path, 'w') as f: + f.write(content) + +def refactor_detach_method(content): + """Replace async operations in detach() with driver.stop().""" + + old_detach = r''' def detach\(self\): + """Detach and shutdown the interface\.""" + RNS\.log\(f"\{self\} detaching interface", RNS\.LOG_INFO\) + self\.online = False + + # MEDIUM #4: Graceful shutdown - wait for operations to complete before stopping event loop + + # Stop GATT server gracefully + if self\.gatt_server: + try: + future = asyncio\.run_coroutine_threadsafe\(self\.gatt_server\.stop\(\), self\.loop\) + future\.result\(timeout=5\.0\) # Wait for graceful shutdown + RNS\.log\(f"\{self\} GATT server stopped", RNS\.LOG_DEBUG\) + except Exception as e: + RNS\.log\(f"\{self\} error stopping GATT server: \{e\}", RNS\.LOG_ERROR\) + + # Disconnect all peers gracefully + disconnect_futures = \[\] + with self\.peer_lock: + for address, \(client, last_seen, mtu\) in list\(self\.peers\.items\(\)\): + try: + future = asyncio\.run_coroutine_threadsafe\(client\.disconnect\(\), self\.loop\) + disconnect_futures\.append\(\(address, future\)\) + except Exception as e: + RNS\.log\(f"\{self\} error scheduling disconnect for \{address\}: \{e\}", RNS\.LOG_ERROR\) + + self\.peers\.clear\(\) + + # Wait for all disconnections \(with timeout\) + for address, future in disconnect_futures: + try: + future\.result\(timeout=2\.0\) + RNS\.log\(f"\{self\} disconnected from \{address\}", RNS\.LOG_DEBUG\) + except Exception as e: + RNS\.log\(f"\{self\} disconnect timeout for \{address\}: \{e\}", RNS\.LOG_WARNING\) + + # Detach spawned interfaces + for peer_if in list\(self\.spawned_interfaces\.values\(\)\): + peer_if\.detach\(\) + self\.spawned_interfaces\.clear\(\) + + # Clear fragmentation state + with self\.frag_lock: + self\.fragmenters\.clear\(\) + self\.reassemblers\.clear\(\) + + # NOW safe to stop event loop \(all operations completed\) + if self\.loop: + self\.loop\.call_soon_threadsafe\(self\.loop\.stop\) + # Give it a moment to actually stop + time\.sleep\(0\.1\) + + RNS\.log\(f"\{self\} detached", RNS\.LOG_INFO\)''' + + new_detach = ''' def detach(self): + """Detach and shutdown the interface.""" + RNS.log(f"{self} detaching interface", RNS.LOG_INFO) + self.online = False + + # Detach spawned interfaces + for peer_if in list(self.spawned_interfaces.values()): + peer_if.detach() + self.spawned_interfaces.clear() + + # Clear fragmentation state + with self.frag_lock: + self.fragmenters.clear() + self.reassemblers.clear() + + # Stop the driver (handles graceful disconnection and cleanup) + try: + self.driver.stop() + RNS.log(f"{self} driver stopped", RNS.LOG_DEBUG) + except Exception as e: + RNS.log(f"{self} error stopping driver: {e}", RNS.LOG_ERROR) + + RNS.log(f"{self} detached", RNS.LOG_INFO)''' + + content = re.sub(old_detach, new_detach, content) + return content + +def refactor_send_methods(content): + """Replace asyncio operations in _send_via_central and _send_via_peripheral with driver.send().""" + + # Replace _send_via_peripheral + old_peripheral = r''' def _send_via_peripheral\(self, fragments\): + """ + Send fragments via GATT server notifications\. + + Args: + fragments: List of fragment bytes to send + + Returns: + bool: True if all fragments sent successfully, False otherwise + """ + if not self\.parent_interface\.gatt_server: + RNS\.log\(f"No GATT server available for \{self\.peer_name\}", RNS\.LOG_ERROR\) + return False + + for i, fragment in enumerate\(fragments\): + try: + # Schedule the async notification in the parent's event loop + future = asyncio\.run_coroutine_threadsafe\( + self\.parent_interface\.gatt_server\.send_notification\(fragment, self\.peer_address\), + self\.parent_interface\.loop + \) + + # Wait for completion \(with timeout\) + future\.result\(timeout=2\.0\) + + self\.txb \+= len\(fragment\) + self\.parent_interface\.txb \+= len\(fragment\) + + except Exception as e: + RNS\.log\(f"Failed to send notification \{i\+1\}/\{len\(fragments\)\} to \{self\.peer_name\}: \{e\}", RNS\.LOG_ERROR\) + return False + + return True''' + + new_peripheral = ''' def _send_via_peripheral(self, fragments): + """ + Send fragments via driver (peripheral mode uses notifications). + + Args: + fragments: List of fragment bytes to send + + Returns: + bool: True if all fragments sent successfully, False otherwise + """ + for i, fragment in enumerate(fragments): + try: + # Driver automatically handles notification vs write based on connection type + self.parent_interface.driver.send(self.peer_address, fragment) + + self.txb += len(fragment) + self.parent_interface.txb += len(fragment) + + except Exception as e: + RNS.log(f"Failed to send fragment {i+1}/{len(fragments)} to {self.peer_name}: {e}", RNS.LOG_ERROR) + return False + + return True''' + + content = re.sub(old_peripheral, new_peripheral, content) + + # Replace _send_via_central + old_central = r''' def _send_via_central\(self, fragments\): + """ + Send fragments via GATT characteristic write \(central mode\)\. + + Args: + fragments: List of fragment bytes to send + + Returns: + bool: True if all fragments sent successfully, False otherwise + """ + # Use stored central_client \(set at initialization for central connections\) + if not self\.central_client or not self\.central_client\.is_connected: + RNS\.log\(f"\{self\} peer \{self\.peer_name\} \(\{self\.peer_address\}\) not connected or disconnected", RNS\.LOG_WARNING\) + return False + + client = self\.central_client + + # Send each fragment via BLE characteristic write + for i, fragment in enumerate\(fragments\): + try: + # Schedule the async write in the parent's event loop + future = asyncio\.run_coroutine_threadsafe\( + client\.write_gatt_char\(BLEInterface\.CHARACTERISTIC_RX_UUID, fragment\), + self\.parent_interface\.loop + \) + + # Wait for completion \(with timeout\) + future\.result\(timeout=2\.0\) + + self\.txb \+= len\(fragment\) + self\.parent_interface\.txb \+= len\(fragment\) + + except asyncio\.TimeoutError: + RNS\.log\(f"\{self\} timeout sending fragment \{i\+1\}/\{len\(fragments\)\} to \{self\.peer_name\}, " + f"packet lost \(Reticulum will retransmit\)", RNS\.LOG_WARNING\) + return False + + # HIGH #3: Comprehensive asyncio exception handling + except \(asyncio\.CancelledError, RuntimeError\) as e: + RNS\.log\(f"\{self\} event loop error sending fragment \{i\+1\}/\{len\(fragments\)\}: " + f"\{type\(e\)\.__name__\}: \{e\}", RNS\.LOG_ERROR\) + # Mark interface as offline if event loop died + if isinstance\(e, RuntimeError\) and "closed" in str\(e\)\.lower\(\): + RNS\.log\(f"\{self\} event loop is closed, marking interface offline", RNS\.LOG_ERROR\) + self\.parent_interface\.online = False + return False + + except ConnectionError as e: + RNS\.log\(f"\{self\} connection lost to \{self\.peer_name\} while sending fragment \{i\+1\}/\{len\(fragments\)\}: " + f"\{type\(e\)\.__name__\}: \{e\}, packet lost", RNS\.LOG_WARNING\) + return False + + except Exception as e: + error_type = type\(e\)\.__name__ + RNS\.log\(f"\{self\} unexpected exception sending fragment \{i\+1\}/\{len\(fragments\)\} to \{self\.peer_name\}: " + f"\{error_type\}: \{e\}, packet lost \(Reticulum will retransmit\)", RNS\.LOG_WARNING\) + # If one fragment fails, the whole packet is lost + # Reticulum's upper layers will handle retransmission + return False + + return True''' + + new_central = ''' def _send_via_central(self, fragments): + """ + Send fragments via driver (central mode uses GATT writes). + + Args: + fragments: List of fragment bytes to send + + Returns: + bool: True if all fragments sent successfully, False otherwise + """ + # Check if peer is still connected + if self.peer_address not in self.parent_interface.driver.connected_peers: + RNS.log(f"{self} peer {self.peer_name} ({self.peer_address}) not connected", RNS.LOG_WARNING) + return False + + # Send each fragment via driver + for i, fragment in enumerate(fragments): + try: + # Driver automatically handles write vs notification based on connection type + self.parent_interface.driver.send(self.peer_address, fragment) + + self.txb += len(fragment) + self.parent_interface.txb += len(fragment) + + except ConnectionError as e: + RNS.log(f"{self} connection lost to {self.peer_name} while sending fragment {i+1}/{len(fragments)}: " + f"{type(e).__name__}: {e}, packet lost", RNS.LOG_WARNING) + return False + + except Exception as e: + error_type = type(e).__name__ + RNS.log(f"{self} unexpected exception sending fragment {i+1}/{len(fragments)} to {self.peer_name}: " + f"{error_type}: {e}, packet lost (Reticulum will retransmit)", RNS.LOG_WARNING) + return False + + return True''' + + content = re.sub(old_central, new_central, content) + return content + +def remove_stale_references(content): + """Remove or update stale references to self.loop, self.gatt_server, etc.""" + + # Remove _start_gatt_when_identity_ready method (replaced in pass 1) + pattern = r' def _start_gatt_when_identity_ready\(self\):.*?(?=\n def )' + content = re.sub(pattern, '', content, flags=re.DOTALL) + + # Remove remaining asyncio imports that aren't needed + # (Keep asyncio since it might still be imported elsewhere, but comment about driver ownership) + + # Update threading model docstring + content = content.replace( + ' THREADING MODEL:\n - Main asyncio loop in separate thread (_run_async_loop)\n - LOCK ORDERING CONVENTION (to prevent deadlocks):\n 1. peer_lock - ALWAYS acquire first for peer state access\n 2. frag_lock - THEN acquire for fragmentation state\n NEVER acquire locks in reverse order! (HIGH #2: deadlock prevention)\n - Uses asyncio.run_coroutine_threadsafe for cross-thread calls', + ' THREADING MODEL:\n - Driver owns async event loop in separate thread\n - LOCK ORDERING CONVENTION (to prevent deadlocks):\n 1. peer_lock - ALWAYS acquire first for peer state access\n 2. frag_lock - THEN acquire for fragmentation state\n NEVER acquire locks in reverse order! (HIGH #2: deadlock prevention)\n - Driver callbacks invoked from driver thread' + ) + + return content + +def main(): + input_file = 'src/RNS/Interfaces/BLEInterface.py' + + print("Reading file...") + content = read_file(input_file) + + print("Step 1: Refactoring detach() method...") + content = refactor_detach_method(content) + + print("Step 2: Refactoring send methods...") + content = refactor_send_methods(content) + + print("Step 3: Removing stale references...") + content = remove_stale_references(content) + + print("Writing refactored file...") + write_file(input_file, content) + + print("Done! Pass 2 complete.") + print("\nRemaining manual tasks:") + print(" - Verify all driver callbacks are correct") + print(" - Test the refactored interface") + print(" - Remove any remaining comments about bleak/bluezero") + +if __name__ == '__main__': + main() diff --git a/src/ble_reticulum/BLEInterface.py b/src/ble_reticulum/BLEInterface.py index eaf3fb9..7c4b1db 100644 --- a/src/ble_reticulum/BLEInterface.py +++ b/src/ble_reticulum/BLEInterface.py @@ -67,17 +67,14 @@ except NameError: if _interface_dir not in sys.path: sys.path.insert(0, _interface_dir) -# Import base Interface class -# When integrated into Reticulum, this will be: -# from ble_reticulum.Interface import Interface -# For now, we'll need to handle the import path +# Import base Interface class from Reticulum try: - from ble_reticulum.Interface import Interface + from RNS.Interfaces.Interface import Interface except ImportError: # Fallback for development import os sys.path.insert(0, os.path.join(os.path.dirname(__file__), '../../../')) - from ble_reticulum.Interface import Interface + from RNS.Interfaces.Interface import Interface # Import fragmentation module # Note: When loaded as external interface, use absolute imports