Microvisor Remote Debugging

Remote debugging allows you to use your development computer to probe your application while it is running on a Microvisor-based device — and to do so across the Internet. Your device may be on your desk in front of you, or installed at a test site on the other side of the world, but wherever your development device is situated, Microvisor remote debugging allows you to connect to it using the popular debugger GDB to set breakpoints, examine variables, view memory, and enumerate the call stack.

Debugging a Microvisor-enabled device remotely is functionally the same as working with the device connected directly to your development machine, and just as secure. Remote debugging sessions use end-to-end encryption using keys you generate. Your public key, passed to Microvisor within your application bundle, is used to encrypt application state data which is decrypted locally with your private key.

This guide will show you how to set up remote debugging and initiate a debugging session. For developers new to GDB, it also includes an introduction to the commands you will use to debug your app.

System setup

Open up a terminal and run the following command sequences.

1. Install the Twilio CLI

If you have not yet installed Twilio CLI, do so now. To install and set up the Twilio CLI on your development machine, please pop over to the Twilio CLI Quickstart and then jump back here when you’re done.

2. Install or update the Microvisor plugin

twilio plugins:install @twilio/plugin-microvisor

3. Install the ARM cross-platform version of GDB

To install the appropriate version of GDB, enter this command:

sudo apt install gdb-multiarch

4. Sign in to Twilio

If you haven’t already done so, run twilio login. You will be asked for your account SID and your account Auth Token, both of which you can call up from the Console if you don’t have them handy. Keep them nearby — you’ll need them again shortly. You will also be prompted to enter a ‘shorthand identifier’, a local ID for your credentials. Enter your initials or any other phrase.

5. Get your device’s SID

If you don’t yet know your device’s SID, the following Microvisor API request will return a list of all of your devices:

twilio api:microvisor:v1:devices:list

This request will return a list of all your devices, with their SIDs and, if set, unique names.

6. Set environment variables

Add the following to your shell profile:

export TWILIO_ACCOUNT_SID=ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export TWILIO_AUTH_TOKEN=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
export MV_DEVICE_SID=UVxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

To generate the API keys and secrets, visit Account > API keys & tokens in the Twilio Console.

7. Get the Microvisor Remote Debugging Demo

Clone our demo code repository:

git clone --recurse-submodules https://github.com/korewireless/Microvisor-Demo-Remote-Debug

8. Create remote debugging authorization keys

To protect your devices and any information flowing from them, public and private authorization keys are used to secure remote debugging sessions. You generate and store these keys. The public key is passed to Microvisor within your application bundle, and is used to encrypt application state and other debugging data before it is transmitted to you. Locally, the Twilio CLI Microvisor Plugin uses the private key to decrypt this data before passing it to the debugger.

Key creation is now part of the Twilio CLI Microvisor Plugin:

twilio microvisor:debug:generate_keypair

This will generate two files, debug-private-key.pem and debug-public-key.pem, in the current directory. You can specify the file names and locations of the generated files by including the options --debug-auth-privkey and --debug-auth-pubkey. For example:

twilio microvisor:debug:generate_keypair \
  --debug-auth-privkey=~/prv-key.pem \
  --debug-auth-pubkey=~/projects/keys/pub-key.pem

Store your two .pem files wherever you wish, but make a note of their location — you will need this information in the next section.

Start a debugging session

1. Build and deploy the application code

Switch to the microvisor-remote-debug-demo directory if you’re not already in it and run:

twilio microvisor:deploy . --devicesid $MV_DEVICE_SID --log \
  --privatekey </path/to/your/private/key.pem> \
  --publickey </path/to/your/public/key.pem>

The bundled script will build the demo app, upload it to the Twilio cloud, and assign it to your device.

2. Open the debugging connection

Open a new terminal tab or window and enter the following command:

twilio microvisor:debug $MV_DEVICE_SID </path/to/your/private/key.pem>

The private key is the debug-private-key.pem file you created earlier.

The command opens a secure channel between the device and the plugin, which operates as a GDB server. You’ll see the following line appear:

GDB server listening on port 8001

Keep this tab or window open. The plugin shouldn’t exit to the shell prompt unless you close it down manually, or GDB exits. If the plugin does exit to the shell outside of these circumstances, review the Troubleshooting section.

3. Run GDB

In a new, third terminal window, switch to the project directory and run GDB:

gdb-multiarch -l 10 ./build/demo/mv-remote-debug-demo.elf

You should see a line indicating that GDB is reading symbols from your application .elf file. The .elf file is output by the compilation process and is located alongside the files used to generated Microvisor App bundles.

Optionally, flip back to the window you opened in Step 2. You’ll see a line like:

Accepted connection from ::ffff:127.0.0.1:51254

This too shows that GDB and the Twilio cloud are now in communication.

Troubleshooting

If you see GDB report localhost:8001: Operation timed out, or the twilio command issues GDB server listening on port 8001 but then exits to the shell, then a connection could not be made to the device. Check that it is online. If it is, then the specified device is most likely running an application from an uploaded bundle that lacked a remote debugging authorization key. First try rebooting the device: it may not yet have received the updated code you uploaded.

If twilio is still exiting to the prompt, check that you correctly passed the public key .pem file to Bundler as shown above. If not, add it now, and upload the new bundle .zip file to the Twilio Cloud.

Latency

You should expect some latency between issuing commands to GDB and their outcomes being reported. Even if you are debugging a device close to you, your instructions are still relayed over your Internet connection to the Twilio Cloud and then back down to the device, and the responses returned along the same path.

As we continue to develop Microvisor’s remote debugging functionality within the Microvisor kernel, in the Twilio Cloud, and in Twilio’s desktop tools, we expect latency to improve, but remote debugging will never be as immediately responsive as debugging over a direct connection. However, this is a small price to pay for the flexibility of being able to debug Internet-connected devices wherever they may be located.

Multithreading

Microvisor has no awareness of, or sensitivity to, your RTOS’ threads, so thread-specific information can’t be relayed to GDB, and GDB can’t provide thread-level control. In this regard, GDB behaves as it your app stack — RTOS, app, middleware — is single-threaded.

Basic debugging

GDB is a powerful application, and providing a full guide to its workings is beyond of the scope of this short tutorial. If you’re new to GDB, here are some common operations you’re likely to want to perform. The commands that trigger them are all issued at the (gdb) prompt.

You can find the official GDB documentation here.

Breakpoints

Breakpoints allow you to halt your code at key points and inspect its state. While the application is suspended, you can view the current value of variables, and can even give them new values. This can help you understand why the application is behaving the way it is at that point, especially if it’s not behaving the way you expect.

You set a breakpoint by entering the b command followed by a line number. For example, b 82 sets a breakpoint at line 82 in your source code, and the application will halt when it reaches that line. If your application’s source code spans multiple files, you’ll need to include the file name too: for example, b main.c:82.

(gdb) b main.c:82
Breakpoint 1 at 0x800c128: file /home/smitty/microvisor-remote-debug-demo/demo/main.c, line 82.
Note: automatically using hardware breakpoints for read-only addresses.

You can also set a breakpoint at the start of a specific function with the form b <FUNCTION_NAME>.

To view the breakpoints you have set, use the info command: info breakpoints.

(gdb) info breakpoints
Num  Type        Disp Enb Address    What
1    breakpoint  keep y   0x0800c128 in main at /home/smitty/microvisor-remote-debug-demo/demo/main.c:82
     breakpoint already hit 2 times
2    breakpoint  keep y   0x0800c568 in http_process_response at /home/smitty/microvisor-remote-debug-demo/demo/main.c:335

When the application hits a breakpoint it will suspend automatically, but you can trigger suspension at any time manually: just hit ctrl-c, which outputs:

Program received signal SIGINT, Interrupt.

In both cases, you’ll then see the (gdb) prompt so you can enter commands. To continue running the application after suspension, whether you run other commands or not, enter the c command.

To restart the application entirely rather than continue from the suspension point, enter the r command.

Use the list command to print a few lines either side of the current one to help you see what’s going on.

When a breakpoint is hit, GDB will tell you which breakpoint triggered the suspension and print the current line:

Breakpoint 1, main () at /home/smitty/microvisor-remote-debug-demo/demo/main.c:82
82              debug_function_parent(&store);

The STM32U585 microcontroller has a limit of eight breakpoints, but GDB will not inform you of this when you enter your ninth breakpoint. Instead, you will be warned when you continue to run the application:

Warning:
Cannot insert hardware breakpoint 9.
Could not insert hardware breakpoints:
You may have requested too many hardware breakpoints/watchpoints.

Command aborted.

To remove a breakpoint, enter the clear command followed by the line number, file name and line number, or function name. Alternatively, you can enter d to delete all current breakpoints. GDB gives each new breakpoint a number and this number always increases in a session, even any or all breakpoints are deleted.

Backtracing

The bt command shows you the functions that were called, in sequence, to get the application to where it currently is. This is called a ‘backtrace’, and GDB presents it as a sequence of the frames added to the stack, one per function call. For example:

(gdb) bt
#0  debug_function_child (vptr=0x2007ff98) at /home/smitty/GitHub/microvisor-remote-debug-demo/demo/main.c:142
#1  0x08001630 in debug_function_parent (vptr=0x2007ffac) at /Users/tsmith/GitHub/microvisor-remote-debug-demo/demo/main.c:135
#2  0x08001506 in main () at /Users/tsmith/GitHub/microvisor-remote-debug-demo/demo/main.c:74

If you add full after bt, you’ll also get a readout of the variables that are currently in scope. For example:

(gdb) bt full
#0  debug_function_child (vptr=0x2007ff9c) at /home/smitty/GitHub/microvisor-remote-debug-demo/demo/main.c:158
No locals.
#1  0x080012c0 in debug_function_parent (vptr=0x2007ffac) at /home/smitty/GitHub/microvisor-remote-debug-demo/demo/main.c:147
        test_var = 43
#2  0x08001184 in main () at /home/smitty/GitHub/microvisor-remote-debug-demo/demo/main.c:82
        status = MV_STATUS_OKAY
        kill_tick = 0
        last_send_tick = 30000011
        last_led_flash_tick = 59906632
        tick = 60000013
        close_channel = false
        store = 43

Stepping through code

GDB allows you to execute your application a line at a time to see the effect each one has on the application’s state. To do so, suspend the application — manually or from a breakpoint — and enter the next command to execute the next line and then halt immediately afterwards.

If the next line to be executed contains a function call, next will call the function and stop at the next line in the current block. For example, in the following code, halted at line 82, next will halt at line 83:

82     debug_function_parent(&store);
83     printf("Debug test variable value: %lu\n", store);

This is called ‘stepping over’ the function. If, however, you want to stop at the very next line to be executed, which may well be within the called function itself, use the step command. This is called ‘stepping into’ the function and, following the example above, will cause the application to halt at line 146:

82     debug_function_parent(&store);
83     printf("Debug test variable value: %lu\n", store);
84
85     // No channel open? Try and send the temperature
86     if (http_handles.channel == 0 && http_open_channel()) {
. . .
145    void debug_function_parent(uint32_t* vptr) {
146      uint32_t test_var = *vptr;
147      debug_function_child(&test_var);
148      *vptr = test_var;
149    }

Step again. Line 147 calls another function. If you don’t care about what happens in debug_function_child() then you can use next to step over it. If you instead enter step, you’ll step into debug_function_child() and halt at the function’s first line.

A variation on step is stepi, which stops the application after the next machine code instruction, which is not necessarily the next line to be executed in the source code.

If you’ve stepped into a function and want to step back out of it, use GDB’s fin (for “finish”) command. This continues execution and has GDB halt again after the function has returned to its caller. It will also output any returned value. For example, if you’ve stepped into debug_function_parent() from main(), and then into debug_function_child(), the fin command will quickly get you back:

(gdb) fin
Run till exit from #0  debug_function_child (vptr=0x2007ff9c) at /home/smitty/microvisor-remote-debug-demo/demo/main.c:158
debug_function_parent (vptr=0x2007ffac) at /home/smitty/microvisor-remote-debug-demo/demo/main.c:148
148     *vptr = test_var;
Value returned is $1 = true
(gdb) fin
Run till exit from #0  debug_function_parent (vptr=0x2007ffac) at /home/smitty/microvisor-remote-debug-demo/demo/main.c:148
main () at /home/smitty/microvisor-remote-debug-demo/demo/main.c:83
83              printf("Debug test variable value: %lu\n", store);

Getting and setting variable values

When the application is suspended, you can view local and global variables’ current values with the p command — follow it with a variable name. This will print the value of the requested variable. For example:

(gdb) p store
$1 = 55

That’s for a scalar value; for structured data you’ll see the structure written out. For instance, if you set a breakpoint at line 335 and print resp_data when the app halts:

(gdb) p resp_data
$3 = {result = MV_HTTPRESULT_OK, status_code = 200, num_headers = 25, body_length = 99}

If the code you’re about to step through or run depends on the value of store, you can see how it reacts to different values by setting store’s value at this point. To do, use the set command:

set store=100

You can then call c or step/next to execute the code that comes after the breakpoint.

For example:

Breakpoint 1, main () at /home/smitty/microvisor-remote-debug-demo/demo/main.c:82
82              debug_function_parent(&store);
(gdb) p store
$1 = 113
(gdb) set store=200
(gdb) n
83              printf("Debug test variable value: %lu\n", store);
(gdb) p store
$2 = 201

You can set memory directly using GDB, which uses C-style operators to get a variable’s memory location (&) as a pointer and to dereference (*) such a pointer. For example:

(gdb) p tick
$1 = 900040
(gdb) p &tick
$2 = (uint32_t *) 0x20006360 <ucHeap+1000>
(gdb) set *((uint32_t *) 0x20006360) = 20
(gdb) p tick
$3 = 20

If the variable you’re printing is of a enum type, GDB will provide the correct constant value:

66     if (status == MV_STATUS_OKAY && tick - last_led_flash_tick > LED_FLASH_PERIOD_US) {
(gdb) p status
$1 = MV_STATUS_OKAY

Examine Registers

You can get GDB to output the contents of the host microcontroller’s registers with the info registers command:

(gdb) info registers
r0             0x1d97d             121213
r1             0x8000              32768
r2             0x19                25
r3             0x40005400          1073763328
r4             0x4040404           67372036
r5             0x5050505           84215045
r6             0x6060606           101058054
r7             0x200062e8          536896232
r8             0x8080808           134744072
r9             0x9090909           151587081
r10            0x10101010          269488144
r11            0x11111111          286331153
r12            0x7ff               2047
sp             0x200062c8          0x200062c8 <ucHeap+848>
lr             0x8003b35           134232885
pc             0x8003b72           0x8003b72 <I2C_WaitOnFlagUntilTimeout+92>
xpsr           0x29000000          687865856
primask        0x0                 0
basepri        0x0                 0
faultmask      0x0                 0
control        0x2                 2
msp            0x2007ffc8          537395144
psp            0x200062c8          536896200
msplim         0x0                 0
psplim         0x20005f80          536895360
sfsr           0x0                 0
sfar           0xe000ede8          -536810008
shcsr          0x0                 0
hfsr           0x0                 0
cfsr           0x0                 0
bfar           0xe000ed38          -536810184
mmfar          0x0                 0

Crashes

Application crashes will be reported by GDB if and when they occur. For example:

Program received signal SIGSEGV, Segmentation fault.
0x08001a30 in start_iot_task (argument=0x0) at /Users/tsmith/GitHub/microvisor-iot-device-demo/App/main.c:257
257                 *ptr = 45;

Use the r command to restart the application after checking variables, registers, and/or memory.

Exit debugging

To end a debugging session, halt the application with Ctrl-C and enter the q command. You’ll be asked if you wish to quit despite killing the application: enter y. GDB will exit to the command line, as will the Twilio CLI Microvisor plugin.

Supported GDB functions

The initial release of Microvisor’s remote debugging functionality supports the majority, but not all, of the commands provided by GDB. Further commands are expected to be added as Microvisor progresses through its Beta phases, but some commands are not expected to be supported.

Not all features can be supported. For example, Microvisor has no awareness of or sensitivity to your RTOS’ threads, so thread-specific information can’t be relayed to GDB, and GDB can’t provide thread-level control. This is also true for other embedded debuggers that have not been extended with RTOS-specific knowledge.

• Supported now
  • Single-threaded debugging.
  • Setting and removing breakpoints (b, clear, delete).
  • Step in (s).
  • Step over (n).
  • Halting (Ctrl-C).
  • Continuing (c).
  • Printing/setting variables.
  • Reading from/writing to RAM (x).
  • Printing the backtrace (bt).
  • Getting/setting registers.
• Support planned
  • monitor commands, in particular monitor reset and monitor reset halt.
• Not expected to be supported
  • Non-stop mode.
  • Multiprocessing/multithreading control/debugging.
  • The load command.

Using Visual Studio Code

Microsoft’s Visual Studio Code (aka VSCode) is a popular editor and IDE, and can be configured to provide a place to work on code and host remote debugging sessions. The remote debug demo repo can be used to sample remote debugging in VSCode, but you will need to perform a few setup tasks first.

VSCode setup

1. Download and install VSCode.
2. Run VSCode and click on the Extensions icon (the four squares) in the left-hand panel.
3. Install the following extensions. You can use the search field at the top of the EXTENSIONS column to jump straight to each one:
   • C/C++ by Microsoft.
   • CMake by twxs.
   • CMake Tools by Microsoft.
   • Native Debug by WebFreak.
     

     

4. Quit VSCode.

Remote debugging

The remote debug demo repo is set up to support remote debugging in VSCode: just open the repo’s folder on your desktop and double-click the mv-remote-debug-demo.code-workspace file to launch VSCode and load in the config files you need.

1. Click on the Run and Debug icon in VSCode’s left-hand panel:
   

   

2. At the top of the Run and Debug column, you’ll see Microvisor Remote. Click the play icon immediately to the left of this:
   

   

3. VSCode will show a Terminal in which you’ll see debug keys being generated and then the app being built and deployed. There’s a short delay between uploading and deployment, so we pause for 30s.
4. You should now see the usual Accepted connection from ::ffff:127.0.0.1:51254:
   

   

5. Switch to the DEBUG CONSOLE tab. Use the control bar at the top of the window to pause the program flow, step over or into functions, and continue running the code.
   

   

6. Set breakpoints by clicking alongside the line number in the main code pane. When a breakpoint is hit, local variables are listed in the Run and Debug column.
   

   

How to use VSCode with your own apps

To run remore debugging sessions for your own Microvisor applications under VSCode, you need to create a .vscode directory in your own code repo then copy two files, launch.json and tasks.json, to this directory from the remote debug demo repo.

launch.json provides VSCode with the start-up information it needs when you initiate a remote debugging session (as in Step 2, above). The config file indicates which debugger to use and how to configure it. To avoid errors, the file explicitly tells gdb-multiarch to ignore local and other .gdbinit files (the --nx switch under debugger_args).

tasks.json defines a series of pre-debugging actions that need to be performed in the order specified by the dependsOn key. These tasks are referenced by launch.json’s preLaunchTask key. The tasks check the tools you need have been installed, generate debugging keys in your home directory, build the app and deploy to be debugged, allow time for deployment, and finally initiate the GDB server.

Most of these tasks call out to the Twilio CLI Microvisor Plugin; one, build_app, uses the deploy.sh script so you will need to edit this task if you use an alternative build script.