Microvisor Notifications

Many Microvisor system calls trigger asynchronous operations: the actions requested by the application take a finite but unknown length of time to complete. For this reason, the Microvisor notification system is used to inform the application when the requested operation has been completed. The notification will include the outcome of the action: whether it succeeded or failed.

Notifications are also used to inform the application when certain key system events take place. The application must register its interest in these events and will then receive notifications if and when they take place. Unlike action completion notifications, status notifications do not come in response to a specific operation requested by the application.

Microvisor dispatches notifications through one or more notification centers. You can establish as many centers as you need: you are limited only by the amount of non-secure memory available to hold the centers' notification-storage buffers. Buffers are sized and allocated for this purpose by the application, and referenced as a pointer during center setup. Each notification center is identified by its own handle.

The memory allocated to receive notifications comprises contiguous bytes but they used as backing for a circular buffer. Each notification center maintains a write pointer to the next entry in its buffer, and when this pointer reaches past the final allocated byte, it is automatically moved back to the start of the buffer, as shown below:

Expand image

With a buffer capable of holding n notifications, Microvisor at some point in the application lifecycle writes a sequence of four 16-byte notifications (red squares 1—4). Because of the circularity of the buffer, notifications 3 and 4 are written at the start of the buffer. The write pointer is set to record the next record (first grey square)

After it has written a notification, the notification center raises an interrupt to signal to the application that a new notification has been posted. When it asks Microvisor to create a notification center, the application provides a non-secure interrupt line's IRQ number. The application may process notifications in its interrupt service routine, or it can choose to defer notification processing to its main run loop by setting a flag or some other record.

To begin receiving notifications, your code first creates a notification center. It calls mvSetupNotifications() and passes a reference to a notification setup structure which contains the information the notification center will need to dispatch notifications to the application:

1
struct MvNotificationSetup {
2
  uint32_t irq;
3
  struct MvNotification *buffer;
4
  uint32_t buffer_size;
5
}

The application can create as many notification centers as it requires, but each one must specify a unique, valid IRQ.

In addition to a NotificationSetup structure, your code also passes in a pointer to four bytes of memory into which Microvisor will write the new notification center's unique handle. System calls that respond with notifications require you to supply the handle of the notification center to which you would like them to dispatch their notifications.

The value of irq in the NotificationSetup structure is the number of the non-secure interrupt line that will be triggered to signal that a new notification has been posted. The call to mvSetupNotifications() will return an error if this value indicates a secure interrupt, or has already been assigned to a notification center.

To receive any kind of notification, the application must allocate memory into which the notification center will write the notifications it posts. All notifications are 16 bytes in size, and the application is expected to allocate space for at least two of them, i.e. 32 bytes in size minimum. However large each buffer is, it must be a multiple of 16 bytes. The first byte of the notification buffer must be aligned to an eight-byte boundary. The buffer must be valid for the lifetime of the notification center that uses it. Each notification center you create must have it own buffer — you cannot share a buffer among multiple notification centers.

1
// Central store for HTTP request management notification records.
2
// Holds HTTP_NT_BUFFER_SIZE_R records at a time -- each record is 16 bytes in size.
3
volatile struct MvNotification http_notification_center[HTTP_NT_BUFFER_SIZE_R] __attribute__((aligned(8)));

Microvisor imposes no other restrictions on a notification buffer.

The application may use one notification center or many, but each one has its own, unique buffer. Some applications will be able to make do with a single notification center which is used (by supplying its handle) by every relevant System Call issued. You use a notification's tag to link action to outcome. Or your might prefer one center for each type of System Call you make. The choice is yours.

Expand image

Applications can have as many or as few notification centers as they require, sharing them across call types, or using them on a per call type basis. But each center must have its own buffer

The application should maintain per buffer read pointers that references where the next notification to be read from a given buffer is located. Microvisor doesn't provide read pointers; this is the responsibility of the application. Because the application may choose to defer reading notifications, Microvisor provides a protocol that allows the application to detect when it has reached the end of a sequence of notifications. Each notification has a four-byte event_type field, and Microvisor zeros this field in the space after the notification it has just written. When the application reads a notification of type zero, it knows it has now read all currently available notifications.

If the application doesn't read a notification in its interrupt service routine, or the interrupt is ignored — i.e., it is masked, has too low a priority, or has been disabled — there is a risk that the notification center's write pointer will overtake the application's read pointer: a buffer overrun. The application can detect this, if it wishes, by zeroing the event_type of notifications as they are read, and on each interrupt checking that the entry prior to the read pointer has been zeroed.

Each notification has the following 16-byte structure:

1
struct MvNotification { 
2
  uint64_t microseconds;
3
  enum MvEventType event_type;
4
  uint32_t tag; 
5
}

The value of microseconds is the value of the system's monotonic microsecond counter when the notification was written. Use this to measure and assess the duration of key operations.

The event_type field indicates what event or action caused the notification to be issued, useful especially if your application uses a single buffer for a range of event types. Your code can check if this value is zero when it is iterating over a number of notifications: an event_type of zero indicates it has now read all the currently available notifications, and its read pointer is placed ready for the next notification to be posted. It can also be used to detect buffer overruns. The available event types are listed under each system call that initiates notifications.

The value of tag will match the tag set during the call that triggered the dispatch of the notification.

To help applications keep track of which notification was the result of what specific action, an application can provide a tag whenever it calls an System function which will result in notifications being dispatched

A tag is an arbitrary 32-bit value, which can be a pointer, a reference to an object, an index in a table, or whatever the application chooses.

The tag is set when the application requests an operation that may issue notifications. Microvisor makes no use of tags itself: it stores them at registration, and the center includes it in any relevant notifications it dispatches subsequently.

Tags are tied to operations or events, so even if a notification center is dispatching notifications for a variety of operations, notifications will always include the correct tag.