Real-time application monitors

Description

Real-time applications may have design flaws such that they experience unexpected latency and fail to meet their time requirements. Often, these flaws follow a few patterns:

  • Page faults: A real-time thread may access memory that does not have a mapped physical backing or must first be copied (such as for copy-on-write). Thus a page fault is raised and the kernel must first perform the expensive action. This causes significant delays to the real-time thread

  • Priority inversion: A real-time thread blocks waiting for a lower-priority thread. This causes the real-time thread to effectively take on the scheduling priority of the lower-priority thread. For example, the real-time thread needs to access a shared resource that is protected by a non-pi-mutex, but the mutex is currently owned by a non-real-time thread.

The rtapp monitor detects these patterns. It aids developers to identify reasons for unexpected latency with real-time applications. It is a container of multiple sub-monitors described in the following sections.

Monitor pagefault

The pagefault monitor reports real-time tasks raising page faults. Its specification is:

RULE = always (RT imply not PAGEFAULT)

To fix warnings reported by this monitor, mlockall() or mlock() can be used to ensure physical backing for memory.

This monitor may have false negatives because the pages used by the real-time threads may just happen to be directly available during testing. To minimize this, the system can be put under memory pressure (e.g. invoking the OOM killer using a program that does ptr = malloc(SIZE_OF_RAM); memset(ptr, 0, SIZE_OF_RAM);) so that the kernel executes aggressive strategies to recycle as much physical memory as possible.

Monitor sleep

The sleep monitor reports real-time threads sleeping in a manner that may cause undesirable latency. Real-time applications should only put a real-time thread to sleep for one of the following reasons:

  • Cyclic work: real-time thread sleeps waiting for the next cycle. For this case, only the clock_nanosleep syscall should be used with TIMER_ABSTIME (to avoid time drift) and CLOCK_MONOTONIC (to avoid the clock being changed). No other method is safe for real-time. For example, threads waiting for timerfd can be woken by softirq which provides no real-time guarantee.

  • Real-time thread waiting for something to happen (e.g. another thread releasing shared resources, or a completion signal from another thread). In this case, only futexes (FUTEX_LOCK_PI, FUTEX_LOCK_PI2 or one of FUTEX_WAIT_*) should be used. Applications usually do not use futexes directly, but use PI mutexes and PI condition variables which are built on top of futexes. Be aware that the C library might not implement conditional variables as safe for real-time. As an alternative, the librtpi library exists to provide a conditional variable implementation that is correct for real-time applications in Linux.

Beside the reason for sleeping, the eventual waker should also be real-time-safe. Namely, one of:

  • An equal-or-higher-priority thread

  • Hard interrupt handler

  • Non-maskable interrupt handler

This monitor’s warning usually means one of the following:

  • Real-time thread is blocked by a non-real-time thread (e.g. due to contention on a mutex without priority inheritance). This is priority inversion.

  • Time-critical work waits for something which is not safe for real-time (e.g. timerfd).

  • The work executed by the real-time thread does not need to run at real-time priority at all. This is not a problem for the real-time thread itself, but it is potentially taking the CPU away from other important real-time work.

Application developers may purposely choose to have their real-time application sleep in a way that is not safe for real-time. It is debatable whether that is a problem. Application developers must analyze the warnings to make a proper assessment.

The monitor’s specification is:

RULE = always ((RT and SLEEP) imply (RT_FRIENDLY_SLEEP or ALLOWLIST))

RT_FRIENDLY_SLEEP = (RT_VALID_SLEEP_REASON or KERNEL_THREAD)
                and ((not WAKE) until RT_FRIENDLY_WAKE)

RT_VALID_SLEEP_REASON = FUTEX_WAIT
                     or RT_FRIENDLY_NANOSLEEP

RT_FRIENDLY_NANOSLEEP = CLOCK_NANOSLEEP
                    and NANOSLEEP_TIMER_ABSTIME
                    and NANOSLEEP_CLOCK_MONOTONIC

RT_FRIENDLY_WAKE = WOKEN_BY_EQUAL_OR_HIGHER_PRIO
                or WOKEN_BY_HARDIRQ
                or WOKEN_BY_NMI
                or KTHREAD_SHOULD_STOP

ALLOWLIST = BLOCK_ON_RT_MUTEX
         or FUTEX_LOCK_PI
         or TASK_IS_RCU
         or TASK_IS_MIGRATION

Beside the scenarios described above, this specification also handle some special cases:

  • KERNEL_THREAD: kernel tasks do not have any pattern that can be recognized as valid real-time sleeping reasons. Therefore sleeping reason is not checked for kernel tasks.

  • KTHREAD_SHOULD_STOP: a non-real-time thread may stop a real-time kernel thread by waking it and waiting for it to exit (kthread_stop()). This wakeup is safe for real-time.

  • ALLOWLIST: to handle known false positives with the kernel.

  • BLOCK_ON_RT_MUTEX is included in the allowlist due to its implementation. In the release path of rt_mutex, a boosted task is de-boosted before waking the rt_mutex’s waiter. Consequently, the monitor may see a real-time-unsafe wakeup (e.g. non-real-time task waking real-time task). This is actually real-time-safe because preemption is disabled for the duration.

  • FUTEX_LOCK_PI is included in the allowlist for the same reason as BLOCK_ON_RT_MUTEX.