.. SPDX-License-Identifier: GPL-2.0

==============
Devlink Health
==============

Background
==========

The ``devlink`` health mechanism is targeted for Real Time Alerting, in
order to know when something bad happened to a PCI device.

  * Provide alert debug information.
  * Self healing.
  * If problem needs vendor support, provide a way to gather all needed
    debugging information.

Overview
========

The main idea is to unify and centralize driver health reports in the
generic ``devlink`` instance and allow the user to set different
attributes of the health reporting and recovery procedures.

The ``devlink`` health reporter:
Device driver creates a "health reporter" per each error/health type.
Error/Health type can be a known/generic (eg pci error, fw error, rx/tx error)
or unknown (driver specific).
For each registered health reporter a driver can issue error/health reports
asynchronously. All health reports handling is done by ``devlink``.
Device driver can provide specific callbacks for each "health reporter", e.g.:

  * Recovery procedures
  * Diagnostics procedures
  * Object dump procedures
  * OOB initial parameters

Different parts of the driver can register different types of health reporters
with different handlers.

Actions
=======

Once an error is reported, devlink health will perform the following actions:

  * A log is being send to the kernel trace events buffer
  * Health status and statistics are being updated for the reporter instance
  * Object dump is being taken and saved at the reporter instance (as long as
    there is no other dump which is already stored)
  * Auto recovery attempt is being done. Depends on:
    - Auto-recovery configuration
    - Grace period vs. time passed since last recover

User Interface
==============

User can access/change each reporter's parameters and driver specific callbacks
via ``devlink``, e.g per error type (per health reporter):

  * Configure reporter's generic parameters (like: disable/enable auto recovery)
  * Invoke recovery procedure
  * Run diagnostics
  * Object dump

.. list-table:: List of devlink health interfaces
   :widths: 10 90

   * - Name
     - Description
   * - ``DEVLINK_CMD_HEALTH_REPORTER_GET``
     - Retrieves status and configuration info per DEV and reporter.
   * - ``DEVLINK_CMD_HEALTH_REPORTER_SET``
     - Allows reporter-related configuration setting.
   * - ``DEVLINK_CMD_HEALTH_REPORTER_RECOVER``
     - Triggers a reporter's recovery procedure.
   * - ``DEVLINK_CMD_HEALTH_REPORTER_DIAGNOSE``
     - Retrieves diagnostics data from a reporter on a device.
   * - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_GET``
     - Retrieves the last stored dump. Devlink health
       saves a single dump. If an dump is not already stored by the devlink
       for this reporter, devlink generates a new dump.
       dump output is defined by the reporter.
   * - ``DEVLINK_CMD_HEALTH_REPORTER_DUMP_CLEAR``
     - Clears the last saved dump file for the specified reporter.

The following diagram provides a general overview of ``devlink-health``::

                                                   netlink
                                          +--------------------------+
                                          |                          |
                                          |            +             |
                                          |            |             |
                                          +--------------------------+
                                                       |request for ops
                                                       |(diagnose,
     mlx5_core                             devlink     |recover,
                                                       |dump)
    +--------+                            +--------------------------+
    |        |                            |    reporter|             |
    |        |                            |  +---------v----------+  |
    |        |   ops execution            |  |                    |  |
    |     <----------------------------------+                    |  |
    |        |                            |  |                    |  |
    |        |                            |  + ^------------------+  |
    |        |                            |    | request for ops     |
    |        |                            |    | (recover, dump)     |
    |        |                            |    |                     |
    |        |                            |  +-+------------------+  |
    |        |     health report          |  | health handler     |  |
    |        +------------------------------->                    |  |
    |        |                            |  +--------------------+  |
    |        |     health reporter create |                          |
    |        +---------------------------->                          |
    +--------+                            +--------------------------+