Running tests with kunit_tool¶
We can either run KUnit tests using kunit_tool or can run tests manually, and then use kunit_tool to parse the results. To run tests manually, see: Run Tests without kunit_tool. As long as we can build the kernel, we can run KUnit.
kunit_tool is a Python script which configures and builds a kernel, runs tests, and formats the test results.
Run command:
./tools/testing/kunit/kunit.py run
We should see the following:
Configuring KUnit Kernel ...
Building KUnit kernel...
Starting KUnit kernel...
We may want to use the following options:
./tools/testing/kunit/kunit.py run --timeout=30 --jobs=`nproc --all`
--timeout
sets a maximum amount of time for tests to run.--jobs
sets the number of threads to build the kernel.
kunit_tool will generate a .kunitconfig
with a default
configuration, if no other .kunitconfig
file exists
(in the build directory). In addition, it verifies that the
generated .config
file contains the CONFIG
options in the
.kunitconfig
.
It is also possible to pass a separate .kunitconfig
fragment to
kunit_tool. This is useful if we have several different groups of
tests we want to run independently, or if we want to use pre-defined
test configs for certain subsystems.
To use a different .kunitconfig
file (such as one
provided to test a particular subsystem), pass it as an option:
./tools/testing/kunit/kunit.py run --kunitconfig=fs/ext4/.kunitconfig
To view kunit_tool flags (optional command-line arguments), run:
./tools/testing/kunit/kunit.py run --help
Creating a .kunitconfig
file¶
If we want to run a specific set of tests (rather than those listed
in the KUnit defconfig
), we can provide Kconfig options in the
.kunitconfig
file. For default .kunitconfig, see:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/tools/testing/kunit/configs/default.config.
A .kunitconfig
is a minconfig
(a .config
generated by running make savedefconfig
), used for running a
specific set of tests. This file contains the regular Kernel configs
with specific test targets. The .kunitconfig
also
contains any other config options required by the tests (For example:
dependencies for features under tests, configs that enable/disable
certain code blocks, arch configs and so on).
To create a .kunitconfig
, using the KUnit defconfig
:
cd $PATH_TO_LINUX_REPO
cp tools/testing/kunit/configs/default.config .kunit/.kunitconfig
We can then add any other Kconfig options. For example:
CONFIG_LIST_KUNIT_TEST=y
kunit_tool ensures that all config options in .kunitconfig
are
set in the kernel .config
before running the tests. It warns if we
have not included the options dependencies.
Note
Removing something from the .kunitconfig
will
not rebuild the .config file
. The configuration is only
updated if the .kunitconfig
is not a subset of .config
.
This means that we can use other tools
(For example: make menuconfig
) to adjust other config options.
The build dir needs to be set for make menuconfig
to
work, therefore by default use make O=.kunit menuconfig
.
Configuring, building, and running tests¶
If we want to make manual changes to the KUnit build process, we
can run part of the KUnit build process independently.
When running kunit_tool, from a .kunitconfig
, we can generate a
.config
by using the config
argument:
./tools/testing/kunit/kunit.py config
To build a KUnit kernel from the current .config
, we can use the
build
argument:
./tools/testing/kunit/kunit.py build
If we already have built UML kernel with built-in KUnit tests, we
can run the kernel, and display the test results with the exec
argument:
./tools/testing/kunit/kunit.py exec
The run
command discussed in section: Running tests with kunit_tool,
is equivalent to running the above three commands in sequence.
Parsing test results¶
KUnit tests output displays results in TAP (Test Anything Protocol)
format. When running tests, kunit_tool parses this output and prints
a summary. To see the raw test results in TAP format, we can pass the
--raw_output
argument:
./tools/testing/kunit/kunit.py run --raw_output
If we have KUnit results in the raw TAP format, we can parse them and
print the human-readable summary with the parse
command for
kunit_tool. This accepts a filename for an argument, or will read from
standard input.
# Reading from a file
./tools/testing/kunit/kunit.py parse /var/log/dmesg
# Reading from stdin
dmesg | ./tools/testing/kunit/kunit.py parse
Filtering tests¶
By passing a bash style glob filter to the exec
or run
commands, we can run a subset of the tests built into a kernel . For
example: if we only want to run KUnit resource tests, use:
./tools/testing/kunit/kunit.py run 'kunit-resource*'
This uses the standard glob format with wildcard characters.
Running tests on QEMU¶
kunit_tool supports running tests on qemu as well as via UML. To run tests on qemu, by default it requires two flags:
--arch
: Selects a configs collection (Kconfig, qemu config options and so on), that allow KUnit tests to be run on the specified architecture in a minimal way. The architecture argument is same as the option name passed to theARCH
variable used by Kbuild. Not all architectures currently support this flag, but we can use--qemu_config
to handle it. Ifum
is passed (or this flag is ignored), the tests will run via UML. Non-UML architectures, for example: i386, x86_64, arm and so on; run on qemu.--cross_compile
: Specifies the Kbuild toolchain. It passes the same argument as passed to theCROSS_COMPILE
variable used by Kbuild. As a reminder, this will be the prefix for the toolchain binaries such as GCC. For example:sparc64-linux-gnu
if we have the sparc toolchain installed on our system.$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux
if we have downloaded the microblaze toolchain from the 0-day website to a directory in our home directory called toolchains.
This means that for most architectures, running under qemu is as simple as:
./tools/testing/kunit/kunit.py run --arch=x86_64
When cross-compiling, we'll likely need to specify a different toolchain, for example:
./tools/testing/kunit/kunit.py run \
--arch=s390 \
--cross_compile=s390x-linux-gnu-
If we want to run KUnit tests on an architecture not supported by
the --arch
flag, or want to run KUnit tests on qemu using a
non-default configuration; then we can write our own``QemuConfig``.
These QemuConfigs
are written in Python. They have an import line
from..qemu_config import QemuArchParams
at the top of the file.
The file must contain a variable called QEMU_ARCH
that has an
instance of QemuArchParams
assigned to it. See example in:
tools/testing/kunit/qemu_configs/x86_64.py
.
Once we have a QemuConfig
, we can pass it into kunit_tool,
using the --qemu_config
flag. When used, this flag replaces the
--arch
flag. For example: using
tools/testing/kunit/qemu_configs/x86_64.py
, the invocation appear
as
./tools/testing/kunit/kunit.py run \
--timeout=60 \
--jobs=12 \
--qemu_config=./tools/testing/kunit/qemu_configs/x86_64.py
Running command-line arguments¶
kunit_tool has a number of other command-line arguments which can be useful for our test environment. Below are the most commonly used command line arguments:
--help
: Lists all available options. To list common options, place--help
before the command. To list options specific to that command, place--help
after the command.Note
Different commands (
config
,build
,run
, etc) have different supported options.--build_dir
: Specifies kunit_tool build directory. It includes the.kunitconfig
,.config
files and compiled kernel.--make_options
: Specifies additional options to pass to make, when compiling a kernel (usingbuild
orrun
commands). For example: to enable compiler warnings, we can pass--make_options W=1
.--alltests
: Enable a predefined set of options in order to build as many tests as possible.Note
The list of enabled options can be found in
tools/testing/kunit/configs/all_tests.config
.If you only want to enable all tests with otherwise satisfied dependencies, instead add
CONFIG_KUNIT_ALL_TESTS=y
to your.kunitconfig
.--kunitconfig
: Specifies the path or the directory of the.kunitconfig
file. For example:lib/kunit/.kunitconfig
can be the path of the file.lib/kunit
can be the directory in which the file is located.
This file is used to build and run with a predefined set of tests and their dependencies. For example, to run tests for a given subsystem.
--kconfig_add
: Specifies additional configuration options to be appended to the.kunitconfig
file. For example:./tools/testing/kunit/kunit.py run --kconfig_add CONFIG_KASAN=y
--arch
: Runs tests on the specified architecture. The architecture argument is same as the Kbuild ARCH environment variable. For example, i386, x86_64, arm, um, etc. Non-UML architectures run on qemu. Default is um.--cross_compile
: Specifies the Kbuild toolchain. It passes the same argument as passed to theCROSS_COMPILE
variable used by Kbuild. This will be the prefix for the toolchain binaries such as GCC. For example:sparc64-linux-gnu-
if we have the sparc toolchain installed on our system.$HOME/toolchains/microblaze/gcc-9.2.0-nolibc/microblaze-linux/bin/microblaze-linux
if we have downloaded the microblaze toolchain from the 0-day website to a specified path in our home directory called toolchains.
--qemu_config
: Specifies the path to a file containing a custom qemu architecture definition. This should be a python file containing a QemuArchParams object.--qemu_args
: Specifies additional qemu arguments, for example,-smp 8
.--jobs
: Specifies the number of jobs (commands) to run simultaneously. By default, this is set to the number of cores on your system.--timeout
: Specifies the maximum number of seconds allowed for all tests to run. This does not include the time taken to build the tests.--kernel_args
: Specifies additional kernel command-line arguments. May be repeated.--run_isolated
: If set, boots the kernel for each individual suite/test. This is useful for debugging a non-hermetic test, one that might pass/fail based on what ran before it.--raw_output
: If set, generates unformatted output from kernel. Possible options are:all
: To view the full kernel output, use--raw_output=all
.kunit
: This is the default option and filters to KUnit output. Use--raw_output
or--raw_output=kunit
.
--json
: If set, stores the test results in a JSON format and prints to stdout or saves to a file if a filename is specified.