>sphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget6/translations/zh_CN/virt/kvm/x86/running-nested-guestsmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/zh_TW/virt/kvm/x86/running-nested-guestsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/it_IT/virt/kvm/x86/running-nested-guestsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/ja_JP/virt/kvm/x86/running-nested-guestsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/ko_KR/virt/kvm/x86/running-nested-guestsmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget6/translations/sp_SP/virt/kvm/x86/running-nested-guestsmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h SPDX-License-Identifier: GPL-2.0h]h SPDX-License-Identifier: GPL-2.0}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhP/var/lib/git/docbuild/linux/Documentation/virt/kvm/x86/running-nested-guests.rsthKubhsection)}(hhh](htitle)}(hRunning nested guests with KVMh]hRunning nested guests with KVM}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh paragraph)}(hA nested guest is the ability to run a guest inside another guest (it can be KVM-based or a different hypervisor). The straightforward example is a KVM guest that in turn runs on a KVM guest (the rest of this document is built on this example)::h]hA nested guest is the ability to run a guest inside another guest (it can be KVM-based or a different hypervisor). The straightforward example is a KVM guest that in turn runs on a KVM guest (the rest of this document is built on this example):}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh literal_block)}(hX+ .----------------. .----------------. | | | | | L2 | | L2 | | (Nested Guest) | | (Nested Guest) | | | | | |----------------'--'----------------| | | | L1 (Guest Hypervisor) | | KVM (/dev/kvm) | | | .------------------------------------------------------. | L0 (Host Hypervisor) | | KVM (/dev/kvm) | |------------------------------------------------------| | Hardware (with virtualization extensions) | '------------------------------------------------------'h]hX+ .----------------. .----------------. | | | | | L2 | | L2 | | (Nested Guest) | | (Nested Guest) | | | | | |----------------'--'----------------| | | | L1 (Guest Hypervisor) | | KVM (/dev/kvm) | | | .------------------------------------------------------. | L0 (Host Hypervisor) | | KVM (/dev/kvm) | |------------------------------------------------------| | Hardware (with virtualization extensions) | '------------------------------------------------------'}hhsbah}(h]h ]h"]h$]h&]hhuh1hhhhK hhhhubh)}(h Terminology:h]h Terminology:}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh bullet_list)}(hhh](h list_item)}(h1L0 – level-0; the bare metal host, running KVM h]h)}(h0L0 – level-0; the bare metal host, running KVMh]h0L0 – level-0; the bare metal host, running KVM}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhhubah}(h]h ]h"]h$]h&]uh1hhhhhhhhNubh)}(hvL1 – level-1 guest; a VM running on L0; also called the "guest hypervisor", as it itself is capable of running KVM. h]h)}(huL1 – level-1 guest; a VM running on L0; also called the "guest hypervisor", as it itself is capable of running KVM.h]hyL1 – level-1 guest; a VM running on L0; also called the “guest hypervisor”, as it itself is capable of running KVM.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK!hjubah}(h]h ]h"]h$]h&]uh1hhhhhhhhNubh)}(hEL2 – level-2 guest; a VM running on L1, this is the "nested guest" h]h)}(hDL2 – level-2 guest; a VM running on L1, this is the "nested guest"h]hHL2 – level-2 guest; a VM running on L1, this is the “nested guest”}(hj2hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK$hj.ubah}(h]h ]h"]h$]h&]uh1hhhhhhhhNubeh}(h]h ]h"]h$]h&]bullet-uh1hhhhKhhhhubhnote)}(hX.The above diagram is modelled after the x86 architecture; s390x, ppc64 and other architectures are likely to have a different design for nesting. For example, s390x always has an LPAR (LogicalPARtition) hypervisor running on bare metal, adding another layer and resulting in at least four levels in a nested setup — L0 (bare metal, running the LPAR hypervisor), L1 (host hypervisor), L2 (guest hypervisor), L3 (nested guest). This document will stick with the three-level terminology (L0, L1, and L2) for all architectures; and will largely focus on x86.h](h)}(hThe above diagram is modelled after the x86 architecture; s390x, ppc64 and other architectures are likely to have a different design for nesting.h]hThe above diagram is modelled after the x86 architecture; s390x, ppc64 and other architectures are likely to have a different design for nesting.}(hjThhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK&hjPubh)}(hXFor example, s390x always has an LPAR (LogicalPARtition) hypervisor running on bare metal, adding another layer and resulting in at least four levels in a nested setup — L0 (bare metal, running the LPAR hypervisor), L1 (host hypervisor), L2 (guest hypervisor), L3 (nested guest).h]hXFor example, s390x always has an LPAR (LogicalPARtition) hypervisor running on bare metal, adding another layer and resulting in at least four levels in a nested setup — L0 (bare metal, running the LPAR hypervisor), L1 (host hypervisor), L2 (guest hypervisor), L3 (nested guest).}(hjbhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK*hjPubh)}(hThis document will stick with the three-level terminology (L0, L1, and L2) for all architectures; and will largely focus on x86.h]hThis document will stick with the three-level terminology (L0, L1, and L2) for all architectures; and will largely focus on x86.}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK0hjPubeh}(h]h ]h"]h$]h&]uh1jNhhhhhhhNubh)}(hhh](h)}(h Use Casesh]h Use Cases}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK6ubh)}(hJThere are several scenarios where nested KVM can be useful, to name a few:h]hJThere are several scenarios where nested KVM can be useful, to name a few:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK8hjhhubh)}(hhh](h)}(hXuAs a developer, you want to test your software on different operating systems (OSes). Instead of renting multiple VMs from a Cloud Provider, using nested KVM lets you rent a large enough "guest hypervisor" (level-1 guest). This in turn allows you to create multiple nested guests (level-2 guests), running different OSes, on which you can develop and test your software. h]h)}(hXtAs a developer, you want to test your software on different operating systems (OSes). Instead of renting multiple VMs from a Cloud Provider, using nested KVM lets you rent a large enough "guest hypervisor" (level-1 guest). This in turn allows you to create multiple nested guests (level-2 guests), running different OSes, on which you can develop and test your software.h]hXxAs a developer, you want to test your software on different operating systems (OSes). Instead of renting multiple VMs from a Cloud Provider, using nested KVM lets you rent a large enough “guest hypervisor” (level-1 guest). This in turn allows you to create multiple nested guests (level-2 guests), running different OSes, on which you can develop and test your software.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK;hjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hkLive migration of "guest hypervisors" and their nested guests, for load balancing, disaster recovery, etc. h]h)}(hjLive migration of "guest hypervisors" and their nested guests, for load balancing, disaster recovery, etc.h]hnLive migration of “guest hypervisors” and their nested guests, for load balancing, disaster recovery, etc.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKBhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hzVM image creation tools (e.g. ``virt-install``, etc) often run their own VM, and users expect these to work inside a VM. h]h)}(hyVM image creation tools (e.g. ``virt-install``, etc) often run their own VM, and users expect these to work inside a VM.h](hVM image creation tools (e.g. }(hjhhhNhNubhliteral)}(h``virt-install``h]h virt-install}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhK, etc) often run their own VM, and users expect these to work inside a VM.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKEhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hjSome OSes use virtualization internally for security (e.g. to let applications run safely in isolation). h]h)}(hhSome OSes use virtualization internally for security (e.g. to let applications run safely in isolation).h]hhSome OSes use virtualization internally for security (e.g. to let applications run safely in isolation).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKHhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubeh}(h]h ]h"]h$]h&]jLjMuh1hhhhK;hjhhubeh}(h] use-casesah ]h"] use casesah$]h&]uh1hhhhhhhhK6ubh)}(hhh](h)}(hEnabling "nested" (x86)h]hEnabling “nested” (x86)}(hj+hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj(hhhhhKMubh)}(hFrom Linux kernel v4.20 onwards, the ``nested`` KVM parameter is enabled by default for Intel and AMD. (Though your Linux distribution might override this default.)h](h%From Linux kernel v4.20 onwards, the }(hj9hhhNhNubj)}(h ``nested``h]hnested}(hjAhhhNhNubah}(h]h ]h"]h$]h&]uh1jhj9ubhv KVM parameter is enabled by default for Intel and AMD. (Though your Linux distribution might override this default.)}(hj9hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKOhj(hhubh)}(hIn case you are running a Linux kernel older than v4.19, to enable nesting, set the ``nested`` KVM module parameter to ``Y`` or ``1``. To persist this setting across reboots, you can add it in a config file, as shown below:h](hTIn case you are running a Linux kernel older than v4.19, to enable nesting, set the }(hjYhhhNhNubj)}(h ``nested``h]hnested}(hjahhhNhNubah}(h]h ]h"]h$]h&]uh1jhjYubh KVM module parameter to }(hjYhhhNhNubj)}(h``Y``h]hY}(hjshhhNhNubah}(h]h ]h"]h$]h&]uh1jhjYubh or }(hjYhhhNhNubj)}(h``1``h]h1}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjYubh[. To persist this setting across reboots, you can add it in a config file, as shown below:}(hjYhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKShj(hhubhenumerated_list)}(hhh](h)}(hOn the bare metal host (L0), list the kernel modules and ensure that the KVM modules:: $ lsmod | grep -i kvm kvm_intel 133627 0 kvm 435079 1 kvm_intel h](h)}(hVOn the bare metal host (L0), list the kernel modules and ensure that the KVM modules::h]hUOn the bare metal host (L0), list the kernel modules and ensure that the KVM modules:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKXhjubh)}(h_$ lsmod | grep -i kvm kvm_intel 133627 0 kvm 435079 1 kvm_intelh]h_$ lsmod | grep -i kvm kvm_intel 133627 0 kvm 435079 1 kvm_intel}hjsbah}(h]h ]h"]h$]h&]hhuh1hhhhK[hjubeh}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hpShow information for ``kvm_intel`` module:: $ modinfo kvm_intel | grep -i nested parm: nested:bool h](h)}(h+Show information for ``kvm_intel`` module::h](hShow information for }(hjhhhNhNubj)}(h ``kvm_intel``h]h kvm_intel}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh module:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhK_hjubh)}(h@$ modinfo kvm_intel | grep -i nested parm: nested:boolh]h@$ modinfo kvm_intel | grep -i nested parm: nested:bool}hjsbah}(h]h ]h"]h$]h&]hhuh1hhhhKahjubeh}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hFor the nested KVM configuration to persist across reboots, place the below in ``/etc/modprobed/kvm_intel.conf`` (create the file if it doesn't exist):: $ cat /etc/modprobe.d/kvm_intel.conf options kvm-intel nested=y h](h)}(hFor the nested KVM configuration to persist across reboots, place the below in ``/etc/modprobed/kvm_intel.conf`` (create the file if it doesn't exist)::h](hOFor the nested KVM configuration to persist across reboots, place the below in }(hjhhhNhNubj)}(h!``/etc/modprobed/kvm_intel.conf``h]h/etc/modprobed/kvm_intel.conf}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh) (create the file if it doesn’t exist):}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKdhjubh)}(h?$ cat /etc/modprobe.d/kvm_intel.conf options kvm-intel nested=yh]h?$ cat /etc/modprobe.d/kvm_intel.conf options kvm-intel nested=y}hj$sbah}(h]h ]h"]h$]h&]hhuh1hhhhKhhjubeh}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(h^Unload and re-load the KVM Intel module:: $ sudo rmmod kvm-intel $ sudo modprobe kvm-intel h](h)}(h)Unload and re-load the KVM Intel module::h]h(Unload and re-load the KVM Intel module:}(hj<hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKkhj8ubh)}(h0$ sudo rmmod kvm-intel $ sudo modprobe kvm-intelh]h0$ sudo rmmod kvm-intel $ sudo modprobe kvm-intel}hjJsbah}(h]h ]h"]h$]h&]hhuh1hhhhKmhj8ubeh}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hkVerify if the ``nested`` parameter for KVM is enabled:: $ cat /sys/module/kvm_intel/parameters/nested Y h](h)}(h7Verify if the ``nested`` parameter for KVM is enabled::h](hVerify if the }(hjbhhhNhNubj)}(h ``nested``h]hnested}(hjjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjbubh parameter for KVM is enabled:}(hjbhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKphj^ubh)}(h/$ cat /sys/module/kvm_intel/parameters/nested Yh]h/$ cat /sys/module/kvm_intel/parameters/nested Y}hjsbah}(h]h ]h"]h$]h&]hhuh1hhhhKrhj^ubeh}(h]h ]h"]h$]h&]uh1hhjhhhhhNubeh}(h]h ]h"]h$]h&]enumtypearabicprefixhsuffix.uh1jhj(hhhhhKXubh)}(h\For AMD hosts, the process is the same as above, except that the module name is ``kvm-amd``.h](hPFor AMD hosts, the process is the same as above, except that the module name is }(hjhhhNhNubj)}(h ``kvm-amd``h]hkvm-amd}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKuhj(hhubeh}(h]enabling-nested-x86ah ]h"]enabling "nested" (x86)ah$]h&]uh1hhhhhhhhKMubh)}(hhh](h)}(h1Additional nested-related kernel parameters (x86)h]h1Additional nested-related kernel parameters (x86)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKzubh)}(hX>If your hardware is sufficiently advanced (Intel Haswell processor or higher, which has newer hardware virt extensions), the following additional features will also be enabled by default: "Shadow VMCS (Virtual Machine Control Structure)", APIC Virtualization on your bare metal host (L0). Parameters for Intel hosts::h]hXAIf your hardware is sufficiently advanced (Intel Haswell processor or higher, which has newer hardware virt extensions), the following additional features will also be enabled by default: “Shadow VMCS (Virtual Machine Control Structure)”, APIC Virtualization on your bare metal host (L0). Parameters for Intel hosts:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK|hjhhubh)}(h$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs Y $ cat /sys/module/kvm_intel/parameters/enable_apicv Y $ cat /sys/module/kvm_intel/parameters/ept Yh]h$ cat /sys/module/kvm_intel/parameters/enable_shadow_vmcs Y $ cat /sys/module/kvm_intel/parameters/enable_apicv Y $ cat /sys/module/kvm_intel/parameters/ept Y}hjsbah}(h]h ]h"]h$]h&]hhuh1hhhhKhjhhubjO)}(hIf you suspect your L2 (i.e. nested guest) is running slower, ensure the above are enabled (particularly ``enable_shadow_vmcs`` and ``ept``).h]h)}(hIf you suspect your L2 (i.e. nested guest) is running slower, ensure the above are enabled (particularly ``enable_shadow_vmcs`` and ``ept``).h](hiIf you suspect your L2 (i.e. nested guest) is running slower, ensure the above are enabled (particularly }(hjhhhNhNubj)}(h``enable_shadow_vmcs``h]henable_shadow_vmcs}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh and }(hjhhhNhNubj)}(h``ept``h]hept}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh).}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jNhjhhhhhNubeh}(h]/additional-nested-related-kernel-parameters-x86ah ]h"]1additional nested-related kernel parameters (x86)ah$]h&]uh1hhhhhhhhKzubh)}(hhh](h)}(hStarting a nested guest (x86)h]hStarting a nested guest (x86)}(hj=hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj:hhhhhKubh)}(hhOnce your bare metal host (L0) is configured for nesting, you should be able to start an L1 guest with::h]hgOnce your bare metal host (L0) is configured for nesting, you should be able to start an L1 guest with:}(hjKhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj:hhubh)}(h$ qemu-kvm -cpu host [...]h]h$ qemu-kvm -cpu host [...]}hjYsbah}(h]h ]h"]h$]h&]hhuh1hhhhKhj:hhubh)}(hThe above will pass through the host CPU's capabilities as-is to the guest, or for better live migration compatibility, use a named CPU model supported by QEMU. e.g.::h]hThe above will pass through the host CPU’s capabilities as-is to the guest, or for better live migration compatibility, use a named CPU model supported by QEMU. e.g.:}(hjghhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj:hhubh)}(h)$ qemu-kvm -cpu Haswell-noTSX-IBRS,vmx=onh]h)$ qemu-kvm -cpu Haswell-noTSX-IBRS,vmx=on}hjusbah}(h]h ]h"]h$]h&]hhuh1hhhhKhj:hhubh)}(hfthen the guest hypervisor will subsequently be capable of running a nested guest with accelerated KVM.h]hfthen the guest hypervisor will subsequently be capable of running a nested guest with accelerated KVM.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj:hhubeh}(h]starting-a-nested-guest-x86ah ]h"]starting a nested guest (x86)ah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hEnabling "nested" (s390x)h]hEnabling “nested” (s390x)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubj)}(hhh]h)}(hoOn the host hypervisor (L0), enable the ``nested`` parameter on s390x:: $ rmmod kvm $ modprobe kvm nested=1 h](h)}(hGOn the host hypervisor (L0), enable the ``nested`` parameter on s390x::h](h(On the host hypervisor (L0), enable the }(hjhhhNhNubj)}(h ``nested``h]hnested}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh parameter on s390x:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(h#$ rmmod kvm $ modprobe kvm nested=1h]h#$ rmmod kvm $ modprobe kvm nested=1}hjsbah}(h]h ]h"]h$]h&]hhuh1hhhhKhjubeh}(h]h ]h"]h$]h&]uh1hhjhhhhhNubah}(h]h ]h"]h$]h&]jjjhjjuh1jhjhhhhhKubjO)}(hOn s390x, the kernel parameter ``hpage`` is mutually exclusive with the ``nested`` parameter — i.e. to be able to enable ``nested``, the ``hpage`` parameter *must* be disabled.h]h)}(hOn s390x, the kernel parameter ``hpage`` is mutually exclusive with the ``nested`` parameter — i.e. to be able to enable ``nested``, the ``hpage`` parameter *must* be disabled.h](hOn s390x, the kernel parameter }(hjhhhNhNubj)}(h ``hpage``h]hhpage}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh is mutually exclusive with the }(hjhhhNhNubj)}(h ``nested``h]hnested}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh) parameter — i.e. to be able to enable }(hjhhhNhNubj)}(h ``nested``h]hnested}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh, the }(hjhhhNhNubj)}(h ``hpage``h]hhpage}(hj-hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh parameter }(hjhhhNhNubhemphasis)}(h*must*h]hmust}(hjAhhhNhNubah}(h]h ]h"]h$]h&]uh1j?hjubh be disabled.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1jNhjhhhhhNubj)}(hhh](h)}(hThe guest hypervisor (L1) must be provided with the ``sie`` CPU feature — with QEMU, this can be done by using "host passthrough" (via the command-line ``-cpu host``). h]h)}(hThe guest hypervisor (L1) must be provided with the ``sie`` CPU feature — with QEMU, this can be done by using "host passthrough" (via the command-line ``-cpu host``).h](h4The guest hypervisor (L1) must be provided with the }(hjfhhhNhNubj)}(h``sie``h]hsie}(hjnhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjfubhc CPU feature — with QEMU, this can be done by using “host passthrough” (via the command-line }(hjfhhhNhNubj)}(h ``-cpu host``h]h -cpu host}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjfubh).}(hjfhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjbubah}(h]h ]h"]h$]h&]uh1hhj_hhhhhNubh)}(hRNow the KVM module can be loaded in the L1 (guest hypervisor):: $ modprobe kvm h](h)}(h?Now the KVM module can be loaded in the L1 (guest hypervisor)::h]h>Now the KVM module can be loaded in the L1 (guest hypervisor):}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubh)}(h$ modprobe kvmh]h$ modprobe kvm}hjsbah}(h]h ]h"]h$]h&]hhuh1hhhhKhjubeh}(h]h ]h"]h$]h&]uh1hhj_hhhhhNubeh}(h]h ]h"]h$]h&]jjjhjjstartKuh1jhjhhhhhKubeh}(h]enabling-nested-s390xah ]h"]enabling "nested" (s390x)ah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(hLive migration with nested KVMh]hLive migration with nested KVM}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hMigrating an L1 guest, with a *live* nested guest in it, to another bare metal host, works as of Linux kernel 5.3 and QEMU 4.2.0 for Intel x86 systems, and even on older versions for s390x.h](hMigrating an L1 guest, with a }(hjhhhNhNubj@)}(h*live*h]hlive}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j?hjubh nested guest in it, to another bare metal host, works as of Linux kernel 5.3 and QEMU 4.2.0 for Intel x86 systems, and even on older versions for s390x.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXOn AMD systems, once an L1 guest has started an L2 guest, the L1 guest should no longer be migrated or saved (refer to QEMU documentation on "savevm"/"loadvm") until the L2 guest shuts down. Attempting to migrate or save-and-load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a ``kernel BUG!`` entry in ``dmesg``, a kernel 'oops', or an outright kernel panic. Such a migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted. Migrating an L1 guest merely configured to support nesting, while not actually running L2 guests, is expected to function normally even on AMD systems but may fail once guests are started.h](hXLOn AMD systems, once an L1 guest has started an L2 guest, the L1 guest should no longer be migrated or saved (refer to QEMU documentation on “savevm”/”loadvm”) until the L2 guest shuts down. Attempting to migrate or save-and-load an L1 guest while an L2 guest is running will result in undefined behavior. You might see a }(hjhhhNhNubj)}(h``kernel BUG!``h]h kernel BUG!}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh entry in }(hjhhhNhNubj)}(h ``dmesg``h]hdmesg}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhXY, a kernel ‘oops’, or an outright kernel panic. Such a migrated or loaded L1 guest can no longer be considered stable or secure, and must be restarted. Migrating an L1 guest merely configured to support nesting, while not actually running L2 guests, is expected to function normally even on AMD systems but may fail once guests are started.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(htMigrating an L2 guest is always expected to succeed, so all the following scenarios should work even on AMD systems:h]htMigrating an L2 guest is always expected to succeed, so all the following scenarios should work even on AMD systems:}(hj6hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hQMigrating a nested guest (L2) to another L1 guest on the *same* bare metal host. h]h)}(hPMigrating a nested guest (L2) to another L1 guest on the *same* bare metal host.h](h9Migrating a nested guest (L2) to another L1 guest on the }(hjKhhhNhNubj@)}(h*same*h]hsame}(hjShhhNhNubah}(h]h ]h"]h$]h&]uh1j?hjKubh bare metal host.}(hjKhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjGubah}(h]h ]h"]h$]h&]uh1hhjDhhhhhNubh)}(hTMigrating a nested guest (L2) to another L1 guest on a *different* bare metal host. h]h)}(hSMigrating a nested guest (L2) to another L1 guest on a *different* bare metal host.h](h7Migrating a nested guest (L2) to another L1 guest on a }(hjuhhhNhNubj@)}(h *different*h]h different}(hj}hhhNhNubah}(h]h ]h"]h$]h&]uh1j?hjuubh bare metal host.}(hjuhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjqubah}(h]h ]h"]h$]h&]uh1hhjDhhhhhNubh)}(h4Migrating a nested guest (L2) to a bare metal host. h]h)}(h3Migrating a nested guest (L2) to a bare metal host.h]h3Migrating a nested guest (L2) to a bare metal host.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjDhhhhhNubeh}(h]h ]h"]h$]h&]jLjMuh1hhhhKhjhhubeh}(h]live-migration-with-nested-kvmah ]h"]live migration with nested kvmah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h!Reporting bugs from nested setupsh]h!Reporting bugs from nested setups}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hDebugging "nested" problems can involve sifting through log files across L0, L1 and L2; this can result in tedious back-n-forth between the bug reporter and the bug fixer.h]hDebugging “nested” problems can involve sifting through log files across L0, L1 and L2; this can result in tedious back-n-forth between the bug reporter and the bug fixer.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hhh](h)}(hMention that you are in a "nested" setup. If you are running any kind of "nesting" at all, say so. Unfortunately, this needs to be called out because when reporting bugs, people tend to forget to even *mention* that they're using nested virtualization. h]h)}(hMention that you are in a "nested" setup. If you are running any kind of "nesting" at all, say so. Unfortunately, this needs to be called out because when reporting bugs, people tend to forget to even *mention* that they're using nested virtualization.h](hMention that you are in a “nested” setup. If you are running any kind of “nesting” at all, say so. Unfortunately, this needs to be called out because when reporting bugs, people tend to forget to even }(hjhhhNhNubj@)}(h *mention*h]hmention}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j?hjubh, that they’re using nested virtualization.}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubh)}(hXWEnsure you are actually running KVM on KVM. Sometimes people do not have KVM enabled for their guest hypervisor (L1), which results in them running with pure emulation or what QEMU calls it as "TCG", but they think they're running nested KVM. Thus confusing "nested Virt" (which could also mean, QEMU on KVM) with "nested KVM" (KVM on KVM). h]h)}(hXVEnsure you are actually running KVM on KVM. Sometimes people do not have KVM enabled for their guest hypervisor (L1), which results in them running with pure emulation or what QEMU calls it as "TCG", but they think they're running nested KVM. Thus confusing "nested Virt" (which could also mean, QEMU on KVM) with "nested KVM" (KVM on KVM).h]hXdEnsure you are actually running KVM on KVM. Sometimes people do not have KVM enabled for their guest hypervisor (L1), which results in them running with pure emulation or what QEMU calls it as “TCG”, but they think they’re running nested KVM. Thus confusing “nested Virt” (which could also mean, QEMU on KVM) with “nested KVM” (KVM on KVM).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1hhjhhhhhNubeh}(h]h ]h"]h$]h&]jLjMuh1hhhhKhjhhubh)}(hhh](h)}(h Information to collect (generic)h]h Information to collect (generic)}(hj.hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj+hhhhhKubh)}(hHThe following is not an exhaustive list, but a very good starting point:h]hHThe following is not an exhaustive list, but a very good starting point:}(hj<hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj+hhubh block_quote)}(hX- Kernel, libvirt, and QEMU version from L0 - Kernel, libvirt and QEMU version from L1 - QEMU command-line of L1 -- when using libvirt, you'll find it here: ``/var/log/libvirt/qemu/instance.log`` - QEMU command-line of L2 -- as above, when using libvirt, get the complete libvirt-generated QEMU command-line - ``cat /sys/cpuinfo`` from L0 - ``cat /sys/cpuinfo`` from L1 - ``lscpu`` from L0 - ``lscpu`` from L1 - Full ``dmesg`` output from L0 - Full ``dmesg`` output from L1 h]h)}(hhh](h)}(h*Kernel, libvirt, and QEMU version from L0 h]h)}(h)Kernel, libvirt, and QEMU version from L0h]h)Kernel, libvirt, and QEMU version from L0}(hjWhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjSubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(h)Kernel, libvirt and QEMU version from L1 h]h)}(h(Kernel, libvirt and QEMU version from L1h]h(Kernel, libvirt and QEMU version from L1}(hjohhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjkubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(hkQEMU command-line of L1 -- when using libvirt, you'll find it here: ``/var/log/libvirt/qemu/instance.log`` h]h)}(hjQEMU command-line of L1 -- when using libvirt, you'll find it here: ``/var/log/libvirt/qemu/instance.log``h](hFQEMU command-line of L1 -- when using libvirt, you’ll find it here: }(hjhhhNhNubj)}(h&``/var/log/libvirt/qemu/instance.log``h]h"/var/log/libvirt/qemu/instance.log}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(hnQEMU command-line of L2 -- as above, when using libvirt, get the complete libvirt-generated QEMU command-line h]h)}(hmQEMU command-line of L2 -- as above, when using libvirt, get the complete libvirt-generated QEMU command-lineh]hmQEMU command-line of L2 -- as above, when using libvirt, get the complete libvirt-generated QEMU command-line}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(h``cat /sys/cpuinfo`` from L0 h]h)}(h``cat /sys/cpuinfo`` from L0h](j)}(h``cat /sys/cpuinfo``h]hcat /sys/cpuinfo}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh from L0}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(h``cat /sys/cpuinfo`` from L1 h]h)}(h``cat /sys/cpuinfo`` from L1h](j)}(h``cat /sys/cpuinfo``h]hcat /sys/cpuinfo}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh from L1}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(h``lscpu`` from L0 h]h)}(h``lscpu`` from L0h](j)}(h ``lscpu``h]hlscpu}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh from L0}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj ubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(h``lscpu`` from L1 h]h)}(h``lscpu`` from L1h](j)}(h ``lscpu``h]hlscpu}(hj;hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj7ubh from L1}(hj7hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhj3ubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(hFull ``dmesg`` output from L0 h]h)}(hFull ``dmesg`` output from L0h](hFull }(hj]hhhNhNubj)}(h ``dmesg``h]hdmesg}(hjehhhNhNubah}(h]h ]h"]h$]h&]uh1jhj]ubh output from L0}(hj]hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhjYubah}(h]h ]h"]h$]h&]uh1hhjPubh)}(hFull ``dmesg`` output from L1 h]h)}(hFull ``dmesg`` output from L1h](hFull }(hjhhhNhNubj)}(h ``dmesg``h]hdmesg}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh output from L1}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1hhjPubeh}(h]h ]h"]h$]h&]jLjMuh1hhhhKhjLubah}(h]h ]h"]h$]h&]uh1jJhhhKhj+hhubeh}(h]information-to-collect-genericah ]h"] information to collect (generic)ah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(hx86-specific info to collecth]hx86-specific info to collect}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(h{Both the below commands, ``x86info`` and ``dmidecode``, should be available on most Linux distributions with the same name:h](hBoth the below commands, }(hjhhhNhNubj)}(h ``x86info``h]hx86info}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubh and }(hjhhhNhNubj)}(h ``dmidecode``h]h dmidecode}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1jhjubhE, should be available on most Linux distributions with the same name:}(hjhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhjhhubjK)}(h- Output of: ``x86info -a`` from L0 - Output of: ``x86info -a`` from L1 - Output of: ``dmidecode`` from L0 - Output of: ``dmidecode`` from L1 h]h)}(hhh](h)}(h"Output of: ``x86info -a`` from L0 h]h)}(h!Output of: ``x86info -a`` from L0h](h Output of: }(hj hhhNhNubj)}(h``x86info -a``h]h x86info -a}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh from L0}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1hhj ubh)}(h"Output of: ``x86info -a`` from L1 h]h)}(h!Output of: ``x86info -a`` from L1h](h Output of: }(hj9 hhhNhNubj)}(h``x86info -a``h]h x86info -a}(hjA hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj9 ubh from L1}(hj9 hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM hj5 ubah}(h]h ]h"]h$]h&]uh1hhj ubh)}(h!Output of: ``dmidecode`` from L0 h]h)}(h Output of: ``dmidecode`` from L0h](h Output of: }(hjc hhhNhNubj)}(h ``dmidecode``h]h dmidecode}(hjk hhhNhNubah}(h]h ]h"]h$]h&]uh1jhjc ubh from L0}(hjc hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhM hj_ ubah}(h]h ]h"]h$]h&]uh1hhj ubh)}(h!Output of: ``dmidecode`` from L1 h]h)}(h Output of: ``dmidecode`` from L1h](h Output of: }(hj hhhNhNubj)}(h ``dmidecode``h]h dmidecode}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh from L1}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1hhj ubeh}(h]h ]h"]h$]h&]jLjMuh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jJhhhMhjhhubeh}(h]x86-specific-info-to-collectah ]h"]x86-specific info to collectah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(hs390x-specific info to collecth]hs390x-specific info to collect}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hPAlong with the earlier mentioned generic details, the below is also recommended:h]hPAlong with the earlier mentioned generic details, the below is also recommended:}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubjK)}(hD- ``/proc/sysinfo`` from L1; this will also include the info from L0h]h)}(hhh]h)}(hB``/proc/sysinfo`` from L1; this will also include the info from L0h]h)}(hj h](j)}(h``/proc/sysinfo``h]h /proc/sysinfo}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1jhj ubh1 from L1; this will also include the info from L0}(hj hhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]jLjMuh1hhhhMhj ubah}(h]h ]h"]h$]h&]uh1jJhhhMhj hhubeh}(h]s390x-specific-info-to-collectah ]h"]s390x-specific info to collectah$]h&]uh1hhjhhhhhMubeh}(h]!reporting-bugs-from-nested-setupsah ]h"]!reporting bugs from nested setupsah$]h&]uh1hhhhhhhhKubeh}(h]running-nested-guests-with-kvmah ]h"]running nested guests with kvmah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksentryfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerjY error_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}nameids}(j3 j0 j%j"jjj7j4jjjjjjj+ j( jjj j j# j u nametypes}(j3 j%jj7jjjj+ jj j# uh}(j0 hj"jjj(j4jjj:jjjjj( jjj+j jj j u footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}Rparse_messages]hsystem_message)}(hhh]h)}(h:Enumerated list start value not ordinal-1: "2" (ordinal 2)h]h>Enumerated list start value not ordinal-1: “2” (ordinal 2)}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]levelKtypeINFOsourcehlineKuh1j hjhhhhhKubatransform_messages] transformerN include_log] decorationNhhub.