sphinx.addnodesdocument)}( rawsourcechildren]( translations LanguagesNode)}(hhh](h pending_xref)}(hhh]docutils.nodesTextChinese (Simplified)}parenthsba attributes}(ids]classes]names]dupnames]backrefs] refdomainstdreftypedoc reftarget,/translations/zh_CN/virt/kvm/x86/timekeepingmodnameN classnameN refexplicitutagnamehhh ubh)}(hhh]hChinese (Traditional)}hh2sbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/zh_TW/virt/kvm/x86/timekeepingmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hItalian}hhFsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/it_IT/virt/kvm/x86/timekeepingmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hJapanese}hhZsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/ja_JP/virt/kvm/x86/timekeepingmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hKorean}hhnsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/ko_KR/virt/kvm/x86/timekeepingmodnameN classnameN refexplicituh1hhh ubh)}(hhh]hSpanish}hhsbah}(h]h ]h"]h$]h&] refdomainh)reftypeh+ reftarget,/translations/sp_SP/virt/kvm/x86/timekeepingmodnameN classnameN refexplicituh1hhh ubeh}(h]h ]h"]h$]h&]current_languageEnglishuh1h hh _documenthsourceNlineNubhcomment)}(h SPDX-License-Identifier: GPL-2.0h]h SPDX-License-Identifier: GPL-2.0}hhsbah}(h]h ]h"]h$]h&] xml:spacepreserveuh1hhhhhhF/var/lib/git/docbuild/linux/Documentation/virt/kvm/x86/timekeeping.rsthKubhsection)}(hhh](htitle)}(h6Timekeeping Virtualization for X86-Based Architecturesh]h6Timekeeping Virtualization for X86-Based Architectures}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh field_list)}(hhh](hfield)}(hhh](h field_name)}(hAuthorh]hAuthor}(hhhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhhhKubh field_body)}(h#Zachary Amsden h]h paragraph)}(hhh](hZachary Amsden <}(hhhhhNhNubh reference)}(hzamsden@redhat.comh]hzamsden@redhat.com}(hhhhhNhNubah}(h]h ]h"]h$]h&]refurimailto:zamsden@redhat.comuh1hhhubh>}(hhhhhNhNubeh}(h]h ]h"]h$]h&]uh1hhhhKhhubah}(h]h ]h"]h$]h&]uh1hhhubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubh)}(hhh](h)}(h Copyrighth]h Copyright}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhKubh)}(h)(c) 2010, Red Hat. All rights reserved. h]henumerated_list)}(hhh]h list_item)}(h%2010, Red Hat. All rights reserved. h]h)}(h$2010, Red Hat. All rights reserved.h]h$2010, Red Hat. All rights reserved.}(hj:hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj6ubah}(h]h ]h"]h$]h&]uh1j4hj1ubah}(h]h ]h"]h$]h&]enumtype loweralphaprefix(suffix)startKuh1j/hj+ubah}(h]h ]h"]h$]h&]uh1hhjubeh}(h]h ]h"]h$]h&]uh1hhhhKhhhhubeh}(h]h ]h"]h$]h&]uh1hhhhhhhhKubh)}(hRContents 1) Overview 2) Timing Devices 3) TSC Hardware 4) Virtualization Problemsh]hRContents 1) Overview 2) Timing Devices 3) TSC Hardware 4) Virtualization Problems}hjmsbah}(h]h ]h"]h$]h&]hhuh1hhhhhhhhKubh)}(hhh](h)}(h 1. Overviewh]h 1. Overview}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj{hhhhhKubh)}(hXiOne of the most complicated parts of the X86 platform, and specifically, the virtualization of this platform is the plethora of timing devices available and the complexity of emulating those devices. In addition, virtualization of time introduces a new set of challenges because it introduces a multiplexed division of time beyond the control of the guest CPU.h]hXiOne of the most complicated parts of the X86 platform, and specifically, the virtualization of this platform is the plethora of timing devices available and the complexity of emulating those devices. In addition, virtualization of time introduces a new set of challenges because it introduces a multiplexed division of time beyond the control of the guest CPU.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj{hhubh)}(hFirst, we will describe the various timekeeping hardware available, then present some of the problems which arise and solutions available, giving specific recommendations for certain classes of KVM guests.h]hFirst, we will describe the various timekeeping hardware available, then present some of the problems which arise and solutions available, giving specific recommendations for certain classes of KVM guests.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj{hhubh)}(hThe purpose of this document is to collect data and information relevant to timekeeping which may be difficult to find elsewhere, specifically, information relevant to KVM and hardware-based virtualization.h]hThe purpose of this document is to collect data and information relevant to timekeeping which may be difficult to find elsewhere, specifically, information relevant to KVM and hardware-based virtualization.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhj{hhubeh}(h]overviewah ]h"] 1. overviewah$]h&]uh1hhhhhhhhKubh)}(hhh](h)}(h2. Timing Devicesh]h2. Timing Devices}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK#ubh)}(hFirst we discuss the basic hardware devices available. TSC and the related KVM clock are special enough to warrant a full exposition and are described in the following section.h]hFirst we discuss the basic hardware devices available. TSC and the related KVM clock are special enough to warrant a full exposition and are described in the following section.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK%hjhhubh)}(hhh](h)}(h2.1. i8254 - PITh]h2.1. i8254 - PIT}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhK*ubh)}(hX?One of the first timer devices available is the programmable interrupt timer, or PIT. The PIT has a fixed frequency 1.193182 MHz base clock and three channels which can be programmed to deliver periodic or one-shot interrupts. These three channels can be configured in different modes and have individual counters. Channel 1 and 2 were not available for general use in the original IBM PC, and historically were connected to control RAM refresh and the PC speaker. Now the PIT is typically integrated as part of an emulated chipset and a separate physical PIT is not used.h]hX?One of the first timer devices available is the programmable interrupt timer, or PIT. The PIT has a fixed frequency 1.193182 MHz base clock and three channels which can be programmed to deliver periodic or one-shot interrupts. These three channels can be configured in different modes and have individual counters. Channel 1 and 2 were not available for general use in the original IBM PC, and historically were connected to control RAM refresh and the PC speaker. Now the PIT is typically integrated as part of an emulated chipset and a separate physical PIT is not used.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK,hjhhubh)}(hXqThe PIT uses I/O ports 0x40 - 0x43. Access to the 16-bit counters is done using single or multiple byte access to the I/O ports. There are 6 modes available, but not all modes are available to all timers, as only timer 2 has a connected gate input, required for modes 1 and 5. The gate line is controlled by port 61h, bit 0, as illustrated in the following diagram::h]hXpThe PIT uses I/O ports 0x40 - 0x43. Access to the 16-bit counters is done using single or multiple byte access to the I/O ports. There are 6 modes available, but not all modes are available to all timers, as only timer 2 has a connected gate input, required for modes 1 and 5. The gate line is controlled by port 61h, bit 0, as illustrated in the following diagram:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK5hjhhubh literal_block)}(hXJ-------------- ---------------- | | | | | 1.1932 MHz|---------->| CLOCK OUT | ---------> IRQ 0 | Clock | | | | -------------- | +->| GATE TIMER 0 | | ---------------- | | ---------------- | | | |------>| CLOCK OUT | ---------> 66.3 KHZ DRAM | | | (aka /dev/null) | +->| GATE TIMER 1 | | ---------------- | | ---------------- | | | |------>| CLOCK OUT | ---------> Port 61h, bit 5 | | | Port 61h, bit 0 -------->| GATE TIMER 2 | \_.---- ____ ---------------- _| )--|LPF|---Speaker / *---- \___/ Port 61h, bit 1 ---------------------------------/h]hXJ-------------- ---------------- | | | | | 1.1932 MHz|---------->| CLOCK OUT | ---------> IRQ 0 | Clock | | | | -------------- | +->| GATE TIMER 0 | | ---------------- | | ---------------- | | | |------>| CLOCK OUT | ---------> 66.3 KHZ DRAM | | | (aka /dev/null) | +->| GATE TIMER 1 | | ---------------- | | ---------------- | | | |------>| CLOCK OUT | ---------> Port 61h, bit 5 | | | Port 61h, bit 0 -------->| GATE TIMER 2 | \_.---- ____ ---------------- _| )--|LPF|---Speaker / *---- \___/ Port 61h, bit 1 ---------------------------------/}hj sbah}(h]h ]h"]h$]h&]hhuh1j hhhK;hjhhubh)}(h"The timer modes are now described.h]h"The timer modes are now described.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKRhjhhubhdefinition_list)}(hhh](hdefinition_list_item)}(hMode 0: Single Timeout. This is a one-shot software timeout that counts down when the gate is high (always true for timers 0 and 1). When the count reaches zero, the output goes high. h](hterm)}(hMode 0: Single Timeout.h]hMode 0: Single Timeout.}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1j3hhhKWhj/ubh definition)}(hhh]h)}(hThis is a one-shot software timeout that counts down when the gate is high (always true for timers 0 and 1). When the count reaches zero, the output goes high.h]hThis is a one-shot software timeout that counts down when the gate is high (always true for timers 0 and 1). When the count reaches zero, the output goes high.}(hjHhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKUhjEubah}(h]h ]h"]h$]h&]uh1jChj/ubeh}(h]h ]h"]h$]h&]uh1j-hhhKWhj*ubj.)}(hMode 1: Triggered One-shot. The output is initially set high. When the gate line is set high, a countdown is initiated (which does not stop if the gate is lowered), during which the output is set low. When the count reaches zero, the output goes high. h](j4)}(hMode 1: Triggered One-shot.h]hMode 1: Triggered One-shot.}(hjfhhhNhNubah}(h]h ]h"]h$]h&]uh1j3hhhK]hjbubjD)}(hhh]h)}(hThe output is initially set high. When the gate line is set high, a countdown is initiated (which does not stop if the gate is lowered), during which the output is set low. When the count reaches zero, the output goes high.h]hThe output is initially set high. When the gate line is set high, a countdown is initiated (which does not stop if the gate is lowered), during which the output is set low. When the count reaches zero, the output goes high.}(hjwhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKZhjtubah}(h]h ]h"]h$]h&]uh1jChjbubeh}(h]h ]h"]h$]h&]uh1j-hhhK]hj*hhubj.)}(hXsMode 2: Rate Generator. The output is initially set high. When the countdown reaches 1, the output goes low for one count and then returns high. The value is reloaded and the countdown automatically resumes. If the gate line goes low, the count is halted. If the output is low when the gate is lowered, the output automatically goes high (this only affects timer 2). h](j4)}(hMode 2: Rate Generator.h]hMode 2: Rate Generator.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j3hhhKdhjubjD)}(hhh]h)}(hXZThe output is initially set high. When the countdown reaches 1, the output goes low for one count and then returns high. The value is reloaded and the countdown automatically resumes. If the gate line goes low, the count is halted. If the output is low when the gate is lowered, the output automatically goes high (this only affects timer 2).h]hXZThe output is initially set high. When the countdown reaches 1, the output goes low for one count and then returns high. The value is reloaded and the countdown automatically resumes. If the gate line goes low, the count is halted. If the output is low when the gate is lowered, the output automatically goes high (this only affects timer 2).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhK`hjubah}(h]h ]h"]h$]h&]uh1jChjubeh}(h]h ]h"]h$]h&]uh1j-hhhKdhj*hhubj.)}(hXMode 3: Square Wave. This generates a high / low square wave. The count determines the length of the pulse, which alternates between high and low when zero is reached. The count only proceeds when gate is high and is automatically reloaded on reaching zero. The count is decremented twice at each clock to generate a full high / low cycle at the full periodic rate. If the count is even, the clock remains high for N/2 counts and low for N/2 counts; if the clock is odd, the clock is high for (N+1)/2 counts and low for (N-1)/2 counts. Only even values are latched by the counter, so odd values are not observed when reading. This is the intended mode for timer 2, which generates sine-like tones by low-pass filtering the square wave output. h](j4)}(hMode 3: Square Wave.h]hMode 3: Square Wave.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j3hhhKphjubjD)}(hhh]h)}(hXThis generates a high / low square wave. The count determines the length of the pulse, which alternates between high and low when zero is reached. The count only proceeds when gate is high and is automatically reloaded on reaching zero. The count is decremented twice at each clock to generate a full high / low cycle at the full periodic rate. If the count is even, the clock remains high for N/2 counts and low for N/2 counts; if the clock is odd, the clock is high for (N+1)/2 counts and low for (N-1)/2 counts. Only even values are latched by the counter, so odd values are not observed when reading. This is the intended mode for timer 2, which generates sine-like tones by low-pass filtering the square wave output.h]hXThis generates a high / low square wave. The count determines the length of the pulse, which alternates between high and low when zero is reached. The count only proceeds when gate is high and is automatically reloaded on reaching zero. The count is decremented twice at each clock to generate a full high / low cycle at the full periodic rate. If the count is even, the clock remains high for N/2 counts and low for N/2 counts; if the clock is odd, the clock is high for (N+1)/2 counts and low for (N-1)/2 counts. Only even values are latched by the counter, so odd values are not observed when reading. This is the intended mode for timer 2, which generates sine-like tones by low-pass filtering the square wave output.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKghjubah}(h]h ]h"]h$]h&]uh1jChjubeh}(h]h ]h"]h$]h&]uh1j-hhhKphj*hhubj.)}(hX Mode 4: Software Strobe. After programming this mode and loading the counter, the output remains high until the counter reaches zero. Then the output goes low for 1 clock cycle and returns high. The counter is not reloaded. Counting only occurs when gate is high. h](j4)}(hMode 4: Software Strobe.h]hMode 4: Software Strobe.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1j3hhhKvhjubjD)}(hhh]h)}(hAfter programming this mode and loading the counter, the output remains high until the counter reaches zero. Then the output goes low for 1 clock cycle and returns high. The counter is not reloaded. Counting only occurs when gate is high.h]hAfter programming this mode and loading the counter, the output remains high until the counter reaches zero. Then the output goes low for 1 clock cycle and returns high. The counter is not reloaded. Counting only occurs when gate is high.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKshjubah}(h]h ]h"]h$]h&]uh1jChjubeh}(h]h ]h"]h$]h&]uh1j-hhhKvhj*hhubj.)}(hX9Mode 5: Hardware Strobe. After programming and loading the counter, the output remains high. When the gate is raised, a countdown is initiated (which does not stop if the gate is lowered). When the counter reaches zero, the output goes low for 1 clock cycle and then returns high. The counter is not reloaded. h](j4)}(hMode 5: Hardware Strobe.h]hMode 5: Hardware Strobe.}(hj"hhhNhNubah}(h]h ]h"]h$]h&]uh1j3hhhK}hjubjD)}(hhh]h)}(hXAfter programming and loading the counter, the output remains high. When the gate is raised, a countdown is initiated (which does not stop if the gate is lowered). When the counter reaches zero, the output goes low for 1 clock cycle and then returns high. The counter is not reloaded.h]hXAfter programming and loading the counter, the output remains high. When the gate is raised, a countdown is initiated (which does not stop if the gate is lowered). When the counter reaches zero, the output goes low for 1 clock cycle and then returns high. The counter is not reloaded.}(hj3hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKyhj0ubah}(h]h ]h"]h$]h&]uh1jChjubeh}(h]h ]h"]h$]h&]uh1j-hhhK}hj*hhubeh}(h]h ]h"]h$]h&]uh1j(hjhhhhhNubh)}(hIn addition to normal binary counting, the PIT supports BCD counting. The command port, 0x43 is used to set the counter and mode for each of the three timers.h]hIn addition to normal binary counting, the PIT supports BCD counting. The command port, 0x43 is used to set the counter and mode for each of the three timers.}(hjShhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hEPIT commands, issued to port 0x43, using the following bit encoding::h]hDPIT commands, issued to port 0x43, using the following bit encoding:}(hjahhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj )}(h|Bit 7-4: Command (See table below) Bit 3-1: Mode (000 = Mode 0, 101 = Mode 5, 11X = undefined) Bit 0 : Binary (0) / BCD (1)h]h|Bit 7-4: Command (See table below) Bit 3-1: Mode (000 = Mode 0, 101 = Mode 5, 11X = undefined) Bit 0 : Binary (0) / BCD (1)}hjosbah}(h]h ]h"]h$]h&]hhuh1j hhhKhjhhubh)}(hCommand table::h]hCommand table:}(hj}hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj )}(hX0000 - Latch Timer 0 count for port 0x40 sample and hold the count to be read in port 0x40; additional commands ignored until counter is read; mode bits ignored. 0001 - Set Timer 0 LSB mode for port 0x40 set timer to read LSB only and force MSB to zero; mode bits set timer mode 0010 - Set Timer 0 MSB mode for port 0x40 set timer to read MSB only and force LSB to zero; mode bits set timer mode 0011 - Set Timer 0 16-bit mode for port 0x40 set timer to read / write LSB first, then MSB; mode bits set timer mode 0100 - Latch Timer 1 count for port 0x41 - as described above 0101 - Set Timer 1 LSB mode for port 0x41 - as described above 0110 - Set Timer 1 MSB mode for port 0x41 - as described above 0111 - Set Timer 1 16-bit mode for port 0x41 - as described above 1000 - Latch Timer 2 count for port 0x42 - as described above 1001 - Set Timer 2 LSB mode for port 0x42 - as described above 1010 - Set Timer 2 MSB mode for port 0x42 - as described above 1011 - Set Timer 2 16-bit mode for port 0x42 as described above 1101 - General counter latch Latch combination of counters into corresponding ports Bit 3 = Counter 2 Bit 2 = Counter 1 Bit 1 = Counter 0 Bit 0 = Unused 1110 - Latch timer status Latch combination of counter mode into corresponding ports Bit 3 = Counter 2 Bit 2 = Counter 1 Bit 1 = Counter 0 The output of ports 0x40-0x42 following this command will be: Bit 7 = Output pin Bit 6 = Count loaded (0 if timer has expired) Bit 5-4 = Read / Write mode 01 = MSB only 10 = LSB only 11 = LSB / MSB (16-bit) Bit 3-1 = Mode Bit 0 = Binary (0) / BCD mode (1)h]hX0000 - Latch Timer 0 count for port 0x40 sample and hold the count to be read in port 0x40; additional commands ignored until counter is read; mode bits ignored. 0001 - Set Timer 0 LSB mode for port 0x40 set timer to read LSB only and force MSB to zero; mode bits set timer mode 0010 - Set Timer 0 MSB mode for port 0x40 set timer to read MSB only and force LSB to zero; mode bits set timer mode 0011 - Set Timer 0 16-bit mode for port 0x40 set timer to read / write LSB first, then MSB; mode bits set timer mode 0100 - Latch Timer 1 count for port 0x41 - as described above 0101 - Set Timer 1 LSB mode for port 0x41 - as described above 0110 - Set Timer 1 MSB mode for port 0x41 - as described above 0111 - Set Timer 1 16-bit mode for port 0x41 - as described above 1000 - Latch Timer 2 count for port 0x42 - as described above 1001 - Set Timer 2 LSB mode for port 0x42 - as described above 1010 - Set Timer 2 MSB mode for port 0x42 - as described above 1011 - Set Timer 2 16-bit mode for port 0x42 as described above 1101 - General counter latch Latch combination of counters into corresponding ports Bit 3 = Counter 2 Bit 2 = Counter 1 Bit 1 = Counter 0 Bit 0 = Unused 1110 - Latch timer status Latch combination of counter mode into corresponding ports Bit 3 = Counter 2 Bit 2 = Counter 1 Bit 1 = Counter 0 The output of ports 0x40-0x42 following this command will be: Bit 7 = Output pin Bit 6 = Count loaded (0 if timer has expired) Bit 5-4 = Read / Write mode 01 = MSB only 10 = LSB only 11 = LSB / MSB (16-bit) Bit 3-1 = Mode Bit 0 = Binary (0) / BCD mode (1)}hjsbah}(h]h ]h"]h$]h&]hhuh1j hhhKhjhhubeh}(h] i8254-pitah ]h"]2.1. i8254 - pitah$]h&]uh1hhjhhhhhK*ubh)}(hhh](h)}(h2.2. RTCh]h2.2. RTC}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhKubh)}(hThe second device which was available in the original PC was the MC146818 real time clock. The original device is now obsolete, and usually emulated by the system chipset, sometimes by an HPET and some frankenstein IRQ routing.h]hThe second device which was available in the original PC was the MC146818 real time clock. The original device is now obsolete, and usually emulated by the system chipset, sometimes by an HPET and some frankenstein IRQ routing.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hXtThe RTC is accessed through CMOS variables, which uses an index register to control which bytes are read. Since there is only one index register, read of the CMOS and read of the RTC require lock protection (in addition, it is dangerous to allow userspace utilities such as hwclock to have direct RTC access, as they could corrupt kernel reads and writes of CMOS memory).h]hXtThe RTC is accessed through CMOS variables, which uses an index register to control which bytes are read. Since there is only one index register, read of the CMOS and read of the RTC require lock protection (in addition, it is dangerous to allow userspace utilities such as hwclock to have direct RTC access, as they could corrupt kernel reads and writes of CMOS memory).}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hX-The RTC generates an interrupt which is usually routed to IRQ 8. The interrupt can function as a periodic timer, an additional once a day alarm, and can issue interrupts after an update of the CMOS registers by the MC146818 is complete. The type of interrupt is signalled in the RTC status registers.h]hX-The RTC generates an interrupt which is usually routed to IRQ 8. The interrupt can function as a periodic timer, an additional once a day alarm, and can issue interrupts after an update of the CMOS registers by the MC146818 is complete. The type of interrupt is signalled in the RTC status registers.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hThe RTC will update the current time fields by battery power even while the system is off. The current time fields should not be read while an update is in progress, as indicated in the status register.h]hThe RTC will update the current time fields by battery power even while the system is off. The current time fields should not be read while an update is in progress, as indicated in the status register.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(hThe clock uses a 32.768kHz crystal, so bits 6-4 of register A should be programmed to a 32kHz divider if the RTC is to count seconds.h]hThe clock uses a 32.768kHz crystal, so bits 6-4 of register A should be programmed to a 32kHz divider if the RTC is to count seconds.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubh)}(h6This is the RAM map originally used for the RTC/CMOS::h]h5This is the RAM map originally used for the RTC/CMOS:}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhKhjhhubj )}(hX Location Size Description ------------------------------------------ 00h byte Current second (BCD) 01h byte Seconds alarm (BCD) 02h byte Current minute (BCD) 03h byte Minutes alarm (BCD) 04h byte Current hour (BCD) 05h byte Hours alarm (BCD) 06h byte Current day of week (BCD) 07h byte Current day of month (BCD) 08h byte Current month (BCD) 09h byte Current year (BCD) 0Ah byte Register A bit 7 = Update in progress bit 6-4 = Divider for clock 000 = 4.194 MHz 001 = 1.049 MHz 010 = 32 kHz 10X = test modes 110 = reset / disable 111 = reset / disable bit 3-0 = Rate selection for periodic interrupt 000 = periodic timer disabled 001 = 3.90625 uS 010 = 7.8125 uS 011 = .122070 mS 100 = .244141 mS ... 1101 = 125 mS 1110 = 250 mS 1111 = 500 mS 0Bh byte Register B bit 7 = Run (0) / Halt (1) bit 6 = Periodic interrupt enable bit 5 = Alarm interrupt enable bit 4 = Update-ended interrupt enable bit 3 = Square wave interrupt enable bit 2 = BCD calendar (0) / Binary (1) bit 1 = 12-hour mode (0) / 24-hour mode (1) bit 0 = 0 (DST off) / 1 (DST enabled) OCh byte Register C (read only) bit 7 = interrupt request flag (IRQF) bit 6 = periodic interrupt flag (PF) bit 5 = alarm interrupt flag (AF) bit 4 = update interrupt flag (UF) bit 3-0 = reserved ODh byte Register D (read only) bit 7 = RTC has power bit 6-0 = reserved 32h byte Current century BCD (*) (*) location vendor specific and now determined from ACPI global tablesh]hX Location Size Description ------------------------------------------ 00h byte Current second (BCD) 01h byte Seconds alarm (BCD) 02h byte Current minute (BCD) 03h byte Minutes alarm (BCD) 04h byte Current hour (BCD) 05h byte Hours alarm (BCD) 06h byte Current day of week (BCD) 07h byte Current day of month (BCD) 08h byte Current month (BCD) 09h byte Current year (BCD) 0Ah byte Register A bit 7 = Update in progress bit 6-4 = Divider for clock 000 = 4.194 MHz 001 = 1.049 MHz 010 = 32 kHz 10X = test modes 110 = reset / disable 111 = reset / disable bit 3-0 = Rate selection for periodic interrupt 000 = periodic timer disabled 001 = 3.90625 uS 010 = 7.8125 uS 011 = .122070 mS 100 = .244141 mS ... 1101 = 125 mS 1110 = 250 mS 1111 = 500 mS 0Bh byte Register B bit 7 = Run (0) / Halt (1) bit 6 = Periodic interrupt enable bit 5 = Alarm interrupt enable bit 4 = Update-ended interrupt enable bit 3 = Square wave interrupt enable bit 2 = BCD calendar (0) / Binary (1) bit 1 = 12-hour mode (0) / 24-hour mode (1) bit 0 = 0 (DST off) / 1 (DST enabled) OCh byte Register C (read only) bit 7 = interrupt request flag (IRQF) bit 6 = periodic interrupt flag (PF) bit 5 = alarm interrupt flag (AF) bit 4 = update interrupt flag (UF) bit 3-0 = reserved ODh byte Register D (read only) bit 7 = RTC has power bit 6-0 = reserved 32h byte Current century BCD (*) (*) location vendor specific and now determined from ACPI global tables}hjsbah}(h]h ]h"]h$]h&]hhuh1j hhhKhjhhubeh}(h]rtcah ]h"]2.2. rtcah$]h&]uh1hhjhhhhhKubh)}(hhh](h)}(h 2.3. APICh]h 2.3. APIC}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hXOn Pentium and later processors, an on-board timer is available to each CPU as part of the Advanced Programmable Interrupt Controller. The APIC is accessed through memory-mapped registers and provides interrupt service to each CPU, used for IPIs and local timer interrupts.h]hXOn Pentium and later processors, an on-board timer is available to each CPU as part of the Advanced Programmable Interrupt Controller. The APIC is accessed through memory-mapped registers and provides interrupt service to each CPU, used for IPIs and local timer interrupts.}(hj-hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hXAlthough in theory the APIC is a safe and stable source for local interrupts, in practice, many bugs and glitches have occurred due to the special nature of the APIC CPU-local memory-mapped hardware. Beware that CPU errata may affect the use of the APIC and that workarounds may be required. In addition, some of these workarounds pose unique constraints for virtualization - requiring either extra overhead incurred from extra reads of memory-mapped I/O or additional functionality that may be more computationally expensive to implement.h]hXAlthough in theory the APIC is a safe and stable source for local interrupts, in practice, many bugs and glitches have occurred due to the special nature of the APIC CPU-local memory-mapped hardware. Beware that CPU errata may affect the use of the APIC and that workarounds may be required. In addition, some of these workarounds pose unique constraints for virtualization - requiring either extra overhead incurred from extra reads of memory-mapped I/O or additional functionality that may be more computationally expensive to implement.}(hj;hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hX\Since the APIC is documented quite well in the Intel and AMD manuals, we will avoid repetition of the detail here. It should be pointed out that the APIC timer is programmed through the LVT (local vector timer) register, is capable of one-shot or periodic operation, and is based on the bus clock divided down by the programmable divider register.h]hX\Since the APIC is documented quite well in the Intel and AMD manuals, we will avoid repetition of the detail here. It should be pointed out that the APIC timer is programmed through the LVT (local vector timer) register, is capable of one-shot or periodic operation, and is based on the bus clock divided down by the programmable divider register.}(hjIhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]apicah ]h"] 2.3. apicah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h 2.4. HPETh]h 2.4. HPET}(hjbhhhNhNubah}(h]h ]h"]h$]h&]uh1hhj_hhhhhM$ubh)}(hXAHPET is quite complex, and was originally intended to replace the PIT / RTC support of the X86 PC. It remains to be seen whether that will be the case, as the de facto standard of PC hardware is to emulate these older devices. Some systems designated as legacy free may support only the HPET as a hardware timer device.h]hXAHPET is quite complex, and was originally intended to replace the PIT / RTC support of the X86 PC. It remains to be seen whether that will be the case, as the de facto standard of PC hardware is to emulate these older devices. Some systems designated as legacy free may support only the HPET as a hardware timer device.}(hjphhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM&hj_hhubh)}(hXThe HPET spec is rather loose and vague, requiring at least 3 hardware timers, but allowing implementation freedom to support many more. It also imposes no fixed rate on the timer frequency, but does impose some extremal values on frequency, error and slew.h]hXThe HPET spec is rather loose and vague, requiring at least 3 hardware timers, but allowing implementation freedom to support many more. It also imposes no fixed rate on the timer frequency, but does impose some extremal values on frequency, error and slew.}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM,hj_hhubh)}(hXIn general, the HPET is recommended as a high precision (compared to PIT /RTC) time source which is independent of local variation (as there is only one HPET in any given system). The HPET is also memory-mapped, and its presence is indicated through ACPI tables by the BIOS.h]hXIn general, the HPET is recommended as a high precision (compared to PIT /RTC) time source which is independent of local variation (as there is only one HPET in any given system). The HPET is also memory-mapped, and its presence is indicated through ACPI tables by the BIOS.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM1hj_hhubh)}(h~Detailed specification of the HPET is beyond the current scope of this document, as it is also very well documented elsewhere.h]h~Detailed specification of the HPET is beyond the current scope of this document, as it is also very well documented elsewhere.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM6hj_hhubeh}(h]hpetah ]h"] 2.4. hpetah$]h&]uh1hhjhhhhhM$ubh)}(hhh](h)}(h2.5. Offboard Timersh]h2.5. Offboard Timers}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM:ubh)}(hX'Several cards, both proprietary (watchdog boards) and commonplace (e1000) have timing chips built into the cards which may have registers which are accessible to kernel or user drivers. To the author's knowledge, using these to generate a clocksource for a Linux or other kernel has not yet been attempted and is in general frowned upon as not playing by the agreed rules of the game. Such a timer device would require additional support to be virtualized properly and is not considered important at this time as no known operating system does this.h]hX)Several cards, both proprietary (watchdog boards) and commonplace (e1000) have timing chips built into the cards which may have registers which are accessible to kernel or user drivers. To the author’s knowledge, using these to generate a clocksource for a Linux or other kernel has not yet been attempted and is in general frowned upon as not playing by the agreed rules of the game. Such a timer device would require additional support to be virtualized properly and is not considered important at this time as no known operating system does this.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM<hjhhubeh}(h]offboard-timersah ]h"]2.5. offboard timersah$]h&]uh1hhjhhhhhM:ubeh}(h]timing-devicesah ]h"]2. timing devicesah$]h&]uh1hhhhhhhhK#ubh)}(hhh](h)}(h3. TSC Hardwareh]h3. TSC Hardware}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMEubh)}(hThe TSC or time stamp counter is relatively simple in theory; it counts instruction cycles issued by the processor, which can be used as a measure of time. In practice, due to a number of problems, it is the most complicated timekeeping device to use.h]hThe TSC or time stamp counter is relatively simple in theory; it counts instruction cycles issued by the processor, which can be used as a measure of time. In practice, due to a number of problems, it is the most complicated timekeeping device to use.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMGhjhhubh)}(hXYThe TSC is represented internally as a 64-bit MSR which can be read with the RDMSR, RDTSC, or RDTSCP (when available) instructions. In the past, hardware limitations made it possible to write the TSC, but generally on old hardware it was only possible to write the low 32-bits of the 64-bit counter, and the upper 32-bits of the counter were cleared. Now, however, on Intel processors family 0Fh, for models 3, 4 and 6, and family 06h, models e and f, this restriction has been lifted and all 64-bits are writable. On AMD systems, the ability to write the TSC MSR is not an architectural guarantee.h]hXYThe TSC is represented internally as a 64-bit MSR which can be read with the RDMSR, RDTSC, or RDTSCP (when available) instructions. In the past, hardware limitations made it possible to write the TSC, but generally on old hardware it was only possible to write the low 32-bits of the 64-bit counter, and the upper 32-bits of the counter were cleared. Now, however, on Intel processors family 0Fh, for models 3, 4 and 6, and family 06h, models e and f, this restriction has been lifted and all 64-bits are writable. On AMD systems, the ability to write the TSC MSR is not an architectural guarantee.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMLhjhhubh)}(hThe TSC is accessible from CPL-0 and conditionally, for CPL > 0 software by means of the CR4.TSD bit, which when enabled, disables CPL > 0 TSC access.h]hThe TSC is accessible from CPL-0 and conditionally, for CPL > 0 software by means of the CR4.TSD bit, which when enabled, disables CPL > 0 TSC access.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMUhjhhubh)}(hXSome vendors have implemented an additional instruction, RDTSCP, which returns atomically not just the TSC, but an indicator which corresponds to the processor number. This can be used to index into an array of TSC variables to determine offset information in SMP systems where TSCs are not synchronized. The presence of this instruction must be determined by consulting CPUID feature bits.h]hXSome vendors have implemented an additional instruction, RDTSCP, which returns atomically not just the TSC, but an indicator which corresponds to the processor number. This can be used to index into an array of TSC variables to determine offset information in SMP systems where TSCs are not synchronized. The presence of this instruction must be determined by consulting CPUID feature bits.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMXhjhhubh)}(hXBoth VMX and SVM provide extension fields in the virtualization hardware which allows the guest visible TSC to be offset by a constant. Newer implementations promise to allow the TSC to additionally be scaled, but this hardware is not yet widely available.h]hXBoth VMX and SVM provide extension fields in the virtualization hardware which allows the guest visible TSC to be offset by a constant. Newer implementations promise to allow the TSC to additionally be scaled, but this hardware is not yet widely available.}(hj(hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM_hjhhubh)}(hhh](h)}(h3.1. TSC synchronizationh]h3.1. TSC synchronization}(hj9hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj6hhhhhMeubh)}(hXThe TSC is a CPU-local clock in most implementations. This means, on SMP platforms, the TSCs of different CPUs may start at different times depending on when the CPUs are powered on. Generally, CPUs on the same die will share the same clock, however, this is not always the case.h]hXThe TSC is a CPU-local clock in most implementations. This means, on SMP platforms, the TSCs of different CPUs may start at different times depending on when the CPUs are powered on. Generally, CPUs on the same die will share the same clock, however, this is not always the case.}(hjGhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMghj6hhubh)}(hXThe BIOS may attempt to resynchronize the TSCs during the poweron process and the operating system or other system software may attempt to do this as well. Several hardware limitations make the problem worse - if it is not possible to write the full 64-bits of the TSC, it may be impossible to match the TSC in newly arriving CPUs to that of the rest of the system, resulting in unsynchronized TSCs. This may be done by BIOS or system software, but in practice, getting a perfectly synchronized TSC will not be possible unless all values are read from the same clock, which generally only is possible on single socket systems or those with special hardware support.h]hXThe BIOS may attempt to resynchronize the TSCs during the poweron process and the operating system or other system software may attempt to do this as well. Several hardware limitations make the problem worse - if it is not possible to write the full 64-bits of the TSC, it may be impossible to match the TSC in newly arriving CPUs to that of the rest of the system, resulting in unsynchronized TSCs. This may be done by BIOS or system software, but in practice, getting a perfectly synchronized TSC will not be possible unless all values are read from the same clock, which generally only is possible on single socket systems or those with special hardware support.}(hjUhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMlhj6hhubeh}(h]tsc-synchronizationah ]h"]3.1. tsc synchronizationah$]h&]uh1hhjhhhhhMeubh)}(hhh](h)}(h3.2. TSC and CPU hotplugh]h3.2. TSC and CPU hotplug}(hjnhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjkhhhhhMwubh)}(hXAs touched on already, CPUs which arrive later than the boot time of the system may not have a TSC value that is synchronized with the rest of the system. Either system software, BIOS, or SMM code may actually try to establish the TSC to a value matching the rest of the system, but a perfect match is usually not a guarantee. This can have the effect of bringing a system from a state where TSC is synchronized back to a state where TSC synchronization flaws, however small, may be exposed to the OS and any virtualization environment.h]hXAs touched on already, CPUs which arrive later than the boot time of the system may not have a TSC value that is synchronized with the rest of the system. Either system software, BIOS, or SMM code may actually try to establish the TSC to a value matching the rest of the system, but a perfect match is usually not a guarantee. This can have the effect of bringing a system from a state where TSC is synchronized back to a state where TSC synchronization flaws, however small, may be exposed to the OS and any virtualization environment.}(hj|hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMyhjkhhubeh}(h]tsc-and-cpu-hotplugah ]h"]3.2. tsc and cpu hotplugah$]h&]uh1hhjhhhhhMwubh)}(hhh](h)}(h 3.3. TSC and multi-socket / NUMAh]h 3.3. TSC and multi-socket / NUMA}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hXMulti-socket systems, especially large multi-socket systems are likely to have individual clocksources rather than a single, universally distributed clock. Since these clocks are driven by different crystals, they will not have perfectly matched frequency, and temperature and electrical variations will cause the CPU clocks, and thus the TSCs to drift over time. Depending on the exact clock and bus design, the drift may or may not be fixed in absolute error, and may accumulate over time.h]hXMulti-socket systems, especially large multi-socket systems are likely to have individual clocksources rather than a single, universally distributed clock. Since these clocks are driven by different crystals, they will not have perfectly matched frequency, and temperature and electrical variations will cause the CPU clocks, and thus the TSCs to drift over time. Depending on the exact clock and bus design, the drift may or may not be fixed in absolute error, and may accumulate over time.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hXIn addition, very large systems may deliberately slew the clocks of individual cores. This technique, known as spread-spectrum clocking, reduces EMI at the clock frequency and harmonics of it, which may be required to pass FCC standards for telecommunications and computer equipment.h]hXIn addition, very large systems may deliberately slew the clocks of individual cores. This technique, known as spread-spectrum clocking, reduces EMI at the clock frequency and harmonics of it, which may be required to pass FCC standards for telecommunications and computer equipment.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(htIt is recommended not to trust the TSCs to remain synchronized on NUMA or multiple socket systems for these reasons.h]htIt is recommended not to trust the TSCs to remain synchronized on NUMA or multiple socket systems for these reasons.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]tsc-and-multi-socket-numaah ]h"] 3.3. tsc and multi-socket / numaah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h3.4. TSC and C-statesh]h3.4. TSC and C-states}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hX_C-states, or idling states of the processor, especially C1E and deeper sleep states may be problematic for TSC as well. The TSC may stop advancing in such a state, resulting in a TSC which is behind that of other CPUs when execution is resumed. Such CPUs must be detected and flagged by the operating system based on CPU and chipset identifications.h]hX_C-states, or idling states of the processor, especially C1E and deeper sleep states may be problematic for TSC as well. The TSC may stop advancing in such a state, resulting in a TSC which is behind that of other CPUs when execution is resumed. Such CPUs must be detected and flagged by the operating system based on CPU and chipset identifications.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hZThe TSC in such a case may be corrected by catching it up to a known external clocksource.h]hZThe TSC in such a case may be corrected by catching it up to a known external clocksource.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]tsc-and-c-statesah ]h"]3.4. tsc and c-statesah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h$3.5. TSC frequency change / P-statesh]h$3.5. TSC frequency change / P-states}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMubh)}(hXTo make things slightly more interesting, some CPUs may change frequency. They may or may not run the TSC at the same rate, and because the frequency change may be staggered or slewed, at some points in time, the TSC rate may not be known other than falling within a range of values. In this case, the TSC will not be a stable time source, and must be calibrated against a known, stable, external clock to be a usable source of time.h]hXTo make things slightly more interesting, some CPUs may change frequency. They may or may not run the TSC at the same rate, and because the frequency change may be staggered or slewed, at some points in time, the TSC rate may not be known other than falling within a range of values. In this case, the TSC will not be a stable time source, and must be calibrated against a known, stable, external clock to be a usable source of time.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hWhether the TSC runs at a constant rate or scales with the P-state is model dependent and must be determined by inspecting CPUID, chipset or vendor specific MSR fields.h]hWhether the TSC runs at a constant rate or scales with the P-state is model dependent and must be determined by inspecting CPUID, chipset or vendor specific MSR fields.}(hj)hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubh)}(hXIn addition, some vendors have known bugs where the P-state is actually compensated for properly during normal operation, but when the processor is inactive, the P-state may be raised temporarily to service cache misses from other processors. In such cases, the TSC on halted CPUs could advance faster than that of non-halted processors. AMD Turion processors are known to have this problem.h]hXIn addition, some vendors have known bugs where the P-state is actually compensated for properly during normal operation, but when the processor is inactive, the P-state may be raised temporarily to service cache misses from other processors. In such cases, the TSC on halted CPUs could advance faster than that of non-halted processors. AMD Turion processors are known to have this problem.}(hj7hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj hhubeh}(h]tsc-frequency-change-p-statesah ]h"]$3.5. tsc frequency change / p-statesah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h3.6. TSC and STPCLK / T-statesh]h3.6. TSC and STPCLK / T-states}(hjPhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjMhhhhhMubh)}(hXExternal signals given to the processor may also have the effect of stopping the TSC. This is typically done for thermal emergency power control to prevent an overheating condition, and typically, there is no way to detect that this condition has happened.h]hXExternal signals given to the processor may also have the effect of stopping the TSC. This is typically done for thermal emergency power control to prevent an overheating condition, and typically, there is no way to detect that this condition has happened.}(hj^hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjMhhubeh}(h]tsc-and-stpclk-t-statesah ]h"]3.6. tsc and stpclk / t-statesah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h3.7. TSC virtualization - VMXh]h3.7. TSC virtualization - VMX}(hjwhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjthhhhhMubh)}(hXEVMX provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP instructions, which is enough for full virtualization of TSC in any manner. In addition, VMX allows passing through the host TSC plus an additional TSC_OFFSET field specified in the VMCS. Special instructions must be used to read and write the VMCS field.h]hXEVMX provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP instructions, which is enough for full virtualization of TSC in any manner. In addition, VMX allows passing through the host TSC plus an additional TSC_OFFSET field specified in the VMCS. Special instructions must be used to read and write the VMCS field.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjthhubeh}(h]tsc-virtualization-vmxah ]h"]3.7. tsc virtualization - vmxah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h3.8. TSC virtualization - SVMh]h3.8. TSC virtualization - SVM}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hX SVM provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP instructions, which is enough for full virtualization of TSC in any manner. In addition, SVM allows passing through the host TSC plus an additional offset field specified in the SVM control block.h]hX SVM provides conditional trapping of RDTSC, RDMSR, WRMSR and RDTSCP instructions, which is enough for full virtualization of TSC in any manner. In addition, SVM allows passing through the host TSC plus an additional offset field specified in the SVM control block.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubeh}(h]tsc-virtualization-svmah ]h"]3.8. tsc virtualization - svmah$]h&]uh1hhjhhhhhMubh)}(hhh](h)}(h3.9. TSC feature bits in Linuxh]h3.9. TSC feature bits in Linux}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hXIn summary, there is no way to guarantee the TSC remains in perfect synchronization unless it is explicitly guaranteed by the architecture. Even if so, the TSCs in multi-sockets or NUMA systems may still run independently despite being locally consistent.h]hXIn summary, there is no way to guarantee the TSC remains in perfect synchronization unless it is explicitly guaranteed by the architecture. Even if so, the TSCs in multi-sockets or NUMA systems may still run independently despite being locally consistent.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hThe following feature bits are used by Linux to signal various TSC attributes, but they can only be taken to be meaningful for UP or single node systems.h]hThe following feature bits are used by Linux to signal various TSC attributes, but they can only be taken to be meaningful for UP or single node systems.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubhtable)}(hhh]htgroup)}(hhh](hcolspec)}(hhh]h}(h]h ]h"]h$]h&]colwidthKuh1jhjubj)}(hhh]h}(h]h ]h"]h$]h&]colwidthK'uh1jhjubhtbody)}(hhh](hrow)}(hhh](hentry)}(hhh]h)}(hX86_FEATURE_TSCh]hX86_FEATURE_TSC}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h The TSC is available in hardwareh]h The TSC is available in hardware}(hj5hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj2ubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(hX86_FEATURE_RDTSCPh]hX86_FEATURE_RDTSCP}(hjUhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjRubah}(h]h ]h"]h$]h&]uh1jhjOubj)}(hhh]h)}(h#The RDTSCP instruction is availableh]h#The RDTSCP instruction is available}(hjlhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjiubah}(h]h ]h"]h$]h&]uh1jhjOubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(hX86_FEATURE_CONSTANT_TSCh]hX86_FEATURE_CONSTANT_TSC}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h'The TSC rate is unchanged with P-statesh]h'The TSC rate is unchanged with P-states}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(hX86_FEATURE_NONSTOP_TSCh]hX86_FEATURE_NONSTOP_TSC}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h!The TSC does not stop in C-statesh]h!The TSC does not stop in C-states}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh](j)}(hhh]h)}(hX86_FEATURE_TSC_RELIABLEh]hX86_FEATURE_TSC_RELIABLE}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubj)}(hhh]h)}(h$TSC sync checks are skipped (VMware)h]h$TSC sync checks are skipped (VMware)}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjubah}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]uh1jhjubeh}(h]h ]h"]h$]h&]colsKuh1jhjubah}(h]h ]h"]h$]h&]uh1jhjhhhhhNubeh}(h]tsc-feature-bits-in-linuxah ]h"]3.9. tsc feature bits in linuxah$]h&]uh1hhjhhhhhMubeh}(h] tsc-hardwareah ]h"]3. tsc hardwareah$]h&]uh1hhhhhhhhMEubh)}(hhh](h)}(h4. Virtualization Problemsh]h4. Virtualization Problems}(hjQhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjNhhhhhMubh)}(hXTimekeeping is especially problematic for virtualization because a number of challenges arise. The most obvious problem is that time is now shared between the host and, potentially, a number of virtual machines. Thus the virtual operating system does not run with 100% usage of the CPU, despite the fact that it may very well make that assumption. It may expect it to remain true to very exacting bounds when interrupt sources are disabled, but in reality only its virtual interrupt sources are disabled, and the machine may still be preempted at any time. This causes problems as the passage of real time, the injection of machine interrupts and the associated clock sources are no longer completely synchronized with real time.h]hXTimekeeping is especially problematic for virtualization because a number of challenges arise. The most obvious problem is that time is now shared between the host and, potentially, a number of virtual machines. Thus the virtual operating system does not run with 100% usage of the CPU, despite the fact that it may very well make that assumption. It may expect it to remain true to very exacting bounds when interrupt sources are disabled, but in reality only its virtual interrupt sources are disabled, and the machine may still be preempted at any time. This causes problems as the passage of real time, the injection of machine interrupts and the associated clock sources are no longer completely synchronized with real time.}(hj_hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjNhhubh)}(hX_This same problem can occur on native hardware to a degree, as SMM mode may steal cycles from the naturally on X86 systems when SMM mode is used by the BIOS, but not in such an extreme fashion. However, the fact that SMM mode may cause similar problems to virtualization makes it a good justification for solving many of these problems on bare metal.h]hX_This same problem can occur on native hardware to a degree, as SMM mode may steal cycles from the naturally on X86 systems when SMM mode is used by the BIOS, but not in such an extreme fashion. However, the fact that SMM mode may cause similar problems to virtualization makes it a good justification for solving many of these problems on bare metal.}(hjmhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjNhhubh)}(hhh](h)}(h4.1. Interrupt clockingh]h4.1. Interrupt clocking}(hj~hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj{hhhhhMubh)}(hX)One of the most immediate problems that occurs with legacy operating systems is that the system timekeeping routines are often designed to keep track of time by counting periodic interrupts. These interrupts may come from the PIT or the RTC, but the problem is the same: the host virtualization engine may not be able to deliver the proper number of interrupts per second, and so guest time may fall behind. This is especially problematic if a high interrupt rate is selected, such as 1000 HZ, which is unfortunately the default for many Linux guests.h]hX)One of the most immediate problems that occurs with legacy operating systems is that the system timekeeping routines are often designed to keep track of time by counting periodic interrupts. These interrupts may come from the PIT or the RTC, but the problem is the same: the host virtualization engine may not be able to deliver the proper number of interrupts per second, and so guest time may fall behind. This is especially problematic if a high interrupt rate is selected, such as 1000 HZ, which is unfortunately the default for many Linux guests.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(hXiThere are three approaches to solving this problem; first, it may be possible to simply ignore it. Guests which have a separate time source for tracking 'wall clock' or 'real time' may not need any adjustment of their interrupts to maintain proper time. If this is not sufficient, it may be necessary to inject additional interrupts into the guest in order to increase the effective interrupt rate. This approach leads to complications in extreme conditions, where host load or guest lag is too much to compensate for, and thus another solution to the problem has risen: the guest may need to become aware of lost ticks and compensate for them internally. Although promising in theory, the implementation of this policy in Linux has been extremely error prone, and a number of buggy variants of lost tick compensation are distributed across commonly used Linux systems.Lh]hXqThere are three approaches to solving this problem; first, it may be possible to simply ignore it. Guests which have a separate time source for tracking ‘wall clock’ or ‘real time’ may not need any adjustment of their interrupts to maintain proper time. If this is not sufficient, it may be necessary to inject additional interrupts into the guest in order to increase the effective interrupt rate. This approach leads to complications in extreme conditions, where host load or guest lag is too much to compensate for, and thus another solution to the problem has risen: the guest may need to become aware of lost ticks and compensate for them internally. Although promising in theory, the implementation of this policy in Linux has been extremely error prone, and a number of buggy variants of lost tick compensation are distributed across commonly used Linux systems.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubh)}(hWindows uses periodic RTC clocking as a means of keeping time internally, and thus requires interrupt slewing to keep proper time. It does use a low enough rate (ed: is it 18.2 Hz?) however that it has not yet been a problem in practice.h]hWindows uses periodic RTC clocking as a means of keeping time internally, and thus requires interrupt slewing to keep proper time. It does use a low enough rate (ed: is it 18.2 Hz?) however that it has not yet been a problem in practice.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhj{hhubeh}(h]interrupt-clockingah ]h"]4.1. interrupt clockingah$]h&]uh1hhjNhhhhhMubh)}(hhh](h)}(h#4.2. TSC sampling and serializationh]h#4.2. TSC sampling and serialization}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhMubh)}(hXMAs the highest precision time source available, the cycle counter of the CPU has aroused much interest from developers. As explained above, this timer has many problems unique to its nature as a local, potentially unstable and potentially unsynchronized source. One issue which is not unique to the TSC, but is highlighted because of its very precise nature is sampling delay. By definition, the counter, once read is already old. However, it is also possible for the counter to be read ahead of the actual use of the result. This is a consequence of the superscalar execution of the instruction stream, which may execute instructions out of order. Such execution is called non-serialized. Forcing serialized execution is necessary for precise measurement with the TSC, and requires a serializing instruction, such as CPUID or an MSR read.h]hXMAs the highest precision time source available, the cycle counter of the CPU has aroused much interest from developers. As explained above, this timer has many problems unique to its nature as a local, potentially unstable and potentially unsynchronized source. One issue which is not unique to the TSC, but is highlighted because of its very precise nature is sampling delay. By definition, the counter, once read is already old. However, it is also possible for the counter to be read ahead of the actual use of the result. This is a consequence of the superscalar execution of the instruction stream, which may execute instructions out of order. Such execution is called non-serialized. Forcing serialized execution is necessary for precise measurement with the TSC, and requires a serializing instruction, such as CPUID or an MSR read.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMhjhhubh)}(hXSince CPUID may actually be virtualized by a trap and emulate mechanism, this serialization can pose a performance issue for hardware virtualization. An accurate time stamp counter reading may therefore not always be available, and it may be necessary for an implementation to guard against "backwards" reads of the TSC as seen from other CPUs, even in an otherwise perfectly synchronized system.h]hXSince CPUID may actually be virtualized by a trap and emulate mechanism, this serialization can pose a performance issue for hardware virtualization. An accurate time stamp counter reading may therefore not always be available, and it may be necessary for an implementation to guard against “backwards” reads of the TSC as seen from other CPUs, even in an otherwise perfectly synchronized system.}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM#hjhhubeh}(h]tsc-sampling-and-serializationah ]h"]#4.2. tsc sampling and serializationah$]h&]uh1hhjNhhhhhMubh)}(hhh](h)}(h4.3. Timespec aliasingh]h4.3. Timespec aliasing}(hjhhhNhNubah}(h]h ]h"]h$]h&]uh1hhjhhhhhM+ubh)}(hX Additionally, this lack of serialization from the TSC poses another challenge when using results of the TSC when measured against another time source. As the TSC is much higher precision, many possible values of the TSC may be read while another clock is still expressing the same value.h]hX Additionally, this lack of serialization from the TSC poses another challenge when using results of the TSC when measured against another time source. As the TSC is much higher precision, many possible values of the TSC may be read while another clock is still expressing the same value.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM-hjhhubh)}(hXThat is, you may read (T,T+10) while external clock C maintains the same value. Due to non-serialized reads, you may actually end up with a range which fluctuates - from (T-1.. T+10). Thus, any time calculated from a TSC, but calibrated against an external value may have a range of valid values. Re-calibrating this computation may actually cause time, as computed after the calibration, to go backwards, compared with time computed before the calibration.h]hXThat is, you may read (T,T+10) while external clock C maintains the same value. Due to non-serialized reads, you may actually end up with a range which fluctuates - from (T-1.. T+10). Thus, any time calculated from a TSC, but calibrated against an external value may have a range of valid values. Re-calibrating this computation may actually cause time, as computed after the calibration, to go backwards, compared with time computed before the calibration.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM2hjhhubh)}(hX3This problem is particularly pronounced with an internal time source in Linux, the kernel time, which is expressed in the theoretically high resolution timespec - but which advances in much larger granularity intervals, sometimes at the rate of jiffies, and possibly in catchup modes, at a much larger step.h]hX3This problem is particularly pronounced with an internal time source in Linux, the kernel time, which is expressed in the theoretically high resolution timespec - but which advances in much larger granularity intervals, sometimes at the rate of jiffies, and possibly in catchup modes, at a much larger step.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM:hjhhubh)}(hThis aliasing requires care in the computation and recalibration of kvmclock and any other values derived from TSC computation (such as TSC virtualization itself).h]hThis aliasing requires care in the computation and recalibration of kvmclock and any other values derived from TSC computation (such as TSC virtualization itself).}(hj. hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM?hjhhubeh}(h]timespec-aliasingah ]h"]4.3. timespec aliasingah$]h&]uh1hhjNhhhhhM+ubh)}(hhh](h)}(h4.4. Migrationh]h4.4. Migration}(hjG hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjD hhhhhMDubh)}(hXmMigration of a virtual machine raises problems for timekeeping in two ways. First, the migration itself may take time, during which interrupts cannot be delivered, and after which, the guest time may need to be caught up. NTP may be able to help to some degree here, as the clock correction required is typically small enough to fall in the NTP-correctable window.h]hXmMigration of a virtual machine raises problems for timekeeping in two ways. First, the migration itself may take time, during which interrupts cannot be delivered, and after which, the guest time may need to be caught up. NTP may be able to help to some degree here, as the clock correction required is typically small enough to fall in the NTP-correctable window.}(hjU hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMFhjD hhubh)}(hXAn additional concern is that timers based off the TSC (or HPET, if the raw bus clock is exposed) may now be running at different rates, requiring compensation in some way in the hypervisor by virtualizing these timers. In addition, migrating to a faster machine may preclude the use of a passthrough TSC, as a faster clock cannot be made visible to a guest without the potential of time advancing faster than usual. A slower clock is less of a problem, as it can always be caught up to the original rate. KVM clock avoids these problems by simply storing multipliers and offsets against the TSC for the guest to convert back into nanosecond resolution values.h]hXAn additional concern is that timers based off the TSC (or HPET, if the raw bus clock is exposed) may now be running at different rates, requiring compensation in some way in the hypervisor by virtualizing these timers. In addition, migrating to a faster machine may preclude the use of a passthrough TSC, as a faster clock cannot be made visible to a guest without the potential of time advancing faster than usual. A slower clock is less of a problem, as it can always be caught up to the original rate. KVM clock avoids these problems by simply storing multipliers and offsets against the TSC for the guest to convert back into nanosecond resolution values.}(hjc hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMLhjD hhubeh}(h] migrationah ]h"]4.4. migrationah$]h&]uh1hhjNhhhhhMDubh)}(hhh](h)}(h4.5. Schedulingh]h4.5. Scheduling}(hj| hhhNhNubah}(h]h ]h"]h$]h&]uh1hhjy hhhhhMWubh)}(hXSince scheduling may be based on precise timing and firing of interrupts, the scheduling algorithms of an operating system may be adversely affected by virtualization. In theory, the effect is random and should be universally distributed, but in contrived as well as real scenarios (guest device access, causes of virtualization exits, possible context switch), this may not always be the case. The effect of this has not been well studied.h]hXSince scheduling may be based on precise timing and firing of interrupts, the scheduling algorithms of an operating system may be adversely affected by virtualization. In theory, the effect is random and should be universally distributed, but in contrived as well as real scenarios (guest device access, causes of virtualization exits, possible context switch), this may not always be the case. The effect of this has not been well studied.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMYhjy hhubh)}(hIn an attempt to work around this, several implementations have provided a paravirtualized scheduler clock, which reveals the true amount of CPU time for which a virtual machine has been running.h]hIn an attempt to work around this, several implementations have provided a paravirtualized scheduler clock, which reveals the true amount of CPU time for which a virtual machine has been running.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM`hjy hhubeh}(h] schedulingah ]h"]4.5. schedulingah$]h&]uh1hhjNhhhhhMWubh)}(hhh](h)}(h4.6. Watchdogsh]h4.6. Watchdogs}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMeubh)}(hXOWatchdog timers, such as the lock detector in Linux may fire accidentally when running under hardware virtualization due to timer interrupts being delayed or misinterpretation of the passage of real time. Usually, these warnings are spurious and can be ignored, but in some circumstances it may be necessary to disable such detection.h]hXOWatchdog timers, such as the lock detector in Linux may fire accidentally when running under hardware virtualization due to timer interrupts being delayed or misinterpretation of the passage of real time. Usually, these warnings are spurious and can be ignored, but in some circumstances it may be necessary to disable such detection.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMghj hhubeh}(h] watchdogsah ]h"]4.6. watchdogsah$]h&]uh1hhjNhhhhhMeubh)}(hhh](h)}(h 4.7. Delays and precision timingh]h 4.7. Delays and precision timing}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhMnubh)}(hXPrecise timing and delays may not be possible in a virtualized system. This can happen if the system is controlling physical hardware, or issues delays to compensate for slower I/O to and from devices. The first issue is not solvable in general for a virtualized system; hardware control software can't be adequately virtualized without a full real-time operating system, which would require an RT aware virtualization platform.h]hXPrecise timing and delays may not be possible in a virtualized system. This can happen if the system is controlling physical hardware, or issues delays to compensate for slower I/O to and from devices. The first issue is not solvable in general for a virtualized system; hardware control software can’t be adequately virtualized without a full real-time operating system, which would require an RT aware virtualization platform.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMphj hhubh)}(hThe second issue may cause performance problems, but this is unlikely to be a significant issue. In many cases these delays may be eliminated through configuration or paravirtualization.h]hThe second issue may cause performance problems, but this is unlikely to be a significant issue. In many cases these delays may be eliminated through configuration or paravirtualization.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhMwhj hhubeh}(h]delays-and-precision-timingah ]h"] 4.7. delays and precision timingah$]h&]uh1hhjNhhhhhMnubh)}(hhh](h)}(h4.8. Covert channels and leaksh]h4.8. Covert channels and leaks}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj hhhhhM|ubh)}(hXcIn addition to the above problems, time information will inevitably leak to the guest about the host in anything but a perfect implementation of virtualized time. This may allow the guest to infer the presence of a hypervisor (as in a red-pill type detection), and it may allow information to leak between guests by using CPU utilization itself as a signalling channel. Preventing such problems would require completely isolated virtual time which may not track real time any longer. This may be useful in certain security or QA contexts, but in general isn't recommended for real-world deployment scenarios.h]hXeIn addition to the above problems, time information will inevitably leak to the guest about the host in anything but a perfect implementation of virtualized time. This may allow the guest to infer the presence of a hypervisor (as in a red-pill type detection), and it may allow information to leak between guests by using CPU utilization itself as a signalling channel. Preventing such problems would require completely isolated virtual time which may not track real time any longer. This may be useful in certain security or QA contexts, but in general isn’t recommended for real-world deployment scenarios.}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhhhM~hj hhubeh}(h]covert-channels-and-leaksah ]h"]4.8. covert channels and leaksah$]h&]uh1hhjNhhhhhM|ubeh}(h]virtualization-problemsah ]h"]4. virtualization problemsah$]h&]uh1hhhhhhhhMubeh}(h]6timekeeping-virtualization-for-x86-based-architecturesah ]h"]6timekeeping virtualization for x86-based architecturesah$]h&]uh1hhhhhhhhKubeh}(h]h ]h"]h$]h&]sourcehuh1hcurrent_sourceN current_lineNsettingsdocutils.frontendValues)}(hN generatorN datestampN source_linkN source_urlN toc_backlinksjfootnote_backlinksK sectnum_xformKstrip_commentsNstrip_elements_with_classesN strip_classesN report_levelK halt_levelKexit_status_levelKdebugNwarning_streamN tracebackinput_encoding utf-8-siginput_encoding_error_handlerstrictoutput_encodingutf-8output_encoding_error_handlerjc error_encodingutf-8error_encoding_error_handlerbackslashreplace language_codeenrecord_dependenciesNconfigN id_prefixhauto_id_prefixid dump_settingsNdump_internalsNdump_transformsNdump_pseudo_xmlNexpose_internalsNstrict_visitorN_disable_configN_sourceh _destinationN _config_files]7/var/lib/git/docbuild/linux/Documentation/docutils.confafile_insertion_enabled raw_enabledKline_length_limitM'pep_referencesN pep_base_urlhttps://peps.python.org/pep_file_url_templatepep-%04drfc_referencesN rfc_base_url&https://datatracker.ietf.org/doc/html/ tab_widthKtrim_footnote_reference_spacesyntax_highlightlong smart_quotessmartquotes_locales]character_level_inline_markupdoctitle_xform docinfo_xformKsectsubtitle_xform image_loadinglinkembed_stylesheetcloak_email_addressessection_self_linkenvNubreporterNindirect_targets]substitution_defs}substitution_names}refnames}refids}nameids}(j> j; jjjjjjjjj\jYjjjjjKjHjhjejjjjjjjJjGjqjnjjjjjCj@j6 j3 jjjjjA j> jv js j j j j j j j. j+ u nametypes}(j> jjjjj\jjjKjhjjjjJjqjjjCj6 jjjA jv j j j j. uh}(j; hjj{jjjjjjjYjjj_jjjHjjej6jjkjjjjjGj jnjMjjtjjj@jj3 jNjj{jjj> jjs jD j jy j j j j j+ j u footnote_refs} citation_refs} autofootnotes]autofootnote_refs]symbol_footnotes]symbol_footnote_refs] footnotes] citations]autofootnote_startKsymbol_footnote_startK id_counter collectionsCounter}Rparse_messages]hsystem_message)}(hhh]h)}(h:Enumerated list start value not ordinal-1: "c" (ordinal 3)h]h>Enumerated list start value not ordinal-1: “c” (ordinal 3)}(hj hhhNhNubah}(h]h ]h"]h$]h&]uh1hhj ubah}(h]h ]h"]h$]h&]levelKtypeINFOsourcehlineKuh1j hj+ubatransform_messages] transformerN include_log] decorationNhhub.