VT Driver Development

VT Technology (Writing a VT Framework)#

1. Introduction to VT Technology#

1. Technology Introduction#

1. VT Technology#

VT technology is a virtualization technology provided by Intel, fully known as Intel Virtualization Technology. It is a set of hardware and software solutions designed to enhance the performance, reliability, and security of virtualization environments. VT technology allows multiple virtual machines to run simultaneously on a single physical computer, with each virtual machine capable of running different operating systems and applications.

Intel VT (Intel Virtualization Technology) enables a single CPU to simulate multiple logical processors (Virtual CPU) in a virtualization environment, thereby allowing multiple operating systems to run concurrently.

VT technology is divided into software virtualization, container virtualization, and virtualization layer translation.

Software Virtualization: This virtualization technology is implemented based on software, running virtualization software (such as VMware Workstation, Virtual PC) on a host operating system, creating and managing virtual machines by simulating the hardware environment. The operating systems and applications running in each virtual machine do not need to be modified.
Container Virtualization: Container virtualization is a lightweight virtualization technology that uses features of the operating system kernel, such as Linux containers (LXC) or Docker containers, to isolate applications and their runtime environments. Containers share the host operating system's kernel, so full operating system virtualization is not required.

2. Abstract "Ring-1 Layer"#

In traditional computer architecture, there is no clearly defined "Ring -1" layer. Typically, the "Ring" hierarchy in computer architecture refers to privilege levels or permission levels used to control access to system resources. Common levels include Ring 0 (kernel mode) and Ring 3 (user mode).

However, Intel's virtualization technology ("VT technology") allows multiple virtual machines to run simultaneously on a single physical computer. Virtualization technology introduces new software and hardware layers to manage the creation, execution, and resource allocation of virtual machines.

In virtualization technology, the host operating system can be viewed as Ring 0, and the operating systems of the virtual machines can be viewed as Ring 3. In this case, it can be considered that virtualization technology introduces an abstract "Ring -1" layer for managing the creation and resource allocation of virtual machines. This abstract layer exists between the host operating system and the virtual machine operating systems and can be seen as a level of virtualization management.

It is important to note that "Ring -1" is merely an abstract concept used to explain the hierarchical relationships in virtualization technology and is not a standard term in traditional computer architecture. In practice, different virtualization technologies may adopt different hierarchical structures and terminologies, depending on the virtualization platform and technology used.

2. Keyword Introduction#

1. VMM (Virtual Machine Monitor)#

The Virtual Machine Monitor refers to software, firmware, or hardware on a computer used to establish and execute virtual machines.

2. VMX (Virtual Machine Extensions)#

Processor support for virtualization is provided by a processor operation form called VMX Operation, which has two types: VMX Root Operation and VMX Non-root Operation. Generally, a VMM will run in VMX Root Operation, while client software runs in VMX Non-root Operation.

3. VM (Virtual Machine)#

VM refers to Virtual Machine.

4. VMCS (Virtual Machine Control Structures)#

Logical processors use Virtual Machine Control Structures (VMCSs) when executing VMX operations. These operations can manage transitions into and out of VMX Non-root Operation (VM entry and VM exit) and the processor behavior in VMX Non-root Operation. This structure is operated by new instructions: VMCLEAR, VMPTRLD, VMREAD, and VMWRITE.

5. VMX Root Operation#

Typically, the VMM will run in this mode.

6. VMX Non-root Operation#

Typically, client software (virtual machines) will run in this environment. The transition between the two types of operations is called VMX transition; transitioning from root operation mode to non-root operation mode is called VMX Entry, while transitioning from non-root operation mode to root operation mode is called VMX Exit.

7. Guest Software#

Each virtual machine (VM) is a client software runtime environment.

3. Lifecycle of VMM Software#

The above diagram describes the lifecycle of VMM software.

Enabling VMX: The lifecycle of VMX operations begins with enabling VMX extensions. At computer startup, the Virtual Machine Monitor (VMM) enables VMX extensions by setting the processor's control registers. This process typically occurs during the operating system boot or when virtualization software is loaded.
Entering VMX Operation Mode: After enabling VMX, the processor enters VMX operation mode. In this mode, the processor supports switching between virtual machines and the virtual machine monitor. The VMM is responsible for managing the creation, configuration, and execution of virtual machines.
Creating and Running Virtual Machines: In VMX operation mode, the VMM can create virtual machine instances and configure their resources. The virtual machine monitor manages the state and control information of virtual machines through VMCS (Virtual Machine Control Structure). The VMM switches between virtual machines and the virtual machine monitor, allowing virtual machines to run on the physical processor.
Monitoring and Controlling Virtual Machines: During the operation of virtual machines, the virtual machine monitor monitors their behavior and provides necessary control. It is responsible for handling interrupts, exceptions, and privileged instructions for virtual machines, ensuring resource isolation between virtual machines and between virtual machines and physical machines.
Exiting VMX Operation Mode: When a virtual machine completes execution or a specific event occurs, the virtual machine monitor switches control from the virtual machine back to the virtual machine monitor. This process is called VMX exit. During the exit process, the virtual machine monitor can read and update the state information of the virtual machine and perform necessary processing.
Disabling VMX: The lifecycle of VMX operations ends when virtualization is no longer needed. When the virtualization software is closed or the computer is shut down, the processor's VMX extensions will be disabled, and the processor will revert to normal non-virtualized mode.

In summary, the lifecycle of VMX Operation encompasses key stages such as enabling VMX extensions, entering VMX operation mode, creating and running virtual machines, monitoring and controlling virtual machines, exiting VMX operation mode, and disabling VMX extensions to support and manage hardware virtualization.

2. VT Technology (Part II) Detecting CPU Support#

1. Using CPUID Instruction to Detect CPU Virtualization Support#

Before entering VMX Operation, a series of checks need to be performed on the CPU to determine whether the CPU hardware supports virtualization technology.

The CPUID instruction can also be used in Ring 3 without needing to enter Ring 0 for detection. Below is a detailed explanation of the usage of the CPUID instruction.

1. CPUID Instruction Usage Explanation#

Syntax

The CPUID instruction requires the query number to be passed into the EAX register during execution.

xor rax, rax 
cpuid

Function

The CPUID instruction returns information and feature support of the processor, including processor vendor, processor family, feature bits, cache configuration, supported extended features, etc.

Input: The EAX register is used to pass the function number for the query.
Output: The EAX, EBX, ECX, and EDX registers are used to return information and features of the processor.

2. CPUID Instruction Parameter Explanation#

Query Processor Vendor Information: By setting EAX to 0 and executing the CPUID instruction, the processor vendor identifier will be returned in the EBX, EDX, and ECX registers in ASCII encoding.
Query Extended Feature Support: By setting EAX to 7 and executing the CPUID instruction, information about the extended features supported by the processor will be returned in the EBX, ECX, and EDX registers.
Query Cache Configuration: By setting EAX to 2 and executing the CPUID instruction, the processor's cache configuration information will be returned in the EAX, EBX, ECX, and EDX registers.
Query CPUID Maximum Supported Function Number: By setting EAX to 0x80000000 and executing the CPUID instruction, the maximum supported function number will be returned in the EAX register.

The above lists several commonly used instruction parameters. For detailed parameters, refer to the Intel white paper Volume 2 Chapter 3 (Instructions A-L) in the CPUID section (detailed in the book).

3. Checking if the CPU Supports VT Technology#

Since the system this article is based on is a 64-bit operating system, the compiler in VS supports inline assembly by default (of course, asm files can also be written). However, considering code compatibility, the built-in functions provided by VS are used in the code. Below is the built-in function query table for VS.

VS x64 Built-in Functions List

VMX Instruction Function Introduction

The instruction to be executed in assembly should be:

mov rax, 1
cpuid

After execution, check if the fifth bit of ECX is set to 1. If it is set to 1, it indicates that the current CPU supports virtualization technology.

Code:

EXTERN_C BOOLEAN Check_CPUID() {
    int Ecx[4];
    __cpuid(Ecx, 1);
    return (Ecx[2] >> 5) & 1; // Check if the sixth bit of Ecx is 1
}

2. Reading MSR Fields to Check if BIOS has Enabled VT Technology#

MSR (Model Specific Register) is a special type of register that contains model-specific information and configuration parameters of the processor. MSR registers are part of the processor architecture, defined and implemented by the processor manufacturer.

Each processor may have a different set of MSR registers, corresponding to specific functions and configuration options. MSR registers are typically used to control certain features of the processor, performance tuning, power management, and virtualization support.

Unlike general-purpose registers, MSR registers cannot be accessed directly but are read and written through special instructions (such as RDMSR and WRMSR). These instructions are used to load the address of the MSR into a specified register (such as ECX) and then perform the corresponding operation.

By reading the MSR_IA32_FEATURE_CONTROL field and checking the value obtained, if the 0th bit is 0, it indicates that VMX entry is protected and cannot directly execute instructions to enter VMX Operation. VT technology needs to be enabled in the BIOS.

1. RDMSR/WRMSR Instructions#

These two instructions are used to operate on MSR registers for reading and writing.

RDMSR

mov eax, msr_index
rdmsr

This instruction is used to read the value of an MSR. The MSR index number is placed in the EAX register, and after executing the instruction, the high 32 bits of the MSR value are stored in the EDX register, while the low 32 bits are stored in the EAX register.

WRMSR

mov eax, values
mov ecx, msr_index
wrmsr

This instruction is responsible for writing to the MSR register. The data to be written is placed in EAX, and the index number of the MSR register is placed in ECX, and executing the instruction will write the data.

C Language Code

Considering that the result to be verified is a binary number, we know that the 0th bit of an odd number is 1, and the 0th bit of an even number is 0, so we only need to check for odd or even.

BOOLEAN Check_CPUID() {
    int Ecx[4];
    __cpuid(Ecx, 1);
    return (Ecx[2] >> 5) & 1; // Check if the sixth bit of Ecx is 1
}

3. Reading CR0 and CR4 to Check if Successfully Entered VMX#

After completing the above work, it indicates that hardware support for entering VMX is already in place. However, it is also necessary to unlock the field in the CR4 register to allow VT to run, and this lock cannot be changed after entering VMX; otherwise, a blue screen will occur until VMX is turned off.

Here, we will introduce the functions of the six CR (Control Register) registers, but actually only five are used; CR1 is not utilized in real situations.

CR0: Controls settings related to protected mode, real mode, paging, and other system operations.
CR1: Reserved and not used.
CR2: Stores the linear address that triggered a page fault exception.
CR3: Stores the physical address of the page table, used for address translation and paging mechanisms.
CR4: Controls specific features and extensions of the processor, such as paging extensions, physical address extensions, etc.
CR8: Used to control the priority of interrupts and exception handling (used only in 64-bit operating systems).

If you are interested in CR registers, you can refer to this article by a senior Introduction to CR Register Bits.

In fact, each bit of the CR register has its own name, and the bit that controls whether VMX has been successfully entered is the VMXE bit of the CR4 register. Before running VT driver code, it is advisable to check whether VT has already been entered. If it has, unnecessary operations should be avoided to prevent the host from blue screening.

3. VMXON Entering VMX#

After completing the checks for CPU support mentioned in section 2, you can officially start writing the code to enter VMX.

1. VMXON and VMXOFF Instructions#

The way to enter VMX Operation mode is by executing the VMXON instruction, and the instruction to exit is VMXOFF.

VMXON

Before executing this instruction, you need to initialize and allocate a segment of memory, which should be naturally aligned and 1KB in size, referred to as "VMX_Region."

mov ecx, VMX_Region_Physical
vmxon ecx

The requirement for executing this instruction is to place the physical memory address corresponding to VMX_Region into a register before executing the instruction.

PVOID VMX_Region = ExAllocatePoolWithTag(NonePagePool, 0x1000, 'VMX');
ULONG_PTR VMX_Region_Phy = MmGetPhysicalAddress(VMX_Region).QuadPart;
// Set up VMX Region
__vmx_on(&VMX_Region_Phy);

VMXOFF

The VMXON instruction is executed directly when exiting VMX.

These two instructions can be used in the VS compiler as __vmx_off() and __vmx_on().

2. Setting VMX_Region#

This segment of memory needs to be allocated as NonePagePool type memory.

The first four bytes of this memory need to be written with VMCS_ID (MSR index 0x480).

By using __readmsr(), read and write it to this memory.

* (ULONG*)VMX_Region = __readmsr(0x480);

Checking for Error Locations#

After executing __vmx_on(), it is necessary to check the relevant bits of the Eflags register to see if the CF field of the eflags register is set to 1. If it is set to 1, it indicates that entering VMX has failed.

*(ULONG_PTR*)(&eflags) = __readeflags();
if (eflags.fields.cf != 0) {
    DbgPrint("[CPU:%d] VMXON ON Failed", index);
}

4. Setting VMCS Fields#

Before reading this chapter, please read Chapter 5 on Entering Virtualization.

After successfully entering the VMX environment, a segment of VMCS (Virtual Machine Control Structure) memory, referred to as "VMCS_Region," is also needed. This memory is primarily used to store and manage the control of virtual machine execution and contains relevant information about the VMCS.

1. vmwrite and vmread Instructions#

vmwrite Instruction

mov ecx, VMCS_Fields
mov eax, VMCS_Data
vmwrite ecx, eax

This instruction places the VMCS_Fields to be written into ECX and the data to be written into EAX, executing the instruction successfully writes the data into VMCS_Fields.

vmread Instruction

mov ecx, VMCS_Fields
vmread eax, ecx

This instruction places the VMCS_Fields to be read into ECX, executing the instruction reads the data into ECX.

The main fields that need to be written in VMCS include Guest_State, Host_State, and VM Execute State.

1. VM Execute State Settings#

Pin Base

Set the virtual machine's IA32_VMX_PINBASED_CTLS. This register controls the allowed settings for most pin-based virtual machine execution controls. The bits 31:0 of the register indicate the allowed 0 settings for these controls. If bit X in the MSR is cleared to 0, VM entry allows control X (the X bit of pin-based virtual machine execution control) to be 0; if bit X in the MSR is set to 1, and control X is 0, VM entry will fail.

CPU Base

Used to set IA32_VMX_PROCBASED_CTLS. This register controls the allowed settings for most processor-based virtual machine execution controls. For specific details on each bit's control, refer to Intel white paper Volume 3 Chapter 24 Section 6 Point 2.

VM Exit Controls

Set the IA32_VMX_EXIT_CTLS register, which controls most VM Exit allowed controls. For details, refer to Volume 3 Chapter 24 Section 7 Point 1.

VM Entry Controls

Used to set IA32_VMX_ENTRY_CTLS, which records most VM Entry allowed settings. For details, refer to Volume 3 Chapter 24 Section 8 Point 1.

Secondary Processor-Based

Used to set IA32_VMX_PROCBASED_CTLS2, mainly for configuring secondary processors, configuring APIC EPT, etc. For details, refer to Volume 3 Chapter 24 Section 6 Point 2.

2. Guest_State Settings#

Register Part

The segment registers, segment selectors, segment limits, AR, segment base addresses that need to be set include (ES, CS, DS, FS, GS, FS, TR, GDTR, IDTR, LDTR, RIP, RSP), etc. The text will provide a document with specific setting parameters.

Non-register

VMCS Link pointer.

3. Host_State Settings#

For details, please refer to the document at the end.

For the index numbers corresponding to the fields mentioned above, refer to Intel white paper Volume 3 Appendix B.

5. Entering Virtualization#

Before setting the above VMCS content, the execution of the vmclear and vmptrld instructions can be represented in code as follows:

__vmx_vmclear(&VMCS_Phy);
__vmx_vmptrld(&VMCS_Phy);
SetupVmcs(); // Set up VMCS
__vmx_vmlaunch(); // Enter virtualization
DbgPrint("Vmlaunch Failed"); // If vmlaunch is executed successfully, this function will not be executed

After executing __vmx_vmlaunch, if the program does not encounter any issues, it will directly enter GUEST mode, with RIP/RSP changing to GUEST_RIP/GUEST_RSP addresses. After a VMX-Exit event occurs, an exception is generated, and RIP/RSP will point to HOST_RIP/HOST_RSP.

If the DbgPrint function is successfully executed, please check the error number of VM_INSTRUCTION_ERROR, which can be found in Intel white paper Volume 3 Chapter 30 Section 4.

6. Handling VM Exit#

After successfully entering GUEST, RIP jumps to the GUEST_RIP location and continues executing code.

During this time, a large number of VMX-Exit events will occur. After a VM-Exit occurs, the virtual machine jumps to the HOST_RIP location. Therefore, a callback function called VMExithandler needs to be set at the HOST_RIP.

In the VMExithandler function, by checking the error number of VM_EXIT_REASON, you can determine the erroneous code and read GUEST_RIP to identify the location of the code execution error, allowing you to troubleshoot the error in the context of the code.

void ExitHandler(){
    DWORD64 ExitReason = __readmsr(VM_EXIT_REASON);
    DWORD64 GUEST_RIP =  __readmsr(GUEST_RIP);
    __DebugBreak();
}

By setting a breakpoint, you can obtain ExitReason and troubleshoot based on the error number (for details on ExitReason error numbers, refer to Volume 3 Appendix B).

The complete process is as follows:

Write the function for host RIP in an asm file, save the register information into the stack, and pass the pointer to the entry point of the Exithandler function via the call instruction.

EXTERN_C ExitHandler:PROC // Import function from another file

PUSHAQ MACRO
    push    rax
    push    rcx
    push    rdx
    push    rbx
    push    -1    
    push    rbp
    push    rsi
    push    rdi
    push    r8
    push    r9
    push    r10
    push    r11
    push    r12
    push    r13
    push    r14
    push    r15
ENDM

POPAQ MACRO
    pop     r15
    pop     r14
    pop     r13
    pop     r12
    pop     r11
    pop     r10
    pop     r9
    pop     r8
    pop     rdi
    pop     rsi
    pop     rbp
    add     rsp, 8    
    pop     rbx
    pop     rdx
    pop     rcx
    pop     rax
ENDM

ExithandlerEntry PROC
    pushaq
    mov rcx, rsp
    sub rsp, 50h
    call ExitHandler
    ····// Subsequent processing, rax controls the next flow
    popaq
    resume // Return to GUEST
ExithandlerEntry ENDP

By using switch case, the program hits Exitreason and sets functions to execute instructions in host mode that cannot be executed in guest mode.

Then, by reading VM_EXIT_INSTRUCTION_LEN, obtain the length of the current instruction pointed to by GUEST_RIP, and reset GUEST_RIP to GUEST_RIP + VM_EXIT_INSTRUCTION_LEN.

void CpuidError(// Accept GUEST register information){
    // Handle instructions that GUEST cannot execute or that generate exceptions in host mode
}
void NextCode(){
    DWORD64 GUESTrip =  __readmsr(GUEST_RIP);
    DWORD64 VM_EXIT_INSTRUCTION =  __readmsr(VM_EXIT_INSTRUCTION_LEN);
    DWORD64 NextCode = GUESTrip + VM_EXIT_INSTRUCTION;
    __vmx_vmwrite(GUEST_RIP, NextCode);
}

EXTERN_C BOOLEAN ExitHandler(// Set up a structure to obtain GUEST register information){
    DWORD64 ExitReason = __readmsr(VM_EXIT_REASON);
    DWORD64 GUESTrip =  __readmsr(GUEST_RIP);
    
    switch(ExitReason):
    case CPUIDError:  // Hit function
    {
        CpuidError();
        NextCode();
        break;
    }
    default:
        DbgPrint("Exit Reason %p\n", ExitReason); // Output unhandled Exit event
        __DebugBreak() // int breakpoint 
        break;
    return TRUE; // Return boolean value to determine if rax is 0
}

By setting the VMXExitHandler, you can handle VM exit events.

7. Exiting Virtualization#

Considering that the VT code is loaded into Windows as a driver, when it is necessary to close VT, the function to unload the driver must be executed. Therefore, it is essential to ensure that the program executes normally to the Return part of the DriverLoad function. Thus, before entering Guest, it is necessary to save stack information and register information to restore the stack and register information after successfully entering GUEST, allowing the program to continue executing normally to ensure the proper execution of the driver unload function.

GUEST is allowed to execute the VMCALL instruction, which directly exits GUEST mode. In the UnloadVT function, execute this code, and through the Exithandler, hit the vmcall error. The vmcall can provide parameters or not. By checking the rax value against a special parameter, if it matches, directly execute the vmxon instruction to close VT.

EXTERN_C void __fastcall AsmVmcall(ULONG_PTR num, ULONG_PTR param);

void UnloadVt(){
    asmcall(vmxexit, 0)    
}

In assembly, check the return value of the ExitHandler function.

ExithandlerEntry PROC
    pushaq
    mov rcx, rsp
    sub rsp, 50h
    call ExitHandler
    ····// Subsequent processing, rax controls the next flow
    test al, al // Check if the return value is 0
    jz ExitVT:
    popaq // Restore modified registers
    resume // Return to GUEST
    jmp Error
ExitVT:
    popaq
    vmxon
    jz Error
    jc Error
    push rax
    popfq                // Restore stack
    mov rsp, rdx            
    push rcx
    ret  
Error:
    int 3
ExithandlerEntry ENDP

In the ExitHandler, set:

VT CloseVT(){
    // Restore some segment attributes, limits, base addresses, etc.
}
BOOLEAN vmcallhandler(GUEST register information){
    // If rax is the set predetermined value
    CloseVt();
    return FALSE;
    // If not, handle the error normally
    return TRUE;
}
EXTERN_C BOOLEAN ExitHandler(// Set up a structure to obtain GUEST register information){
    DWORD64 ExitReason = __readmsr(VM_EXIT_REASON);
    DWORD64 GUESTrip =  __readmsr(GUEST_RIP);
    ret = TRUE;
    switch(ExitReason):
    case vmcall:  // Hit function
    {
        ret = vmcallhandler();
        break;
    }
    default:
        DbgPrint("Exit Reason %p\n", ExitReason); // Output unhandled Exit event
        __DebugBreak() // int breakpoint 
        break;
    return ret; // Return boolean value to determine if rax is 0
}

Summary#

The code has been uploaded to the GitHub repository.

All experts are welcome to critique, and if you find it good, please give it a star.

yifaang/VTDemo

A VT Demo By yifang

C10

The code in the article may have some issues; please refer to the source code of the GitHub project.

The documents needed in the article are here!!!

Intel white paper full volume & VMCS setting reference document:

Link: https://pan.baidu.com/s/1cmTCIKwaT_eGlnmpO178ZQ
Extraction code: yftx