Intel TDX Demystified: A Top-Down Approach

3.1 TCB

The TCB of TDX consists of the TDX-enabled Intel processors and the built-in technologies, such as VT, MKTME, and SGX. The TCB also includes software modules signed by Intel, including the TDX Module, the NP/P-SEAM Loaders, and architectural SGX enclaves for remote attestation. The software stacks running within TDs are owned by the tenant and are considered part of the TCB. The cryptographic primitives used in TDX are considered sound and their implementation secure, including the generation of random numbers and the absence of side-channel attacks like timing attacks.

Tenants must trust the processor manufacturer, Intel, for developing, manufacturing, building, and signing the hardware/software components used by TDX. The source code packages for the TDX Module, the NP/P-SEAM Loaders, and the DCAP for attestation are publicly available for audit purposes, allowing tenants to assess their trustworthiness. However, tenants must also trust that the version signed by Intel is equivalent to the one they have reviewed, which involves placing trust in the compilation process to protect against supply chain attacks.

Moreover, tenants are required to trust Intel’s Provisioning Certification Service (PCS) for remote attestation. The PCS, which originally supported SGX attestation, has been expanded to include retrieval of Provisioning Certification Key (PCK) certificates, revocation lists, and TCB information for TDX.

4.1 AMD SEV

SEV [44] is a confidential computing feature in AMD EPYC processors. It protects sensitive data stored within VMs from privileged software or administrators in a multi-tenant cloud environment. SEV relies on AMD Secure Memory Encryption (SME) and AMD Virtualization (AMD-V) to enforce cryptographic isolation between VMs and the hypervisor. Each VM is assigned a unique ephemeral Advanced Encryption Standard (AES) key, which is used for runtime memory encryption. The AES engine in the on-die memory controller encrypts or decrypts data written to or read from the main memory. The per-VM keys are managed by the AMD Platform Security Processor (PSP), which is a 32-bit Arm Cortex-A5 micro-controller integrated within the AMD System-on-Chip (SoC). The C-bit (bit 47) in physical addresses determines memory page encryption. SEV also provides a remote attestation mechanism that allows the VM owners to verify the trustworthiness of VMs’ launch measurements and the SEV platforms. The PSP generates the attestation report signed by an AMD-certified attestation key. The VM owners can verify the authenticity of the attestation report and the embedded platform/guest measurements.

AMD has released three generations of SEV. The first generation SEV [44] only protects the confidentiality of a VM’s memory. The second generation SEV-ES (Encrypted State) [43] adds protection for CPU register state during hypervisor transition, and the third generation SEV-SNP (Secure Nested Paging) [3] adds integrity protection to prevent memory corrupting, replaying, and remapping attacks. Particularly, SEV-SNP provides memory integrity protection using Reverse Mapping Table (RMP). Reverse Mapping Table (RMP) tracks each page’s ownership and permissions to prevent unauthorized access. SEV-SNP also introduces the Virtual Machine Privilege Level (VMPL) feature by dividing the guest address space into four levels and providing additional security isolation within a VM. The privilege levels range from zero to three, where VMPL0 is the highest level of privilege and VMPL3 is the lowest. For instance, the Linux Secure VM Service Module (SVSM) [61] makes extensive use of the RMP and VMPL features to perform sensitive services, e.g., live migration and vTPM, in a secure manner.

4.2 IBM Confidential Computing

IBM’s early exploration of confidential computing can be traced back to the research on SecureBlue++ [10, 70], which included running on an emulated POWER processor on the Mambo CPU simulator [9]. Today, IBM Systems support two architectures for confidential computing: Secure Execution [24], offered on IBM Z and LinuxONE, and PEF [22], released as an open source project on OpenPOWER systems.

IBM Secure Execution. IBM Secure Execution provides support for Secure Virtual Machines (SVMs) that run inside isolated TEEs since IBM Z15 and LinuxONE III. Secure Execution protects the confidentiality, integrity, and authenticity of code and data in an SVM from any unauthorized access and snooping or tampering. Secure Execution leverages trusted firmware, called the Ultravisor, to perform security-sensitive tasks to bootstrap and run SVMs. The Ultravisor shields the SVM’s memory and its state during context switches and protects the SVM from a potentially compromised or malicious hypervisor. Tenants using Secure Execution can embed their encrypted sensitive data in the VM images and rely on the Ultravisor to decrypt and expose them to the SVMs executing inside the TEEs. Specifically, tenants can encrypt their confidential data with a symmetric data key, which they embed in the IBM Secure Execution Header. They further encrypt this header with the key obtained from the verified Host Key Document and embed the header in their VM image. The header can contain multiple key slots that allow an image to run on multiple target hosts. The Host Key Document, signed by the hardware manufacturer, contains the public key linked with the private key embedded in the hardware of IBM Z or LinuxONE. Ultravisor, the only component having access to the hardware private key and the data key, enforces that only the expected tenant’s SVM executing inside the TEE has access to the unencrypted data. In addition to embedding built-in secrets within the VM image, Secure Execution also supports remote attestation starting from IBM Z16 and LinuxONE Emperor 4. This allows tenants to verify the SVM’s measurements before releasing their secrets.

IBM PEF. PEF provides a VM-based TEE using extensions to the IBM Power Instruction Set Architecture (ISA) that are supported in most POWER9 and POWER10 processors. PEF firmware, tooling to prepare SVMs, and OS extensions, were released as open source software [23]. To protect sensitive data and code, PEF introduces a trusted firmware called Protected Execution Ultravisor (Ultravisor) that shields the SVM execution and enforces the security guarantees with the help of the CPU architectural changes. The PEF relies on the secure and trusted boot of the system and the Ultravisor executing in a new, highest privileged CPU state called Secure State. The hypervisor starts the VM, which invokes the Ultravisor to transition to an SVM using the Enter Secure Mode (ESM) call. The Ultravisor converts the VM into an SVM by moving it to the secure memory that is inaccessible to untrusted code. Before executing the SVM, the Ultravisor performs integrity checking. It decrypts the payload attached to the SVM image to decode the integrity information and a passphrase for the encrypted file system. After ensuring the integrity of the SVM, the Ultravisor exposes the passphrase to the SVM booting system that decrypts the tenant’s file system. The Ultravisor uses the Trusted Platform Module (TPM) to get access to the symmetric seed required to check integrity and decrypt the payload. The symmetric seed is guarded using the Platform Configuration Register (PCR) sealing mechanism and accessed by establishing a secure channel to the TPM. The TPM only grants access to an Ultravisor on a correctly booted system. If the Ultravisor gets access to the symmetric seed, it generates the HMAC key and symmetric key that are used to verify integrity and decrypt the passphrase.

4.3 Arm CCA

CCA [50] was introduced in the Armv9 architecture. Traditionally, Arm TrustZone allows secure execution by having two separated worlds, the Normal World and the Secure World. TrustZone prevents software in the Normal World from accessing data in the Secure World. CCA introduces the Realm Management Extension (RME) with two additional worlds, the Realm World and the Root World. The Realm World provides mutually distrusting execution environments for confidential VMs, isolating workloads from any other security domains, including host OS, hypervisor, other Realms, and TrustZone. To enforce the isolation of address spaces, CCA uses a Granule Protection Table (GPT), which is an extension to the page table that tracks the ownership of each page with different worlds. The Monitor in the Root World handles the creation and management of the GPT, preventing a hypervisor or an OS from directly changing it. The Monitor can dynamically move physical memory between different worlds by updating the GPT. CCA also supports attestation to measure and verify the CCA platform and the initial state of the Realms.

4.4 RISC-V CoVE

CoVE [57] is a reference confidential computing architecture for RISC-V. Its protected instance is called a TEE Virtual Machine (TVM). The architecture introduces the TEE Security Manager (TSM) driver, which is an M-mode (highest privilege level in RISC-V) firmware component for switching between confidential and non-confidential environments. The TSM driver tracks the assignment of memory pages to TVMs through the Memory Tracking Table (MTT). The TSM driver measures and loads the TSM, which is a trusted intermediary between the hypervisor and the TVMs. CoVE defines the Application Binary Interface (ABI) for the hypervisor to request virtual machine management services from the TSM. CoVE adopts a layered attestation architecture, which begins with the hardware and progresses through the TSM driver, TSM, and TVM. Each layer is loaded, measured, and certified by the previous layer. This approach provides a secure chain of trust that can be used to verify the integrity of the system. The TVM can obtain a certificate from the TSM that contains attestation evidence rooted back to the hardware. This certificate provides a mechanism for verifying the authenticity of the TVM and the software it runs.

5.1 Intel VT

Intel VT [64] is a set of hardware-assisted virtualization features in Intel processors. Using VT, Virtual Machine Monitors (VMMs) or hypervisors can achieve better performance, isolation, and security compared to software-based virtualization. Intel’s VT portfolio includes, among others, the virtualization of CPU, memory, and I/O.

Processors with VT-x technology have a special instruction set, called Virtual Machine Extensions (VMX), which enables control of virtualization. Processors with VT-x technology can operate in two modes: VMX root mode and VMX non-root mode. The hypervisor runs in VMX root mode while the guest VMs run in the VMX non-root mode. VT-x defines two new transitions, VM entry and VM exit, to switch between the guest and the hypervisor. The Virtual Machine Control Structure (VMCS) is a data structure that stores VM and host state information for mode transitions. It also controls which guest operations can cause VM exits.

Intel VT-x utilizes Extended Page Table (EPT) for implementing Second Level Address Translation (SLAT). Each guest kernel maintains its page table to translate Guest Virtual Address (GVA) to Guest Physical Address (GPA). The hypervisor manages EPT to map GPA to Host Physical Address (HPA).

VMs can use different I/O models, including software-based and hardware-based models, to access I/O devices. Software-based I/O models involve emulated devices or para-virtualized devices, while hardware-based I/O models include direct device assignment, Single Root I/O virtualization (SR-IOV) devices, and Scalable I/O virtualization (S-IOV) devices.

Intel VT for Directed I/O (VT-d) enables the isolation and restriction of device accesses to entities managing the device. It includes I/O device assignment, DMA remapping, interrupt remapping, and interrupt posting. With the support of VT-d, VMs can directly access physical I/O memory through virtual-to-physical address translation with the help of the IOMMU. VT-d also provides flexibility in I/O device assignments to VMs and eliminates the need for the hypervisor to handle interrupts and DMA transfers. Overall, VT-d enhances the performance and security of virtualized environments that require direct access to I/O devices.

VT \(\Rightarrow\) TDX. TDX is a VM-based TEE. It relies on the VT to provide isolation among TDs. As the hypervisor is no longer trusted in the new threat model, the functionalities of managing TDs have been enclosed within the TDX Module. The TDX Module and TDs run in the new SEAM VMX root/non-root mode with additional protection. TDX still leverages EPT to manage GPA-to-HPA translation. But currently, it maintains two EPTs for each TD, a protected one for private (encrypted) memory and another one for shared (unencrypted) memory. We provide a detailed explanation of the TDX’s architecture and the TDX Module in Sections 6.1 and 7.

It is worth noting that currently nested virtualization is not supported in TDX 1.0, which means that running VMs within a TD is not allowed. Attempting to use VMX instructions within a TD can result in Undefined Instruction (UD) exceptions. But the TD partitioning architecture specification draft [37] indicates that nested virtualization will be supported in TDX 1.5 in the future.

5.2 Intel TME/MKTME

Total Memory Encryption (TME) was first introduced with the Intel 11th Generation Core vPro mobile processor. This feature is designed to protect against attackers who have physical access to a computer’s memory and attempt to steal data. TME encrypts the entire computer’s memory using a single transient key. The key is generated at boot-time through a combination of hardware-based random number generators and security measures integrated into the system’s chipset. Memory encryption is performed by encryption engines on each memory controller. The encryption process uses the NIST standard AES-XTS algorithm with 128-bit or 256-bit keys.

MKTME [31] extends TME to support multiple keys and memory encryption at page granularity. For each memory transaction, MKTME extracts Host Key Identifier (HKID) from the physical memory address and selects a corresponding key to encrypt/decrypt memory. HKID occupies a configurable number of bits from the top of the physical address. The HKIDs range is set by the BIOS during system boot. MKTME allows for software-provided keys and introduces a new instruction, PCONFIG, for programming the key and encryption mode associated with a particular HKID. These ⟨ HKID, key⟩ tuples are stored in the Key Encryption Table (KET), which is held by each MKTME encryption engine. The keys in the KET never leave the processor and are never exposed to software. MKTME can be used in both native and virtualized environments. In the virtualized environments, hypervisors control the memory encryption for different VMs by attaching HKIDs to VM’s physical addresses in EPT.

MKTME \(\Rightarrow\) TDX. To use MKTME in the virtualized environments, the hypervisor must be trusted to control the memory encryption, which violates the new threat model for confidential computing. Therefore, in TDX, the TDX Module is responsible for controlling memory encryption for TDs. The HKID space has been partitioned to private HKIDs and shared HKIDs. The TDX Module ensures that a unique private HKID is assigned to each TD. Therefore, this HKID can be used to represent the identity of a specific TD. The private HKIDs can only be used for encrypting the private memory of TDs. The TDX Module still leverages MKTME to protect TD’s memory. More information about how TDX uses MKTME can be found in Sections 6.2 and 8.

5.3 Intel SGX

Intel introduced SGX [53] in 2015 with the 6th Generation Core processors to protect against memory bus snooping and cold boot attacks. It enables developers to partition their applications and protect selected code and data within enclaves. The memory of an enclave can only be accessed by authorized code. SGX uses hardware-based memory encryption to protect the enclave’s contents. Any unauthorized attempts to access or tamper with the enclave’s memory can trigger exceptions. SGX adds 18 new instructions into Intel’s ISA and enables secure offloading of computations to environments where the underlying host components (such as host application, host kernel, SMM, and peripheral devices) are untrustworthy. SGX’s security ultimately depends on the security of the firmware and microcode that implement its features.

The Enclave Page Cache (EPC) is a special memory region that contains the enclave’s code and data, where each page is encrypted using the Memory Encryption Engine (MEE). The Enclave Page Cache Map (EPCM) stores the page metadata, such as configuration, permissions, and type of each page. At boot time, keys are generated and used for decrypting the contents of encrypted pages inside the CPU. The keys are controlled by the MEE and never exposed to the outside. Thus, only this particular CPU can decrypt the memory. The CPU stores these keys internally and prevents access to them by any software. Additionally, privileged software out of enclaves is not allowed to read or write the EPC or EPCM pages.

SGX offers both local and remote attestation to verify the integrity and authenticity of enclaves. Local attestation is used to establish trust between two enclaves within the same platform, while remote attestation verifies the trustworthiness of an enclave to a third-party entity off the platform. In local attestation, an enclave can verify another enclave’s integrity and the genuineness of the underlying hardware platform. To do so, the first enclave generates a report and uses the identity information of the second enclave to sign it. The second enclave retrieves its Report Key and verifies the report using this Report Key. A third party may want to establish trust with a remotely executed enclave before provisioning it with secrets. In this scenario, remote attestation is necessary. To perform remote attestation, SGX utilizes a special architectural enclave known as the Quoting Enclave (QE). The QE is developed and signed by Intel. The QE receives a report from another enclave, locally verifies it, and transforms it into a remotely verifiable quote by signing it with the Attestation Key. The relying party can send this quote to the Intel Attestation Service (IAS), which verifies the quote to identify and assess the trustworthiness of the SGX enclave. The QE’s role is to provide a secure and trustworthy environment for the transformation of a report into a quote and to ensure the quote cannot be modified or falsified. Intel also provides DCAP [58], which is a composition of software packages, for data centers to deploy their own ECDSA attestation infrastructures for SGX enclave attestation.

Researchers have used SGX to provide secure containers (e.g., Scone [4]) and shielded execution for unmodified applications (e.g., Haven [6]). Graphene [63], an SGX-based framework, provides techniques for running unmodified applications as well as dynamic libraries inside SGX enclaves. Besides, SGX has a wide spectrum of applications ranging from the function encryption system (e.g., Iron [16]), source code partitioning to protect security-sensitive data and functions (e.g., Glamdring [51]), machine learning [20, 21, 55, 62], network security [7], fault-tolerant [8], encrypted data search (e.g., HardIDX [17]), secure databases (e.g., EnclaveDB [56]), secure coordination for distributed system (e.g., SecureKeeper [12]), and secure distributed computations (e.g., VC3 [59]).

Identifying vulnerabilities of SGX is another important line of research. Researchers also have identified a wide range of attack vectors targeting SGX, such as controlled-channel attacks [66, 67, 69, 71], cache attacks [11, 18, 54, 60], branch prediction attacks [15, 49], and speculative execution attacks [13, 48].

SGX \(\Rightarrow\) TDX. SGX and TDX protect memory at different granularities. But on the same platform, TDX and SGX are within the same TCB. Thus, they can locally attest to each other. TDX leverages the remote attestation mechanism provided by SGX. The attestation report of a TDX platform can be verified and signed within a QE. More details about TDX’s remote attestation can be found in Sections 6.4 and 9.

It is worth noting that at the moment, running an SGX enclave within a TD is not allowed, as invoking ENCLS/ENCLV instructions within a TD can lead to UD exceptions.

6.1 TDX System Architecture

Figure 1 illustrates the runtime architecture of TDX. It is composed of two key components: (1) TDX-enabled processors, which offer architectural functionalities like hardware-assisted virtualization, memory encryption/integrity protection, and the ability to certify TEE platforms, (2) TDX Module, an Intel-signed and CPU-attested software module that leverages the features of TDX-enabled processors to facilitate the construction, execution, and termination of TDs while enforcing the security guarantees. The TDX Module provides two sets of interface functions, host-side interface functions for a TDX-enlightened hypervisor and guest-side interface functions for TDs. It is loaded and executed in the SEAM Range, which is a portion of system memory reserved via UEFI/BIOS. The P-SEAM Loader, which also resides in the SEAM Range, can install and update the TDX Module. More information on the loading process of the TDX Module can be found in Section 7.1.

Fig. 1.

View Figure

Secure-Arbitration Mode (SEAM) is an extension of the VMX architecture and provides two new execution modes: SEAM VMX root mode and SEAM VMX non-root mode. A TDX-enlightened hypervisor operates in the traditional VMX root mode and utilizes the SEAMCALL instruction to call host-side interface functions (function names start with TDH) of the TDX Module. Upon execution of the SEAMCALL instruction, the logical processor (LP) transitions from the VMX root mode into SEAM VMX root mode and starts executing code within the TDX Module. Once the TDX Module has completed its task, it returns to the hypervisor in VMX root mode by executing the SEAMRET instruction.

On the other hand, TDs run in the SEAM VMX non-root mode. TDX supports the execution of unmodified user-level applications within a TD, much like in a standard VM. However, the guest OS kernel, illustrated as the TDX-enlightened OS in Figure 1, must undergo modifications to align with the underlying TDX platform, accommodating both the architectural paradigms and the security imperatives of TDX. These modifications include managing new TDX exceptions via an in-guest Virtualization Exception (VE) handler, implementing a hypercall-like mechanism for communication between a TD and the TDX Module, transitioning memory pages from private to shared for I/O operations, and integrating attestation support. The specific implementation details may vary depending on the OS type. For instance, the detailed implementation of the enlightened guest Linux kernel has been described in the kernel documentation [14]. TDs can trap into the TDX Module either through a TD exit or by invoking the TDCALL instruction. In both cases, the LP transitions from the SEAM VMX non-root mode into the SEAM VMX root mode and starts executing in the context of the TDX Module. The names of guest-side interface functions handling TDCALLs start with TDG. Details about the TDX context switches can be found in Section 7.5.

The confidentiality assurances offered by confidential computing render it a prime target for research on side-channel information leakage. The unveiling of a succession of micro-architectural attacks [5, 47, 52, 65, 68] exploiting the speculative execution of CPUs highlights a concerning issue: the isolation of security domains enforced in the architectural states may not be consistent as in the micro-architectural states. As TDX becomes more widely available in the market, it is expected to attract increased attention from security researchers. Our primary emphasis lies in examining the existing defenses integrated into the TDX Module to address known attack vectors. For detailed information, please refer to Section 7.8.

6.2 TDX Memory Protection

TDX leverages VMX to enforce memory isolation for TDs. Similar to legacy VMs, TDs are unable to access the memory of other security domains, such as SMM, hypervisors, the TDX Module, and other VMs/TDs. With VMX, hypervisors maintain EPTs to enforce memory isolation. However, since hypervisors are no longer trusted, TDX has moved the tasks of memory management to the TDX Module, which controls the address translation of TD’s private memory.

A more intriguing aspect of TDX’s security model is its protection of TD’s memory from privileged software, corrupted devices, and unprincipled administrators on the host. TDX achieves this by implementing access control and cryptographic isolation. Access control prevents other security domains on the same computer from accessing a TD’s data. Cryptographic isolation is utilized to prevent malicious DMA devices or adversaries with physical access to the main memory from directly reading or corrupting TD’s private memory.

Memory Partitioning. With TDX enabled, the entire physical memory space is partitioned into two parts: normal memory and secure memory. The sensitive data of TDs, including the private memory, virtual CPU state, and its associated metadata, should be stored in secure memory. TDs can also specify memory regions as shared memory for I/O, which is not protected through TDX. Thus, these memory regions belong to normal memory. All other software, which is not executing in the SEAM mode, belongs to normal memory and is not allowed to access secure memory, regardless of its privilege level. The memory controller, an architectural component inside the processor, enforces memory access checks.

To make a physical page part of the secure memory, the TD Owner Bit is enabled (Section 8.2). Each TD Owner Bit is associated with a memory segment corresponding to a cache line.¹ The TD Owner Bits are stored in the Error Correction Code (ECC) memory associated with these segments. The TDX Module controls the conversion of physical memory pages to secure memory by attaching private HKIDs to their physical addresses. The HKID is encoded in the upper bits of the physical address. The set of private HKIDs is controlled by TDX and can only be used for TDs and the TDX Module. When the memory controller writes to a physical address with a private HKID, it sets the TD Owner Bit to 1. When it writes to an address that does not have a private HKID, it clears the TD Owner Bit. Access control is enforced on each cache line read. The read request passes through the memory controller, which permits only processes executing in SEAM mode to read a cache line with a TD Owner Bit set to 1. Any read request not in the SEAM mode receives all zeros when trying to read such a cache line.

When building a TD, the (untrusted) hypervisor selects the memory pages from the normal memory to become part of the secure memory. The TDX Module gradually moves these pages to the secure memory. It uses them for the metadata (Section 7.4) and the main memory of each TD. A TD must explicitly accept these pages before they can be used for its main memory. The TDX Module performs sanity checks of the secure memory setup by maintaining a Physical Address Metadata Table (PAMT), which is described in more detail in Section 7.7.

Memory Confidentiality. TDX leverages MKTME (Section 5.2) for encrypting TD’s private memory and its metadata. MKTME is responsible for transparent memory encryption and decryption of data passing through the memory controller. The TDX Module programs the keys used by the MKTME to encrypt specific cache lines when they are written to memory. The keys are associated with the HKIDs embedded in the physical addresses. MKTME decodes HKIDs and uses the referenced cryptographic keys to perform the cryptographic operations.

MKTME stores cryptographic keys in its internal memory, never exposing them to the outside. The cryptographic keys can only be referenced by their HKIDs. When building a new TD, the hypervisor selects an unused private HKID, and the TDX Module requests the processor to generate a new cryptographic key related to this HKID. The TDX Module binds this ⟨ HKID, key⟩ tuple to the TD. It guarantees that the memory of each TD is encrypted with a different cryptographic key.

MKTME encrypts memory at the cache line granularity using AES-128 XTS cryptography when the cache line is being written back to main memory. The encryption can prevent some physical attacks, like the cold boot attack. Please see Section 8.1 for more details on MKTME and HKIDs.

Memory Integrity. TDX provides two distinct mechanisms for ensuring memory integrity: Logical Integrity (Li) and Cryptographic Integrity (Ci).

Li protects the integrity against unauthorized writes at the software level by using the TD Owner Bit. Since the TDX only allows the use of private HKIDs in the SEAM mode, any unauthorized writes to a TD’s private memory from outside the SEAM mode will clear the TD Owner Bit. When the modified private memory is read, the cleared TD Owner Bit will trigger an exception. However, this feature cannot prevent adversaries from bit flipping (e.g., via a Rowhammer attack [45]) the main memory.

Ci is a more advanced mechanism that addresses the limitations of Li. In addition to the TD Owner Bit, Ci also computes a Message Authentication Code (MAC) on a cache line when it is being written back to memory. The MAC is computed using a 128-bit MAC key generated during system initialization and is stored as part of the memory metadata during the write-back. When the memory is read, the MAC is recalculated. Any tampering with the memory content will be detected by Ci if the TD Owner Bit or the recalculated MAC mismatch with the stored metadata. However, neither Li nor Ci can detect the memory replay attack if the adversary can roll back both the memory content and the metadata. We provide a more detailed technical discussion of the memory integrity protection in Section 8.2.

6.3 TDX I/O Model

According to the TDX threat model, hypervisors and peripheral devices are considered untrusted and are prohibited from directly accessing the private memory of TDs. It is the responsibility of TDs and their owners to secure I/O data before it leaves the trust boundary. This requires sealing the I/O data buffers and placing them in shared memory, which is identified by the shared bit in the GPA. Hypervisors or peripheral devices can then move the data in and out of the shared memory. This necessitates modifications to the guest kernel to support this I/O model. Furthermore, all I/O data that is transferred into the TDs from hypervisors or peripheral devices must be thoroughly examined and validated, as it is no longer considered trustworthy.

In the Linux guest support for TDX, all MMIO regions and DMA buffers have been mapped as shared memory within the TDs. The Linux guest is enforced to use SWIOTLB to allocate and convert DMA buffers in unified locations. To protect against malicious inputs from I/O, only a limited number of hardened drivers [30] are allowed within TDs.

6.4 TDX Attestation

Remote attestation is a method for verifying the identity and trustworthiness of a TEE. The attester can provide proof to a challenger to show that computations are being executed within protected domains. The challenger validates the evidence by checking the digital signatures and comparing the measurements to reference values.

On a TDX-enabled machine, the attester operates within a TD and is responsible for handling remote attestation requests. When a request is received from a challenger, such as a tenant, the attester provides evidence of proper instantiation of the TD through the generation of a TD quote. This quote, which serves as the evidence, is produced by the TDX module and signed by the Quoting Enclave. It contains measurements of the TDX’s TCB and the software components loaded in the TD. The quote also includes a certificate chain anchored by a certificate issued by Intel. Upon receipt of the quote, the challenger verifies its authenticity by checking the quote and determining if the attester is running on a genuine TDX-enabled platform and if the TD has the expected software measurements. If the quote is successfully validated, the challenger can proceed to establish a secure channel with the attester or release secrets to the attester. We provide a more detailed technical discussion of remote attestation in Section 9.

6.5 Future Features

Live migration and trusted I/O are crucial features for confidential VMs but are currently not supported in TDX 1.0. However, according to documents [35, 36, 38], Intel is planning to include the support for live migration in TDX 1.5 and trusted I/O in TDX 2.0. These plans are still in progress and may be subject to change in the future. Here we provide a brief overview of these two features and explain their design.

Live Migration. Live migration is an essential feature for cloud service providers as it enables them to transfer running VMs from one physical host to another without any service interruptions. This functionality is important for maintenance tasks such as hardware upgrades, software patches, and load balancing. However, migrating a TD is more complex than migrating a traditional VM due to the security concerns of confidential computing. Since the hypervisor is considered untrusted, it is not allowed to directly access and transfer the CPU state and private memory of the TD from the source to the destination platform. Furthermore, tenants should have the ability to define and enforce migration policies. For instance, if the destination platform does not meet the TCB requirements specified in the policy, the migration should be canceled.

Intel introduces Service TDs to expand the trust boundary of the TDX Module. Rather than making the TDX Module overly complex and bloated, it is more convenient and flexible to add customized and specialized functionalities into a Service TD. A Service TD can be bound to regular TDs via the TDX Module with access privileges to their assets.

Migration TD (MigTD) is a Service TD that is specifically designed for live migration. The entire live migration session is under the control of the TDX Module and the MigTDs. The untrusted hypervisor, which is controlled by the cloud service provider, is only responsible for transferring the encrypted TD’s assets over networks. These assets include the TD’s metadata, CPU state, and private memory, and are protected by a Migration Session Key (MSK) that is only accessible by the MigTDs and TDX Module.

Both the source and destination platforms have a running MigTD. MigTDs are respectively bound to the source TD (to be migrated) and the destination TD (initially as a TD template waiting for migration). The MigTDs are responsible for remote attestation between source and destination platforms and evaluate their TCB levels based on security policies. Once the platforms are deemed acceptable for migration, a secure channel is established between the two MigTDs. The source MigTD generates an MSK, which is shared to the destination MigTD through this secure channel. Both MigTDs program the MSK into the corresponding TDX Modules. The source TDX Module exports and encrypts the TD’s assets with the MSK, while the destination TDX Module decrypts the assets with the same key and imports them into the destination TD. It is worth noting that the source and destination TDs have their HKIDs assigned independently, thus protected with different TD private keys.

Trusted I/O. A computer consists of various functional components. However, confidential computing has conceptually shattered the unified trust model. As a result, each component, made by different vendors, can no longer trust each other. This creates a serious impediment to efficient I/O, as untrusted devices cannot read and write data in the private memory of TEEs. To address this issue, Intel has proposed TDX Connect in TDX 2.0, aiming to extend the trust from a TD to external devices. This requires changes to the devices and the TDX platform to use a compatible protocol to establish mutual trust and enable secure communication channels. The key principle is that a TD and a device should be able to securely exchange and verify their identities and measurements. Additionally, the data paths between a TD and a device are not trusted and may be vulnerable to interception by attackers. Therefore, an end-to-end secure channel is necessary to protect the data transmitted between a TD and a device. The detailed protocols for TDX Connect can be found in the proposals [35, 38].

7.1 Loading TDX Module

Figure 2 illustrates the two-stage process of loading the TDX Module. The process begins with the loading of the Intel Non-Persistent SEAM Loader (NP-SEAM Loader), which is an Intel Authenticated Code Module (ACM). ACMs are Intel-signed modules that run within the internal RAM of the processor. The NP-SEAM Loader is authenticated and loaded by the Intel Trusted Execution Technology (TXT) [19] through the GETSEC[ENTERACCS] function. The NP-SEAM Loader contains the image of the Intel Persistent SEAM (P-SEAM) Loader, which is then verified and loaded by the NP-SEAM Loader. The P-SEAM Loader is then responsible for installing or updating the TDX Module.

Fig. 2.

View Figure

It is important to note that both the P-SEAM Loader and the TDX Module are loaded in the SEAM Range, which is a portion of system memory reserved via UEFI/BIOS. The range’s base address and size are specified by the IA32_SEAMRR_PHYS_BASE and IA32_SEAMRR_PHYS_MASK MSRs. This range is partitioned into Module_Range for the TDX Module and P_Seamldr_Range for the P-SEAM Loader. Both modules run in the SEAM VMX root mode and use SEAMCALL/SEAMRET to interact with external software. The NP-SEAM Loader, P-SEAM Loader, and TDX Module are all provided and signed by Intel, establishing a chain of trust to bootstrap the TDX Module.

The P-SEAM Loader provides a SEAMCALL interface function seamldr_install for loading the TDX Module. The TDX Module’s image is pre-loaded into a memory buffer (not in the SEAM Range). The physical addresses of the buffer and a seam_sigstruct (signature of the TDX Module) are passed as the parameters to the seamldr_install. The seam_sigstruct contains the hash value and the Security Version Number (SVN) of the TDX Module, the number of per-LP stack pages, the number of per-Logical Processor (LP) data pages, and the number of global data pages. These numbers are used by seamldr_install to determine the physical/linear addresses and the sizes of the TDX Module’s various memory regions.

The seamldr_install must be called on all LPs serially. When it is called on the first LP, an installation session starts. On each LP, seamldr_install checks that the LP is not already in an installation session (started by another LP), and clears the LP’s VMCS cache. When seamldr_install is called on the last LP, it does the following:

In addition to the SEAMCALL to install the TDX Module, the P-SEAM Loader also provides other interface functions to shut down itself and retrieve the loader’s system information.

7.2 Memory Layout of TDX Module

Here we discuss the physical and linear memory layout for the TDX Module, respectively.

Physical Memory Layout. Figure 3 depicts the physical memory layout of the TDX Module within the Module_Range. The layout starts with a 4 KB page that holds the Sysinfo_Table of the TDX Module. The Sysinfo_Table consists of 2 KB platform information populated by Mcheck from the NP-SEAM Loader and the next 2 KB populated by the P-SEAM Loader with the TDX Module’s information, such as the SEAM Range base address and size, the base linear addresses of the memory regions, number of LPs, and range of private HKIDs. After the Sysinfo_Table, there is the per-LP VMCS region. Each LP has a 4 KB SEAM Transfer VMCS (see Section 7.5). Following the per-LP VMCS region, there is the data region, which is partitioned into per-LP data region and a global data region. Next, there is the TDX Module’s 4-level page table, followed by the per-LP stack regions, and finally, the code region for the TDX Module’s executable code. Linear Memory Layout. The TDX Module has its own linear address space and maintains a page table to translate addresses. Figure 4 illustrates the layout of the TDX Module’s linear address space, which is established by the P-SEAM Loader through the construction of the TDX Module’s page table. To prevent memory corruption attacks, the P-SEAM Loader randomizes bits 34 to 46 of the linear addresses, which are represented by the boxes in Figure 4. The linear addresses and the sizes of all regions are recorded in the fields of the Sysinfo_Table. The Page Table Entriess (PTEs) for code, stack, data, and Sysinfo_Table can be statically populated in advance and require no changes to the page table at runtime. However, the Keyhole region serves to map data passed from external software dynamically during the execution of the TDX Module. This requires the addition of the Keyhole-Edit region to allow runtime editing of the PTEs for the Keyhole’s mapping. A detailed discussion of the Keyhole and Keyhole-Edit regions can be found in Section 7.6.

Fig. 3.

View Figure

Fig. 4.

View Figure

7.3 Initialization and Configuration of TDX Module

After the TDX Module is loaded, the host kernel is responsible for initializing and configuring the TDX Module. The host kernel makes a SEAMCALL[TDH.SYS.INIT] to globally initialize the TDX Module. Then, the host kernel makes a SEAMCALL[TDH.SYS.LP.INIT] on each LP to check and initialize per-LP parameters, such as Keyholes (Section 7.6), data regions, and stack regions (Section 7.2). Next, the host kernel allocates a global private HKID and passes it to the TDX Module through a SEAMCALL[TDH.SYS.CONFIG], which also initializes the Trust Domain Memory Region (TDMR) (Section 7.7). The SEAMCALL[TDH.SYS.KEY.CONFIG] on each processor package generates a TDX global private key and binds the key with this HKID. This key is used to encrypt memory that holds PAMT and Trust Domain Root (TDR) (Section 7.4) of each TD. Finally, the host kernel calls SEAMCALL[TDH.SYS.TDMR.INIT] multiple times to gradually initialize the PAMT (Section 7.7) for each TDMR.

7.4 Metadata for TDs

The TDX Module is responsible for managing the entire life cycle of TDs. As such, it needs to maintain metadata for each TD instance. The TDX Module ensures that the memory encryption is applied to the metadata to prevent the hypervisor from accessing or modifying it.

Each TD’s metadata consists of the following control structures: TDR, Trust Domain Control Structure (TDCS), Trust Domain Virtual Processor State (TDVPS), and Secure EPT (SEPT). Figure 5 illustrates the relationships between these control structures. TDR. TDR is the initial structure that is created at the inception of a TD and is destroyed when the TD is terminated. During the entire life cycle of the TD, SEAMCALLs use the physical address of the corresponding TDR to refer to the TD. The TDR comprises the key information for memory encryption and references to the TDCX pages (physical memory pages for the Trust Domain Control Structure (TDCS)). As the TDR is created before the TD’s private key is generated, it is protected with the global private key of the TDX Module. The subsequent metadata (TDCS, Trust Domain Virtual Processor State (TDVPS), and Secure EPT (SEPT)), along with the TD’s memory pages, can be associated with the TDR through the owner attribute in the PAMT (Section 7.7).

Fig. 5.

View Figure

TDCS. TDCS is a control structure that manages the operations and stores the state at the scope of a TD. It consists of four continuous TDCX memory pages, each allocated for a specific purpose, such as TD’s management structures, MSR bitmaps, SEPT root page, and a special zero page. TDCS is encrypted with the TD’s private key, which is generated when the TDR is created.

TDVPS. TDVPS is a control structure for each virtual CPU of a TD. It consists of six memory pages, starting from a TDVPR page that contains references to multiple TDVPX pages. The first TDVPR page holds the fields for VE information, virtual CPU management, guest state, and guest MSR state. The second page is for the TD Transfer VMCS (Section 7.5), which controls the TD’s entry and exit. The third page is a Virtual APIC (VAPIC) page, followed by three pages for guest extension information. Like the TDCS, the TDVPS is also protected by the TD’s private key.

SEPT. For legacy VMs, hypervisors manage address translations from GPA to HPA using EPT. However, in TDX, guest address translations must be protected from untrusted hypervisors. To achieve this, TDX has two types of EPT: SEPT and Shared EPT. SEPT is used to translate addresses of a TD’s private memory and is protected by the TD’s private key. The reference to the SEPT and the SEPT root page are stored in the TDCS. Shared EPT, on the other hand, is used to translate addresses for memory explicitly shared by the TD with a hypervisor, such as in the case of virtualized I/O. It remains under the control of the hypervisor. The guest kernel in the TD can determine which memory pages to share by setting the shared bit in the GPA. Shared memory pages are not encrypted with the TD’s private key.

7.5 Context Switches

There are two types of context switches for TDX: the first occurs between the hypervisor and the TDX Module, while the second occurs between TDs and the TDX Module. We delve into each of these in more detail.

Hypervisor \(\leftrightarrow\) TDX Module. In TDX, a hypervisor is prohibited from directly managing TDs. Instead, it must interact with the TDX Module through SEAMCALL interface functions. When a SEAMCALL is made, the processor transitions from the VMX root mode to the SEAM VMX root mode. The TDX Module’s SEAM Transfer VMCS is loaded. This SEAM Transfer VMCS is set up by the P-SEAM Loader for each LP and is stored within the Module_Range.

The repurposing of VMCS for context switches between the hypervisor and the TDX Module may seem confusing initially, as the hypervisor is not a “guest VM” and the TDX Module is not a “host hypervisor.” We can disregard the guest/host concept and only view the SEAM Transfer VMCS as a means of switching the execution context between the hypervisor and the TDX Module.

Figure 6 depicts the location and layout of the SEAM Transfer VMCS regions in the Module_Range, which begins at the SEAMRR_Base. The first 4 KB page in the Module_Range is the Sysinfo_Table. Starting from SEAMRR_Base + 4 KB, there is an array of per-LP SEAM Transfer VMCS regions. Each region is a 4 KB page. This array is indexed by the identifier of the LP, denoted by LP_ID.

Fig. 6.

View Figure

When a SEAMCALL instruction is executed, the processor searches for the VMCS address based on the current LP_ID. The address is determined by: SEAMRR_Base + 4 KB + (LP_ID × 4 KB). The TDX Module’s state is stored in the Host State Area of the VMCS. For instance, the host RIP is set to the tdx_seamcall_entry_point of the TDX Module, and the host CR3 is set to the physical address of the TDX Module’s PML4 base. Moreover, the host FS_BASE is set to the linear address of the Sysinfo_Table and the host GS_BASE is set to the per-LP data region.

When the LP transitions into the TDX Module through the SEAMCALL instruction, the information stored in the SEAM Transfer VMCS is loaded onto the processor. Therefore, the FS_BASE and GS_BASE now point to the Sysinfo_Table and the local data region of the TDX Module, respectively. The CR3 register points to the TDX Module’s page table, thereby switching Memory Management Unit (MMU) to operate in the TDX Module’s linear address space. The TDX Module starts to handle SEAMCALLs and dispatch them to corresponding interface functions.

TD \(\leftrightarrow\) TDX Module. In traditional virtualization, the hypervisor handles VM exits, which are controlled by the VM’s Transfer VMCS. Each VMCS is associated with one virtual CPU and stores the virtual CPU state for recovering the guest execution in the next VM resume. However, this operation leaks the virtual CPU state as VMCS is visible to hypervisors. In TDX, synchronous TDCALLs or asynchronous TD exits are designed to trap into the TDX Module. This is controlled by the TD Transfer VMCS, which is set up when a virtual CPU of a TD is created and stored in the TD’s TDVPS. The TDVPS is encrypted with the TD’s private key (Section 7.4). Therefore, the TD Transfer VMCS is inaccessible to untrusted hypervisors. When a TD calls a TDCALL or triggers a TD exit, the LP loads the state of the TDX Module stored in the TD Transfer VMCS to switch context.

In TDX, certain TD exits cannot be fully handled by the TDX Module and instead require a hypervisor to emulate certain operations, such as port I/O, HLT, CPUID, and more. However, traditional hypervisors have access to the entire virtual CPU states and memory, exposing more information than necessary to handle these exits. TDX addresses this issue by introducing a new mechanism for handling TD exits. All TD exits first trap into the TDX Module, which injects a VE into the TD to handle the exit. The TD’s guest kernel includes a corresponding VE handler that prepares a minimized set of parameters and invokes a TDCALL to re-enter the TDX Module. At this point, the TDX Module can safely ask the hypervisor to handle the requests with minimal exposure to sensitive information.

7.6 Keyholes

All memory buffers passed through SEAMCALLs use their physical addresses as references. The TDX Module must map these buffers into its own linear address space to access them. This mapping process is facilitated by the Keyhole and Keyhole-Edit regions, which serve as temporary “leases” of linear addresses.

The Keyhole region is a reserved linear address range specifically for address mapping. The region is comprised of an array of Keyholes. This array is further divided into 128-Keyhole segments, with each segment assigned to one LP. The TDX Module organizes free Keyholes in an LRU list when setting up per-LP data structures. Each Keyhole corresponds to a 4 KB-aligned linear address and links to a physical memory page. Since multiple memory buffers can exist within the same memory page, each Keyhole maintains a reference count to track the number of referenced buffers on the page.

When the TDX Module is installed by the P-SEAM Loader, all the linear addresses of Keyholes are mapped to an empty physical address. This is achieved by setting all the leaf-level PTEs for the Keyhole region in the TDX Module’s page table to zero. Simultaneously, the physical addresses of the corresponding PTEs for the Keyholes are mapped to the Keyhole-Edit region. This enables the TDX Module to locate and modify the Keyhole’s address mappings in its page table during runtime.

When processing a SEAMCALL that refers to an external memory buffer with a physical address, the TDX Module checks if the buffer’s memory page is already mapped by a Keyhole. If so, it increments the Keyhole’s reference count and returns the mapped linear address. If not, it selects a free Keyhole from the LRU list and maps the linear address of this Keyhole to the page table by updating the corresponding PTE referenced in the Keyhole-Edit region. Once the buffer is mapped, the TDX Module can access it using the Keyhole’s linear address. At the end of each SEAMCALL, the reference counts of corresponding Keyholes decrement, and any non-referenced Keyholes return to the LRU list.

7.7 Physical Memory Management

The TDX Module manages physical memory by using a set ofTDMRs and their control structures, PAMTs. TDMRs are constructed by the hypervisor based on a list of Convertible Memory Regions (CMRs), which are the memory regions that can be used for TD’s private memory or metadata. These regions are subject to MKTME encryption and TDX memory integrity protection. This list of CMRs is prepared by the UEFI/BIOS.

Each TDMR is a single range of physical memory that is 1 GB-aligned and has a size that is an integral multiple of 1 GB, but does not necessarily need to be a power of two. Two TDMRs cannot overlap. A TDMR may contain reserved areas that cannot be used by the TDX Module. A reserved area is an array of 4 KB-aligned memory pages (each page is 4 KB). Memory in a TDMR, except for the reserved areas, must be convertible. It should be noted that TDMR configuration is managed by software without using hardware range registers.

The TDX Module uses PAMT to track page attributes of each physical memory page in a TDMR. The attributes contain the information about the page owner, page type, and page size. The page attributes allow the TDX Module to ensure that a physical memory page in a TDMR has a proper type and is only assigned to at most one TD. When a page is assigned to a TD’s private memory, the TDX Module can check whether the page size in the SEPT and PAMT are consistent.

A PAMT is divided into blocks, where each block tracks page addresses within the 1 GB size range. Each block has three levels to track metadata for pages with sizes 4 KB, 2 MB, and 1 GB, respectively. The first level tracks a single 1 GB page, the second level tracks 512 2 MB pages, and the third level tracks \(512\times 512\) 4 KB pages. Given a physical address, the TDX Module can perform a PAMT hierarchical walk to retrieve its page attributes for a sanity check.

The TDX Module manages the data structure by updating the attributes of each page it uses during runtime. Any operation that requires accessing, removing, or adding a page causes the TDX Module to walk through PAMT to adjust the page attributes and check corresponding access rights. The memory for PAMT is allocated by the hypervisor and is encrypted with the TDX Module’s global private key.

7.8 Side Channel Mitigation

Some of the known CPU vulnerabilities have been addressed in hardware fixes. During the initialization of the TDX Module, it reads the IA32_ARCH_CAPABILITIES MSR and verifies a set of capability bits, including rdcl_no, irbs_all, mds_no, if_pschange_mc_no, taa_no, misc_package_ctls, skip_l1dfl_vmentry, energy_filtering_ctl, and tsx_ctrl. Each bit corresponds to a specific vulnerability and indicates whether the processor is susceptible to that particular attack. For example, rdcl_no indicates that the processor has been patched against the Rogue Data Cache Load (RDCL) [2] vulnerability. The detailed list can be found in Intel’s documentation [34]. Any missing capability will lead to the failure of initializing the TDX Module.

Furthermore, to counteract the Bounds Check Bypass (BCB) [1] vulnerability, a software-level mitigation strategy is deployed, employing memory barriers such as LFENCE to halt speculation at specific locations within the TDX Module where untrusted inputs might be encountered. These critical locations comprise the TD exit entry point, SEAMCALL entry point, Keyhole manager, among others. However, this approach remains manual and ad-hoc in nature. A more principled approach is needed to determine the essential placement of memory barriers.

8.1 HKID Space Partitioning

The HKID space is partitioned once during the boot process into two ranges, private HKIDs and shared HKIDs. Only software in the SEAM mode, namely the TDX Module and TDs, can read and write memory whose contents are encrypted by keys associated with private HKIDs. Keys associated with shared HKIDs can be used to encrypt memory outside the SEAM mode, such as the memory of legacy VMs and the host kernel.

When the hypervisor requests the TDX Module to establish a TD, it allocates a private HKID for the TD. The TDX Module, using the PCONFIG instruction, asks MKTME to generate a unique random key for the HKID. This key is called the TD’s ephemeral private key. It is used to encrypt all the private memory and metadata of the TD and is never exposed outside MKTME. This ⟨ HKID, key⟩ binding is valid for the lifetime of the TD.

A physical memory page associated with a HKID stores the HKID in the upper bits of the page’s physical address, as shown in Figure 7. At boot time, the number of bits used for HKIDs (MK_TME_KEYID_BITS) and the number of bits used for private HKIDs (TDX_RESERVED_KEYID_BITS) are set in the IA32_TME_ACTIVATE MSR. The IA32_MKTME_KEYID_PARTITIONING MSR can be used for reading the numbers of private and shared HKIDs. Intel reserves a range of upper bits in the 64-bit physical address. The HKID uses these reserved bits. The remaining bits following the HKID correspond to the physical memory address. The upper bits of the HKID field, the TDX_RESERVED_KEYID_BITS are reserved for private HKIDs. For example, if MK_TME_KEYID_BITS is 6 and TDX_RESERVED_KEYID_BITS is 4, then HKIDs from 0 to 3 are shared, and HKIDs from 4 to 63 are private.

Fig. 7.

View Figure

The hypervisor and the TDX Module configure the memory encryption by setting the HKID in the upper bits of the physical address of a memory page. The hypervisor can only use shared HKIDs, while the TDX Module can use both shared and private HKIDs. An exception will be raised if any software executing outside SEAM mode tries to access memory through a physical address with a private HKID.

8.2 TD Memory Integrity Protection

TDX always protects the integrity of the TD’s private memory content. This protection is required because an entity outside the SEAM mode,e.g., a malicious hypervisor or a DMA device, can write to the TD’s private memory. TDX cannot prevent such modification, but it can detect and flag it. It prevents a TD or the TDX Module from reading the tampered content. To detect such tampering, TDX supports two memory integrity modes that can be configured on a system:

Both Li and Ci apply to a physical memory segment with the size of a cache line and whose address is cache line aligned. Ci can detect modifications made by direct physical access to the memory or bit flips, such as the Rowhammer attack [45], which Li cannot detect.

In addition to Li and Ci, if a program outside the SEAM mode reads the private memory of a TD or the TDX Module, the read will always return zeros. This is to prevent ciphertext cryptanalysis and side channels in which a program outside the SEAM mode could determine whether a program in the SEAM mode changes the memory content.

If a TD or the TDX Module writes to a memory segment belonging to a TD’s private memory, the corresponding TD Owner Bit is set to 1. Due to the way a TD’s memory is set up, all TD Owner Bits of a TD’s private memory should be set to 1. However, if an entity outside the SEAM mode writes to a segment belonging to the private memory, the corresponding TD Owner Bit is cleared to 0. Later, when the TD or the TDX Module reads the segment, the segment is marked as poisoned. If the reader is the TD, this poisoned marking causes a TD exit for the TD. The TDX Module can capture this TD exit and put the TD into a fatal state, which prevents any further entry into the TD and leads to the tearing down of the TD. If the TDX Module reads the poisoned content, the TDX Module and the TDX’s hardware extension in the processor are marked as disabled. Any further SEAMCALLs leads to the VMFailInvalid error.

If Ci is enabled, the processor generates a 128-bit MAC key during system initialization. On each write, TDX uses this key to calculate and store a 28-bit MAC in the ECC memory corresponding to the cache line. On each read, the memory controller recalculates the MAC and compares it with the value read from the ECC memory. The mismatch indicates integrity or authenticity violation and results in the cache line being marked as poisoned. The MAC is calculated over (1) the ciphertext (encrypted content of the cache line), (2) the tweak values used for AES-XTS encryption, (3) the TD Owner Bit, and (4) the 128-bit MAC key.

9.1 Attestation Process

Several steps are involved when generating and extending a local report of a TD to enable remote attestation. The first step is to take measurements of the loaded software during the build-time and runtime of the TD. The next step is to retrieve the TD’s measurements and platform TCB information, i.e., generating a TD report. The final step is to derive a quote from the TD report. A third party can use the quote to verify whether the TD runs on a genuine TDX platform with the expected TCB versions and software measurements.

Taking Measurements. TDX provides two types of measurement registers for each TD: a build-time measurement register called Measurement of Trust Domain (MRTD) and four Runtime Measurement Registers (RTMRs). These measurement registers are comparable to the TPM’s PCRs, see Table 3 derived from [41] showing the mapping between TDX measurement registers and TPM PCRs.

The MRTD contains a measurement of the TD build process. At the TD creation, when the hypervisor adds initial memory pages to the TD, it extends the MRTD in the TDCS with measurements of these pages. The hypervisor calls SEAMCALL[TDH.MEM.PAGE.ADD] to add a page to the TD’s memory and to initiate the measurement of the page. It first calculates a SHA384 update over the ASCII string “MEM.PAGE.ADD” and the GPA of the page. Then it extends the MRTD with the hash value. Once the page is copied into the TD’s memory, i.e., mapped and available in the SEPT, the hypervisor calls SEAMCALL[TDH.MR.EXTEND] multiple times to measure the content of the page. The page is measured in blocks of 256 B. For each block, the extension operation first calculates a SHA384 update over the ASCII string “MR.EXTEND” and the GPA of the block. Second, it calculates another SHA384 update over the content of this block. Both hash values extend the MRTD. These initial pages contain the TD’s virtual firmware. The MRTD’s measurement does not include pages containing control structures, i.e., TDR, TDCS, and TDVPS, nor the SEPT. After the initial set of pages is added, the hypervisor finalizes the MRTD measurement using the SEAMCALL[TDH.MR.FINALIZE]. This disables future operations to extend the MRTD. For example, when initializing a TD, KVM, as a Linux hypervisor, measures the TDVF [33] (virtual firmware of a TD) code into the MRTD.

RTMRs are general measurement registers labeled 0 through 3 for TD’s runtime measurements. A TD can use these registers to provide a measured boot, i.e., measuring all software loaded after booting. These measurement registers are initialized to zero. The TD calls TDCALL[TDG.MR.RTMR.EXTEND] to extend the content of a RTMR. The arguments of this call consist of an index to the measurement register and a 64 B-aligned physical address of the 48 B extension buffer containing the value. This call calculates a SHA384 hash over the current value of the given index measurement register concatenated with the value in the extension buffer, i.e., RTMR[index] = SHA384(RTMR[index] || value). For example, TDVF measures the static/dynamic configuration data into the RTMR[0] and the OS kernel, boot parameters, and initrd into the RTMR[1].

Generating TD Reports. A report is generated inside a TD. The TD calls TDCALL[TDG.MR.REPORT], which is the TDX Module’s report function, with a newly initialized report structure and some user report data, named REPORTDATA. The REPORTDATA is 64 B and it can be used as a nonce to verify freshness of the TD report. To service the call, the TDX Module invokes the newly added SEAMOPS[SEAMRERPORT] instruction with the TD’s measurements and REPORTDATA. The CPU adds TCB information related to the SEAM and returns a TD report. This TD report is integrity protected using an HMAC key maintained by the CPU. The HMAC key is only available to the CPU. The TDX Module returns this report to the TD. Using the EVERIFYREPORT2 instruction, an enclave can verify the report on the same platform but not off the platform.

Figure 8 illustrates a TD report consisting of three components: REPORTMACSTRUCT, TEE_TCB_INFO, and TD_INFO. The REPORTMACSTRUCT structure contains header information specifying the structure type as TDX and has fields for CPU SVN and hashes of the TEE_TCB_INFO and TD_INFO components. It also includes the REPORTDATA provided as the input to the TDCALL[TDG.MR.REPORT] function. Finally, there is an HMAC over the entire header that protects TEE_TCB_INFO and TD_INFO components. The TEE_TCB_INFO structure contains information about the SVN and measurements of the TDX Module. The TD_INFO contains TD attestable properties. Examples of these properties include the initial TD configuration and values of the measurement registers.

Fig. 8.

View Figure

Deriving Quotes. To enable verification off the platform by a third party, the TD report must be converted into a quote. TDX tends to reuse the remote attestation mechanism of SGX. A TD makes a call to request the QE running on the host platform to sign the TD report. This call can be implemented over a VSOCK or a TDCALL[TDG.VP.VMCALL], depending on how the quoting service is provided on the platform. The QE calls the EVERIFYREPORT2 instruction to verify the TD report’s HMAC. If this call is successful, the QE signs the TD report using its certified attestation key to generate a quote. This operation basically replaces the MAC integrity protection of the TD report with the digital signature protection, allowing any party to verify the provenance and integrity of the quote using public key certificates. Section 9.2 describes the operations for enabling the local attestation infrastructure on the platform to support remote attestation.

9.2 Platform Setup

Configuring the attestation infrastructure involves registering the platform with the Intel PCS, running architectural enclaves for generating quotes, and retrieving certificates required for verifying quotes. Intel extends the existing DCAP [58] to support remote attestation for TDX.

Registration. On multiple-package platforms, platform keys are derived at platform assembly time. These keys are shared between CPU-packages and are encrypted by the CPU’s unique hardware key. Provisioning Certification Keys (PCKs) are derived from the platform keys and used for certifying (signing) attestation keys. Since PCKs are not recognized by the attestation infrastructure, they must be registered with Intel PCS.

To register a platform, we need to run the PCK Cert ID Retrieval Tool to extract a manifest from the platform. This manifest contains information on CPU packages, e.g., CPU ID (128-bit), SVN, and hardware TCB information. When the Intel PCS gets the register server request, it checks whether CPUs and TCB are in good standing before issuing a PCK certificate. The manifest is signed with keys derived from the CPU package’s hardware keys and the Intel PCS checks whether these signatures are valid. If registration succeeds, the Intel PCS returns an Intel-issued certificate for the PCK.

Typically in DCAP, a Provisioning Certification Caching service (PCCS) runs on the host platform to facilitate PCK certificate retrieval. This service can run anywhere. It forwards the PCK requests from the PCK Cert ID Retrieval Tool to the Intel PCS and caches the returned PCK certificates locally. The Intel PCS also provides certificates and revocation lists for PCKs in all genuine Intel platforms. PCCS maintains local caches of these artifacts as well.

Before registering, a platform must have the appropriate UEFI/BIOS settings and access to the Intel PCS. Both TDX and SGX must be enabled in the UEFI/BIOS on the host platform. An Intel account is required for retrieving API keys for registering a platform with the Intel PCS. If the PCCS is utilized, it must be configured with the API keys and Intel PCS server’s address. Architectural Enclaves. To enable quote generation on the platform. Intel provides two architectural enclaves: Provisioning Certificate Enclave (PCE) and QE. The PCE acts as a local certification authority for the QE. In its initialization process, the QE generates an attestation key pair. It sends the public part to the PCE. The PCE authenticates that this is a legitimate QE on the platform and then signs the attestation public key certificate with the PCK. This signature creates a quote certificate chain from an Intel-issued PCK certificate to the QE attestation public key. Figure 9 illustrates the quote’s certificate chain. The PCK certificate is used for verifying the QE attestation public key certificate, and the QE attestation public key in turn for verifying the signature on the quote. Remote Attestation Flow. Figure 10 shows a remote third party performing attestation with an attestation agent running on a TD. The remote party sends an attestation request providing a nonce to the attestation agent (Step 1). The nonce provides freshness to the request and prevents replay attacks. The attestation agent retrieves a TD report from the TDX Module providing the nonce as the REPORTDATA (Step 2) and then subsequently requests the QE to sign the TD report using its attestation key (Step 3). The QE verifies that the TD report is generated on the platform before signing with its attestation key. The attestation agent then returns the quote to the remote party (Step 4).

Fig. 9.

View Figure

Fig. 10.

View Figure

The remote party requires the platform’s PCK certificate to verify the quote, so it may download the PCK certificate from a PCCS (Step 5) or retrieve directly from the Intel PCS (Step 6). The party then proceeds to validate the quote (Step 7). It checks for the nonce in the quote and verifies the integrity of the signature chain from the Intel-issued PCK certificate to the signed quote, walking the certificate chain to determine whether the quote has a valid signature. The party also checks that no keys in the chain have been revoked and whether the TCB is up-to-date. Finally, the party checks if the measurements, i.e., values in MRTD and RTMRs, in the quote match a set of reference values. If it successfully validates the quote, the remote party can trust that the TD has been properly instantiated on a TDX platform.

9.3 Use Cases

Secure Channel Establishment. Remote attestation can be integrated with establishing a secure channel [46], linking channel setup with the endpoint’s TEE identity, state, and configuration. This integration prevents relay attacks since an attacker cannot forward a challenger’s attestation request from a compromised system to a trusted system to service the request.

In a typical scenario when a client negotiates a secure channel with a server running in a TEE, it wants to ensure a connection with a properly instantiated server. The server, serving as an attester, generates an ephemeral public and private key pair. It computes the hash of the public key and then creates a TD report providing this hash as the REPORTDATA. The server requests a quote of the report and generates a self-signed certificate with the quote embedded in the certificate. It provides this self-signed certificate as the server certificate in the TLS handshake protocol. When the client, serving as the challenger, receives the server certificate, it verifies signatures on the certificate and validates the embedded quote in the certificate, including the measurements. It also checks if the quote includes the hash of the public key since this links the key to the TEE. When establishing a secure channel, both client and server can assume the roles of attester and verifier. This allows endpoints running in TEEs to mutually authenticate each other by validating TEEs. Encrypted Boot. Using remote attestation, we can launch a TD with an encrypted partition image and let the tenant control the release of the partition decryption key. Figure 11 illustrates the execution flow of an encrypted boot of a TD. An attestation agent is placed within the TD’s initrd and starts when a TD is launched. The agent retrieves the TD’s quote, which contains the TCB measurements (from the TDX platform up to the initrd), and is signed by Intel’s QE. The quote is sent over a secure channel to an attestation server controlled by the tenant for verification. Once a quote is verified with the expected measurements, the attestation server provides the decryption key for the partition to the attestation agent. The agent then uses this key to decrypt and mount the encrypted partition that contains the secrets. This ensures that secrets can only be loaded into encrypted memory and are not visible to the host.

Fig. 11.

View Figure

Tenants also have the flexibility to integrate the attestation agent in the virtual BIOS and wrap the entire VM workload within an encrypted image. In this case, the virtual BIOS needs to be extended to support the functionalities of retrieving the quote and fetching the key for mounting the encrypted disk image.