DEV Community

Stefor07
Stefor07

Posted on

Nested Virtualization on Windows 11: The VBS Conflict Explained

Introduction

After upgrading to a modern system with full virtualization support — including VT-x, EPT, IOMMU, TPM 2.0, and Secure Boot — I expected everything to work flawlessly. Instead, I encountered crashes and instability when running nested virtualization scenarios, particularly when attempting to run VMware inside VMware Workstation Pro.

At first, I suspected hardware instability, BIOS misconfiguration, or a VMware bug. However, after extensive testing, I discovered the real culprit: Windows Virtualization-Based Security (VBS).

This article covers:

  • How VBS affects desktop hypervisors on modern Windows 11 (24H2/25H2)
  • Why nested virtualization fails under certain configurations
  • Practical solutions for advanced lab environments requiring full VT-x access

The Problem: VMware Nested Virtualization Failures

The issue surfaced when I upgraded to a 14th generation Intel CPU with hybrid cores (P-cores and E-cores). While VMware Workstation Pro ran perfectly on the host, problems arose when I tried running VMware inside a Windows 11 24H2 VM.

Attempting to boot a Windows 11 ISO inside the nested VM resulted in an immediate crash: VMware reported that an exception had caused a breakpoint, and the VM shut down.

Understanding the Root Cause: VBS

Virtualization-Based Security (VBS) is fully integrated into Windows 11's kernel and takes exclusive control of VT-x/AMD-V. This prevents desktop hypervisors from accessing hardware virtualization directly, causing conflicts with nested virtualization.

Initial Troubleshooting Attempts

I tried several manual approaches to disable VBS:

  1. Disabling Core Isolation (Windows Security → Device Security)
  2. Disabling hypervisor launch:
   bcdedit /set hypervisorlaunchtype off
Enter fullscreen mode Exit fullscreen mode
  1. Disabling HypervisorIOMMUPolicy:
   bcdedit /set HypervisorIOMMUPolicy off
Enter fullscreen mode Exit fullscreen mode
  1. Group Policy and Registry edits

None of these methods worked completely. The hypervisors continued to crash or remained unstable.

The Only Working Solution: Microsoft's DG_Readiness_Tool

After extensive testing, I discovered that the only reliable solution is using Microsoft's official script for disabling VBS:

Download: DG_Readiness_Tool

Usage:

DG_Readiness_Tool_v3.6.ps1 -Disable
Enter fullscreen mode Exit fullscreen mode

After running the script and rebooting, VBS is successfully disabled and nested virtualization works correctly.

Important: This change is temporary — Windows re-enables VBS after subsequent reboots, so the script must be rerun when needed.

Why Other Approaches Don't Work

HypervisorIOMMUPolicy Alone Is Not Enough

Initially, I believed that disabling HypervisorIOMMUPolicy would solve issues for lightweight hypervisors and Android emulators while keeping VBS active. This turned out to be incorrect.

Even with HypervisorIOMMUPolicy disabled, hypervisors running inside VMware VMs remain unstable or fail to launch because:

  1. VBS maintains VT-x/AMD-V control: As long as VBS is active, Windows Hypervisor Platform retains exclusive access to hardware virtualization extensions
  2. Nested hypervisors require direct hardware access: VMware, VirtualBox, and even lightweight emulators need unmediated access to VT-x/AMD-V
  3. Security layer interference: VBS creates a security boundary that conflicts with nested virtualization regardless of IOMMU settings

The Real Solution

Complete VBS disabling via the DG_Readiness_Tool is the only reliable approach for running any hypervisor inside VMware virtual machines, including:

  • VMware Workstation Pro
  • VirtualBox
  • Android emulators (AVD, Genymotion)
  • KVM (on Linux guests)
  • Other lightweight hypervisors

Complete Solutions

Since VBS cannot be permanently removed from Windows 11 24H2/25H2, here are your options:

Solution 1: Temporary VBS Disable (Recommended for Testing)

  • Run DG_Readiness_Tool_v3.6.ps1 -Disable before each session requiring nested virtualization
  • Provides full VT-x access temporarily
  • Limitation: Must be rerun after each reboot

Solution 2: Downgrade or Dual-Boot (Recommended for Permanent Use)

  • Use Windows 11 22H2 (doesn't enforce VBS)
  • Install as dual-boot alongside 24H2/25H2
  • Provides permanent, stable hardware virtualization without workarounds

Solution 3: Switch to Linux (Best for Advanced Labs)

  • No VBS restrictions whatsoever
  • Superior virtualization performance
  • Native KVM support with full hardware access
  • Popular options: Ubuntu, Fedora, Debian, Arch-based distributions

Compatibility Matrix

The table below shows hypervisor compatibility when running inside VMware VMs on Windows 11 24H2/25H2:

Hypervisor / Software VBS Enabled VBS Disabled (DG_Readiness_Tool) Notes
VMware Workstation Pro ❌ Crash ✅ Works Requires full VBS disable
VirtualBox ❌ Crash/Unstable ✅ Works Requires full VBS disable
Android Emulators (AVD) ❌ Crash/Unstable ✅ Works Requires full VBS disable
KVM (Linux VM) ❌ Cannot run ✅ Works Requires full VBS disable
Hyper-V (Windows VM) ❌ Not supported ❌ Limited Nested Hyper-V not functional on VMware
Other lightweight hypervisors ❌ Unstable ✅ Works Requires full VBS disable

Legend: ✅ Works reliably | ❌ Fails or unstable

Key Finding: All hypervisors running inside VMware VMs require VBS to be completely disabled. Partial solutions like disabling only HypervisorIOMMUPolicy or hypervisorlaunchtype do not provide stable nested virtualization.

Key Takeaways

  1. VBS must be fully disabled for any nested virtualization scenario on Windows 11 24H2/25H2:
   DG_Readiness_Tool_v3.6.ps1 -Disable
Enter fullscreen mode Exit fullscreen mode
  1. Partial solutions don't work: Disabling only HypervisorIOMMUPolicy or hypervisorlaunchtype is insufficient

  2. Hardware requirements:

    • Enable VT-x/AMD-V in BIOS/UEFI
    • Expose virtualization to guest VM in VMware settings
    • Use multi-core CPU allocation for better stability
  3. Long-term solutions:

    • For frequent nested virtualization use: Switch to Windows 11 22H2 or Linux
    • For occasional testing: Use the DG_Readiness_Tool and accept the need to rerun it after reboots
    • For production lab environments: Linux hosts provide the most stable and performant solution

Conclusion

Modern Windows 11's VBS creates fundamental incompatibilities with nested virtualization. While VBS enhances system security, it takes exclusive control of hardware virtualization extensions, making it impossible for hypervisors running inside VMs to function properly.

The critical discovery: Unlike what initial testing suggested, there is no partial workaround. Complete VBS disabling via Microsoft's DG_Readiness_Tool is mandatory for any nested hypervisor scenario, regardless of whether you're running VMware, VirtualBox, or lightweight Android emulators.

For users requiring frequent nested virtualization:

  • Windows 11 22H2 in dual-boot configuration offers a permanent VBS-free environment
  • Linux hosts eliminate VBS restrictions entirely and provide superior performance
  • Temporary VBS disabling works but requires re-running the script after every reboot

Understanding this limitation helps set realistic expectations and choose the right platform for advanced virtualization labs and development environments.

Top comments (0)