VFIO NVidia GPU Passthrough with Host Access

This is a write up of what I needed to do to enable passthru of NVidia discrete GPU with host access to the dGPU.

System:

• Intel 12600K CPU with iGPU (supported by i915 driver)
• NVidia 3060 dGPU (supported by proprietary nvidia driver)
• Kubuntu 23.04

What I want to be able to do:

• Use my Intel iGPU for X and display on the Linux host system.
• Use my NVidia dGPU for ML + deep learning and occasional video encoding on Linux host system when VM is not running.
• Run a Windows VM that uses NVidia dGPU for display and run games on it.

Apart from following the guides from my previous post, the NVidia proprietary drivers + X11 + Kernel Mode Setting combine to make dynamic passthru a nightmare. In particular, X11 somehow latched on to the NVidia GPU no matter what I tried (with the exception of blacklisting the nvidia drivers which defeats the purpose of what I list above as what I want to do). After a lot of research, these are the set of changes I had to make to get the system work as I wanted:

• Change /lib/udev/rules.d/71-nvidia.rules as follows:
  • Comment out the Action=="add"...Run+="/sbin/modprobe nvidia-modeset" lines. This prevents the nvidia proprietary drivers from being constantly attempted to be loaded.
• Move all *nvidia* files out of /usr/share/X11/xorg.conf.d folder
• Change my /etc/X11/xorg.conf to the following

Section "ServerFlags"
       Option "AutoAddGPU" "off"
       Option "AutoBindGPU" "off"
EndSection

Section "ServerLayout"
       Identifier "layout"
       Screen 0 "igpu"
EndSection

Section "Device"
       Identifier "igpu"
       BusID "PCI:0:2:0"
       Driver "intel"
       Option "PrimaryGPU" "yes"
EndSection

Section "Screen"
       Identifier "igpu"
       Device "igpu"
       GPUDevice "igpu"
EndSection

Section "Device"
       Identifier "nvidia"
       Driver "nvidia"
       Option "Accel" "off"
       Option "ProbeAllGpus" "off"
       Option "AllowEmptyInitialConfiguration" "off"
       Option "AllowNVIDIAGPUScreens" "off"
       Option "UseDisplayDevice" "none"
       Option "PrimaryGPU" "no"
EndSection


• Add the following to /etc/default/grub
  • GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on iommu=pt pcie_aspm=off nogpumanager nomodeset nvidia_drm.modeset=0 noveau.modeset=0 i915.modeset=1"
• Change my /etc/modprobe.d/vfio.conf to the following (get your intel device ids with lspci -nnk and use the right ones for your system):
  • options vfio_pci ids=10de:2487,10de:228b
• Run update-grub and update-initramfs -k `uname -r` -u