{"id":438,"date":"2023-07-16T10:32:00","date_gmt":"2023-07-16T18:32:00","guid":{"rendered":"https:\/\/nramkumar.org\/tech\/?p=438"},"modified":"2023-07-16T10:32:00","modified_gmt":"2023-07-16T18:32:00","slug":"vfio-nvidia-gpu-passthrough-with-host-access","status":"publish","type":"post","link":"https:\/\/nramkumar.org\/tech\/blog\/2023\/07\/16\/vfio-nvidia-gpu-passthrough-with-host-access\/","title":{"rendered":"VFIO NVidia GPU Passthrough with Host Access"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">This is a write up of what I needed to do to enable passthru of NVidia discrete GPU with host access to the dGPU.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">System:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Intel 12600K CPU with iGPU (supported by i915 driver)<\/li>\n\n\n\n<li>NVidia 3060 dGPU (supported by proprietary nvidia driver)<\/li>\n\n\n\n<li>Kubuntu 23.04<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">What I want to be able to do:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Use my Intel iGPU for X and display on the Linux host system.<\/li>\n\n\n\n<li>Use my NVidia dGPU for ML + deep learning and occasional video encoding on Linux host system when VM is not running.<\/li>\n\n\n\n<li>Run a Windows VM that uses NVidia dGPU for display and run games on it.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Apart from <a href=\"https:\/\/nramkumar.org\/tech\/blog\/2022\/12\/01\/gpu-passthrough-of-nvidia-3060-to-virtual-machine-with-intel-igp-for-host\/\">following the guides from my previous post,<\/a> the NVidia proprietary drivers + X11 + Kernel Mode Setting combine to make dynamic passthru a nightmare. In particular, X11 somehow latched on to the NVidia GPU no matter what I tried (with the exception of blacklisting the nvidia drivers which defeats the purpose of what I list above as what I want to do). After a lot of research, these are the set of changes I had to make to get the system work as I wanted:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Change <code>\/lib\/udev\/rules.d\/71-nvidia.rules<\/code> as follows:\n<ul class=\"wp-block-list\">\n<li>Comment out the <code>Action==\"add\"...Run+=\"\/sbin\/modprobe nvidia-modeset\"<\/code> lines. This prevents the nvidia proprietary drivers from being constantly attempted to be loaded.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Move all <code>*nvidia*<\/code> files out of <code>\/usr\/share\/X11\/xorg.conf.d folder<\/code><\/li>\n\n\n\n<li>Change my <code>\/etc\/X11\/xorg.conf<\/code> to the following<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\"><code>Section \"ServerFlags\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"AutoAddGPU\" \"off\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"AutoBindGPU\" \"off\"<br>EndSection<br><br>Section \"ServerLayout\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Identifier \"layout\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Screen 0 \"igpu\"<br>EndSection<br><br>Section \"Device\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Identifier \"igpu\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0BusID \"PCI:0:2:0\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Driver \"intel\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"PrimaryGPU\" \"yes\"<br>EndSection<br><br>Section \"Screen\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Identifier \"igpu\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Device \"igpu\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0GPUDevice \"igpu\"<br>EndSection<br><br>Section \"Device\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Identifier \"nvidia\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Driver \"nvidia\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"Accel\" \"off\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"ProbeAllGpus\" \"off\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"AllowEmptyInitialConfiguration\" \"off\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"AllowNVIDIAGPUScreens\" \"off\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"UseDisplayDevice\" \"none\"<br>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0Option \"PrimaryGPU\" \"no\"<br>EndSection<br><\/code><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Add the following to <code>\/etc\/default\/grub<\/code>\n<ul class=\"wp-block-list\">\n<li><code>GRUB_CMDLINE_LINUX_DEFAULT=\"quiet splash intel_iommu=on iommu=pt pcie_aspm=off nogpumanager nomodeset nvidia_drm.modeset=0 noveau.modeset=0 i915.modeset=1\"<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Change my <code>\/etc\/modprobe.d\/vfio.conf<\/code> to the following (get your intel device ids with <code>lspci -nnk<\/code> and use the right ones for your system):\n<ul class=\"wp-block-list\">\n<li><code>options vfio_pci ids=10de:2487,10de:228b<\/code><\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Run <code>update-grub<\/code> and <code>update-initramfs -k `uname -r` -u<\/code><br><br><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>This is a write up of what I needed to do to enable passthru of NVidia discrete GPU with host access to the dGPU. System: What I want to be able to do: Apart from following the guides from my previous post, the NVidia proprietary drivers + X11 + Kernel Mode Setting combine to make&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-438","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/438","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/comments?post=438"}],"version-history":[{"count":1,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/438\/revisions"}],"predecessor-version":[{"id":439,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/posts\/438\/revisions\/439"}],"wp:attachment":[{"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/media?parent=438"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/categories?post=438"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/nramkumar.org\/tech\/wp-json\/wp\/v2\/tags?post=438"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}