I mentioned in an earlier post that one of my side goals for this project was to set up a (Windows) VM which would allow me to utilize Nvidia Gamestream and Moonlight (or an Nvidia Shield) to play games which run on Window exclusively. I did this previously using a Windows 10 VM on Proxmox with GPU passthrough, but I couldn't get the video drivers to intialize the card. It got an error 43 in device manager.
It turns out that the problem was that the Proxmox host initialized the GPU first so it was not available for the Nvidia drivers in the VM to initilize. I needed to have two GPUs - one of the Proxmox host and one to passthrough to the VPN. I initially swapped out the Geforce 1650 super for my son's old Geforce 950. It didn't help, but I let him keep the better card anyway. The 950 should be sufficient for my purposes.
I purchased a cheap MSI GT710 and installed it in the other PCIe slot and configured the BIOS to use that card as the primary video card. I follow this guide with a few extra steps, but I will highlight what I did.
Prepare Proxmox Host
First, I had to make sure the Proxmox host is set up correctly to allow GPU Passthrough:
- Edit /etc/default/grub to add an extra option to the Grub commandline:
GRUB_CMDLINE_LINUX_DEFAULT="quiet iommu=pt amd_iommu=on video=efifb:off"
- Run update-grub after editing grub configuration
- Edit /etc/modules and make sure it contains these modules:
vfio vfio_iommu_type1 vfio_pci vfio_virqfd
- Configure the module configuration options for IOMMU:
options vfio_iommu_type1 allow_unsafe_interrupts=1
options kvm ignore_msrs=1
- Blacklist drivers so they don't grab the GPU:
blacklist radeon blacklist nouveau blacklist nvidia
- Add the GPU to VFIO by getting the GPU card vendor ID's
# lspci -v | grep NVIDIA 04:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 710B] (rev a1) (prog-if 00 [VGA controller]) 04:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1) 09:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 950] (rev a1) (prog-if 00 [VGA controller]) 09:00.1 Audio device: NVIDIA Corporation GM206 High Definition Audio Controller (rev a1)
For the card which is going to be used for passthrough, 09:00.00, I used this command to get the vendor IDs for both the video and audio devices:
# lspci -n -s 09:00 09:00.0 0300: 10de:1402 (rev a1) 09:00.1 0403: 10de:0fba (rev a1)
Now I use the vendor IDs to create /etc/modprobe.d/vfio.conf:
options vfio-pci ids=10de:1402,10de:0fba disable_vga=1
- I ran ran this command and rebooted:
Create the VM
- Download a Windows 10 installation ISO into Proxmox storage
- Download the Windows VirtIO drivers ISO into Proxmox storage
At this point, I connected a monitor to the video card that's to be passed into VM along with a USB keyboard and USB mouse. The first time around, I configured the VM to initally use NoVNC for the console waited to remove the display adaptor until I was ready to install the Nvidia drivers, but this worked just as well.
I created the VM with this hardware confguration:
- Memory - 8GB
- CPUs - 4 sockets, 2 cores
- Args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off'
- Bios - OVMF (UEFI) (must add a UEFI disk)
- Display - None
- Machine - q35
- SCSI Controller - Virtio SCSI single
- CD/DVD (ide2) - Mount Windows 10 Installation ISO
- Hard disk (scsi0) - 132G (can be extended as needed)
- Network (net0) - e1000, bridge=vmbr0, firewall=1
- EFI disk - size 4MB
- USB device (usb0) - USB Keyboard
- USB device (usb1) - USB Mouse
- PCI Device - 09:00, pcie=1, x-vga=1 (select All functions, ROM-Bar, Primary GPU, and PCI-Express)
- Audio Device - AC97, driver=spice
Under options, these are the important ones:
- Set boot order to CD-ROM or be prepared to hit the key to boot from the CD-ROM on power-on.
- OS Type - Micrsoft Windows
- QEMU Guest Agent - Enabled
Here is the final configuration found in /etc/pve/qemu-server/<vmid>.conf:
agent: 1 args: -cpu 'host,+kvm_pv_unhalt,+kvm_pv_eoi,hv_vendor_id=NV43FIX,kvm=off' audio0: device=AC97,driver=spice bios: ovmf boot: dcn bootdisk: scsi0 cores: 2 cpu: host,hidden=1,flags=+pcid efidisk0: local-lvm:vm-107-disk-1,size=4M hostpci0: 09:00,pcie=1,x-vga=1 ide2: qnap-iso:iso/win10dvd.iso,media=cdrom,size=4071872K machine: q35 memory: 8096 name: wearevirtual net0: e1000=FA:1D:DA:34:0E:EB,bridge=vmbr0,firewall=1 numa: 0 ostype: win10 scsi0: local-lvm:vm-107-disk-0,size=132G scsihw: virtio-scsi-single smbios1: uuid=2988df98-74c1-476a-842c-df54d1a4ba2d sockets: 4 spice_enhancements: videostreaming=all usb0: host=1bcf:0002 usb1: host=0461:4e26 vga: none vmgenid: c3c47c4d-ab4d-4722-a628-b099b6bfab30
Install and Configure Windows 10
- Boot from the Windows 10 Install Media
- I installed the Home edition, but if you want to be able to connect via Remote Desktop then select Pro. I installed TightVNC and saved $60.
- At the screen to select the install disk, nothing will show up. I had to switch the CD-ROM device media to the VirtIO ISO in order to select and format the disk. Then I had to mount the Windows 10 ISO again to proceed to the next step.
- Continuing with installing Windows 10, I skipped the product code and activated Windows later once I was sure it all worked.
After the initial install, I downloaded and installed Nvidia Experience to both update the video drivers and enable Gamestream. At this point, I had a perfectly good gaming machine setup running on a VM which I can access by directly using the keyboard, mouse, and video attached to Proxmox host. I had to check the device manager to ensure that the video card was being seen and wasn't showing an error 43. Refer to the guide for troubleshooting tips if this doesn't work for you.
For this next part, I figured out that I cannot be connected to the VM using Remote Desktop to enable Gamestream. Using the host hardware passed through or using TightVNC worked, but not RDP. To enable Gamestream, I followed the Moonlight Setup Guide. I was also able to install Steam and enable in-home streaming. They both worked well, but I've had slightly better performance with Gamestream.
Once Steam was installed, it showed up as a "Game" in the Moonlight client and I was able to see the Steam desktop client to install and launch games. Additional games can be manually added in Nvidia Experience if it doesn't find them in the Shield section of the settings.