Gaming on a virtualized Windows with VFIOSunday, June 12, 2016
I use Linux both at home, and at work, the only reason I have a Windows partition is gaming. I have used a dual boot linux/windows setup in the past 5ish years, and I always hated how I had to choose at boot time if I want to play some games, or do something else. Now with the addition of the vfio driver into the kernel, it is now possible to achieve a much more comfortable setup.
I try to play games on linux whenever I can, either using a native client if it exists, or going through Wine. Unfortunately native clients are still rare (but the situation is much better now than 5 years ago, and it's getting better and better, which I'm really happy about), and Wine might not support the game I'm looking for, which leaves me with dual booting Windows as the only option.
There are several pain points with the dual booting setup:
- Requires a dedicated NTFS partition for Windows
- You have to choose at boot time whether you want to play games, or do something else
- Configuration needed in the boot loader
- Have to keep some application specific files in sync, eg. browser histories
The second point is the most annoying one, because if I decide to play some games, I have to close everything that is open, I can't just leave stuff running in the background, and then come back later.
Virtual function I/O
VFIO stands for Virtual Functio I/O. There is a detailed paper explaining what it does exactly, but the short summary is that it allows direct access to hardware, from inside a VM, and the reason this is so fucking awesome is because it lets you pass your GPU to a virtualized Windows running inside a VM, and achieve near bare metal performance. I'm actually kinda tempted to put the near in quotes, because in my case I get slightly higher results on performance tests.
Passing through an entire GPU has some requirements unfortunately:
- You need 2 GPUs, one for running Linux on, and one for passing inside the VM
- Your CPU and motherboard must support hardware virtualization (VT-x/AMD-V)
- They also must support something called IOMMU
- The guest GPU must support UEFI
The first point is self-explanatory. A GPU can't be rebound from the host to the guest on the fly, it has to be a separate card that is not in use at all. Yes, you can use the integrated GPU in your Intel CPU, and pass in the PCI-E GPU. For the second point, this shouldn't be a problem, most CPUs made in the past few years support hardware virtualization.
The third point is tricky. There are some lists about which hardware supports it, but they are quite incomplete as of today, my own motherboard was not on the list and it works just fine. As always, do your own research before you buy.
The fourth point was a surprise to me, because I had no idea that it is even possible for a GPU to have support for UEFI, I thought this is something that only the motherboard is concerned about. A list of GPUs can be found here.
So the idea is that you run Linux on your less powerful GPU, probably the integrated one, launch a virtualized VM, pass in your AMD/NVIDIA GPU, and game on that. Removes all the aforementioned problems, and the performance is the same as running the game on a native Windows install. I didn't want to believe it, but even extremely demanding titles like The Witcher III run perfectly.
The time spent on configuring/setup/tinkering is definitely a con. I would advise following the guide on the Arch Wiki, it's fairly comprehensive, and easy to follow. You will also need to set up networking to the VM, if you are familiar with routing/iptables/networking in general, it shouldn't be hard, I had no previous experience and managed to do it after a few hours of reading.
While you are still configuring, use the
-vga qxl flag. It will run the VM in a normal window, without using the passed in GPU, this will allow you to easily alt tab out of the VM to research stuff.
The thing that gave me the most amount of headache was the sound. Sound from inside the VM was crackling, and I couldn't figure out why. Turns out was the
-vga qxl flag, because it added just enough performance overhead that sound became garbled, but after running the VM with
-vga none (so it uses the passed in GPU), sound was fine.
The only thing that remains for me to solve, even today, is recording sound (think Teamspeak, Mumble). I have tried passing in my USB headphones, but when I try to record sound inside Windows, I get an error about "Not enough resources on this USB controller". I have also tried just using Pulseaudio to specify the headset as the sound source for qemu, but that resulted in a major lag (20 seconds) in sound input. Solved, see edits at bottom.
Another thing which isn't really a con, but still worth mentioning, is that if you want to use the VM and the guest at the same time, you will need an extra set of keyboard/mouse/screen, because whatever you pass in to the VM, stays with the VM until you shut it down. If you are fine with not using both machines in parallel, you can just connect a single monitor both to your integrated and the real GPU, with two cables, and switch input on it when you launch the VM. Or you can give some input sharing programs like Synergy a go, or try a KVM switch.
The pros in my opinion far outweigh the cons. All the problems from the list mentioned above disappear. I'm personally passing in a whole SSD into the VM as a raw device, but you can also use qcow2 images, that way you can even get snapshot support. Both of them allow you to just mount the Windows partitions into the host without a problem, and if you use a raw device you can even boot the Windows VM natively if you want to.
Here are two pictures, running the Unigine Heaven benchmark, left one is on the VM, right one is native. Bonus picture of my host and the guest running at the same time while writing this article.
Overall, I'm extremely happy with this setup, and have completely ditched the windows partition with the dual boot setup.
In case you also want to try it, read the write up on the arch wiki on how to set it up, it's not hard, but definitely takes a few hours if this is the first time you are doing it. I ended up not using libvirt, because qemu itself was a new thing to me that I had to get familiar with, and I didn't want to add another layer of unfamiliarity, and it works just as well.
According to the qemu mailing list, the recording issue is a bug in qemu, and there is a patch for that is supposed to solve the recording delay, but it hasn't been merged, and it was left abandoned since then.
There is a fix I found on r000t's blog, that solved it for me. Turns out that if you change the sample rate in windows for the recording device to
16 bit, 48000 Hz (DVD quality) it makes all the problems disappear. So, to recap, pass in your USB headset/mic with
-usbdevice host:xxxx:yyyy, then set it as the default device for playback/recording in Windows, apply the above fix, and you should have both crystal clear sound, and working recording.