The link to the video by LinusTechTips for anyone interested[0]. He actually had a heck of a time getting it working fully with all 7 graphics cards, as shutting a VM down wouldn't correctly release the card's resources (apparently an AMD-specific issue, but they denied it[1]), causing him to have to do a full reboot of the whole system every time one VM needed a restart[1].

[0] - https://www.youtube.com/watch?v=LXOaCkbt4lI (14:57)

[1] - https://www.youtube.com/watch?v=lThX7UovKCc (1:35:15 - really long stream of him disassembling it live, so forgive me for not finding exactly where he said each thing).

Here's the link to where he talks about AMD on that stream https://youtu.be/lThX7UovKCc?t=34m

Sounds suspiciously like this hardware quirk[0] that was only supposed to affect Hawaii/Bonaire... Guess it's a little more widespread. Easily fixed, I guess[1].

Then again Nvidia cards have their own set of horrible problems too, although theirs are somewhat on the side of malicious intent rather than incompetence.

[0] - https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks....

[1] - https://github.com/qemu/qemu/blob/master/hw/vfio/pci-quirks....

Holly crap, I always dreamed to achieve what you did and never knew direct GPU access was possible, even though I used Qemu for work before.

Right now I am basically running a windows pc for gaming and a linux laptop for work. It seems IOMMU is compatible with all my gaming tower, and with many modern cpus/chipsets. I will make a test over a live CD anb if it works migrate my whole pc and blog about it.

Thanks very much for the ida

What's your blog url?

Here: https://medium.com/@ChakibBenz

I started a diary article here.

Link: https://medium.com/@ChakibBenz/making-a-home-made-mini-cloud...

Just general disclosure that Steam on Linux does have 1600 games now, and at least a few dozen AAA titles. I have no problem gaming on Linux like any console owner would - if it releases for my platform, awesome, new games. If it does not, it might as well not exist to me. I have no shortage of games to play (I still have to beat Witcher 2 and I'm 20 hours in... 3 years later). And Blizzard stuff / anything older than 2010 runs perfectly in Wine nowadays, and with Gallium Nine I can play Skyrim on my 290 at its max DX9 settings at ~100 fps.

1600 "games". most shouldn't be called such because of their quality.

vanilla skyrim never required a good computer... its a different story if it's fully modded, but i doubt you've done that, as you'd be pushing 30 fps with a 290 in that case

How much of a performance hit do you get running a VM with vtx and vtd versus running windows bare metal?

Is it as simple as it seems? I'd love to switch over to a VM for windows games backed by vtx and vtd, but I've been worried that it'd be waay more work.

I tried this a while back. My results can be seen at http://miffe.org/bench/

The largest problem is that the NVIDIA driver refuses to work when it detects that it's in a VM. The workaround is to hide the VM status for the guest os, by using -cpu host,kvm=off and disabling the tweaks that makes windows work better (hv_time,hv_relaxed,hv_vapic).

Wow, that's a great resource! I'm glad to see your disk benchmarks were also all over the place. I couldn't seem to make much sense of CrystalMark's results, even when increasing the test size above my RAM size to eliminate caching. I'm using virtio to an SSD dedicated to the VM and saw, like you, some markers were better in the VM than they should be, and some where significantly worse. Very weird. Luckily I won't matter for gaming.

If you made a nice blog post about this, in detail, and shared it, a lot of folks would find it extremely interesting!

A better workaround would be to buy AMD GPUs.

I always wonder why they don't really advertise this to gamers as a feature at which they are better than nvidia.

Thats my plan. I got the GTX 980 for free, so I'm using until the next gen AMD cards are released.

> How's much of a performance hit do you get running a VM with vtx and vtd versus running windows bare metal?

I've seen research that shows the GPU being 98-99% when using QEMU/KVM. I haven't used the system long enough to accurately quantify my personal experience.

> Is it as simple as it seems?

For me getting the GPU working was the easiest part. It worked on the first try. The hardest part was finding good docs (Arch wiki & VFIO blog), trying to understand their little ecosystem, and finding the right compatible parts (hint: besides the CPU, it's really easy). I'd never used QEMU/KVM before (I used VirtualBox previously) so that was a big learning curve. It seems like things have matured quite a bit since the VFIO blog wrote its articles, as a number of things had changed and become easier.

Does "getting the GPU to work" mean you can give your Windows VM exclusive rights to the ports on the back of that graphics card as well? That would be amazing.

I'd love to, say, run a Hackintosh setup with Windows 10 in a VM that could fully utilize it's own GPU + external ports -- this way I could use a modern VR setup (Oculus or Vive) without having to use Windows full time!

In fact, that's the only way to do it. This VFIO/IOMMU setup virtualizes the PCI-E port itself. It routes the PCI-E port that the graphics card is plugged into to the VM. So the VM has exclusive and direct access to that PCI-E port. Therefore, the GPU, the ports on it, everything hooked up to that PCI-E port belongs to the VM. Most importantly, this also means the GPU's SMBus for temperature monitoring is also routed and owned by the VM. From the VM's perspective it isn't any different than a physical card.

You can also route USB hubs to it the same way, sound cards, etc.

A coworker of mine has been wanting to do this for ages, and I've always wondered - what's the workflow like for getting the mouse and keyboard from the Linux screen to the Windows screen?

For the past few years I've been running two desktops side by side (one with Windows and one with Linux), then use Synergy to get from one to the other painlessly.

When I next upgrade, IOMMU might let me consolidate the number of PCs available, but only if it's relatively painless to get from one OS to the other.

So, there are several ways to handle mouse+keyboard that I've seen.

1) Virtualized Mouse+Keyboard. If you've ever used a VM before, this is what you most likely are familiar with. You click within the VM's window, it captures your mouse and keyboard, and then everything is like normal within the VM. To release, you hit some special key combo and your mouse+keyboard is released back to the host. The VGA Passthrough setup that we're talking about doesn't normally support this method in QEMU/KVM, but you can hack it in. For me, it works as expected, except the mouse+keyboard randomly stop working for brief periods of time every so often. So... not great.

2) Synergy. I've seen this method mentioned a lot, so I guess it works. I haven't tried it. I've heard there are occasional issues, and sometimes Synergy will stop working so you need some kind of fallback method to get back into the Windows VM to get it working again. Probably this method plus method 1 above as a backup will work well.

3) USB passthrough. You could get a second mouse+keyboard, and then use USB filtering to pass them through to the VM. I haven't tried it; it probably works fine if you're okay with having two mouse+keyboard sets on your desk.

4) IOMMU USB passthrough. Similar to #3, but you use IOMMU to give an entire USB hub to the VM. This is likely to have the least number of issues, as it gives direct and exclusive control of a USB hub to the VM. Whether you can do it or not depends on the USB hub configuration on your mobo or whether you feel like adding a USB PCI/PCI-E card.

5) KVM + USB passthrough. Use a KVM and then USB passthrough of some kind (#3 or #4 above).

6) Steam In-Home Streaming. Steam In-Home Streaming can be hacked to share the entire desktop of the target machine. So you could use this to not only handle keyboard+mouse, but also video if you don't want a second monitor and/or don't want to use your existing monitor to switch HDMI/DisplayPort. Since the streaming will be done locally, latency will be very low, but this will burn some CPU/GPU resources. VNC is also an option, but will likely fallover for video games/watching videos/etc.

Hah, I can't escape Synergy. Which is ok, I use it 24/7 because it's an amazing, peerless piece of software.

But yeah, modern synergy crashes pretty often. There was a lag in development a few years back when it looked like synergy was dead, but the most recent version at the time was sooo stable. Now synergy has more features but tends to crap out or start dropping your clipboard every few hours or days.

If you're connecting multiple Linux machines (I suppose it would work for Mac too) then you can hack in this fix that I came up with:

Have a once-a-minute cronjob on the server tail the synergy log. Grep the tail for "server dead" messages and restart the server. Grep the tail for "client dead" messages and ssh into the client and restart synergy.

You have to use commandline synergy for this to work of course, but you probably want that anyway - otherwise you have to log in to each machine with a physical mouse and keyboard and manually start synergy at each boot. synergyc and synergys are your friends.

I can't believe I've never heard of this before. I'm honestly astonished! Thanks for your reply, I can't wait to try this.

Keep in mind there's more to it than pure frames per second. You also got latency and jitter to worry about which is vital but almost always ignored, probably because it's difficult to measure and understand. Doing it properly still required special hardware and software I believe.

Indeed, I've seen some talk about latency/jitter issues on the vfio mailing list. I haven't tested it myself yet, so we'll see how it goes. Though I figure things will just naturally get better over time as the kernel's support improves.

I'm happy to hear that gaming works well, and apparently my decidedly older CPU (AMD Phenom X4 965) supports the technology as well, which is interesting to learn.

However, my main reason for needing Windows is for music recording and mixing via Cubase and other DAW software. This requires really low-latency access to my USB and PCI sound cards.

Does anyone have experience with running applications like this in a VM? Is it feasible?

I would be thrilled if I could finally move to Linux or FreeBSD entirely and just keep a Windows 7 VM, all sandboxed up with no Internet access, just for the music recording.

The simplest thing to do of course is just try it. You can install Windows in a VM without a product key and try it without limitation for something like 30 days.

That said, "really low latency" is relative. Device assignment will give you excellent I/O performance in terms of sending data to/from the device, particularly for bulk DMA transfers. Where you (likely) still pay a penalty is in terms of interrupt delivery, as on most chips these are still routed through the host even in "passthrough" mode. Even so, you're looking at delays at the outside on the tens of microseconds scale, depending on which microarchitecture you're on and how well tuned the fiddly bits of your hypervisor (presumably KVM for most users) are. Some configurations will be much better behaved here. If your audio hardware goes into polling mode (or can be made to do so) when it is recording you can potentially avoid the interrupts entirely.

(source: I work on Google Compute Engine, in particular on virtualized networking, historically on performance. Virtual interrupt delivery latency is a topic near and dear to my heart, although I don't know a damned thing about audio gear and its tolerances there :)

I personally install KXStudio[0] for guys just like you almost monthly. here's some appropriate[1] pornography[2]. Imagine being able to have a skype call with someone, and be able to record their vocals (provided they have a good mic) apply effects, and then record _and_ pipe the vocals back to the vocalist in realtime. Or make recording snippets of audio off youtube trivial. Or stream live instruments in other rooms over wifi/ethernet using only a raspberry pi and a good microphone. Or stream 8 of those tracks/instruments (on different devices) to one computer for mastering/final mixing. It's audio nerd nirvana, basically.

What you need: Wine Wineasio (for Wine support in FL Studio or others) KXstudio

1. http://kxstudio.linuxaudio.org/index.php 2. https://www.youtube.com/watch?v=GBD-wsMiZCk 3. https://www.youtube.com/watch?v=PhfmGGPcwNc

For lack of a response from someone who has actually tried getting ASIO working on such a setup, I suggest you just try it. Grab a Live CD and a spare external hard drive and just give it a whack. Passing through the sound cards should work. USB might be trickier if you're not just passing through a whole USB hub. But considering how new IOMMU is, I imagine the intersection between people who have tried it and people who use Cubase is incredibly tiny.

I have found that a lot of the audio editing software you can get away with quite low specs. I record my podcasts on a 8 year old laptop running Windows XP. It rarely connects to the internet and I record to a thumb drive that is backed up so there is nothing to lose on it. In this particular case something similar might be the best option. For pod-casting anything over 2 or 3 ms in my headphones throws me off my game.

Unlike podcasts, music recording and mixing is extremely CPU intensive. Especially if you run a lot of analog simulating synths and stuff.

Virtual CPUs with VTx can be made to run at essentially native speed for most workloads on most fairly modern CPUs. Memory performance can be subpar, although there's public documentation on techniques that can improve that: https://help.ubuntu.com/community/KVM%20-%20Using%20Hugepage...

I/O performance is a whole other story, dependent on many workload and hypervisor specific factors.

I stand corrected.

Really, the 965 supports it? I have the 970 and was under the impression the Phenom II's in general just don't support it. Can you show me where you found that information?

It would be amazing if you could write a separate post with further details. I think many people here would love to know more.

There are many guides about this, just search for something like "VGA PCI Passthrough"

https://www.google.com/search?safe=off&q=VGA+PCI+Passthrough...

For anyone that tries this route, following random tutorials on VGA PCI Passthrough is a route to sure frustration and time wasting (I tried it ~2 or 3 years ago, whenever I built my previous PC). You have to make sure your video card, motherboard chipset, audio, and anything else you want to pass through are all supported. It's much simpler if you can just follow a list that says "this combination of hardware + software works perfectly."

All I did was check the Wikipedia page [1] for IOMMU-supporting hardware, and I needed to upgrade my CPU from a Phenom to FX. You don't need special audio, just the right motherboard (chipset specifically) and CPU. If you do PCI passthrough, it doesn't matter if the Linux host can talk to the card, the guest OS will take care of everything.

Otherwise, check the forums. This has been done many times over the years, and there's no shortage of existing information.

[1] https://en.wikipedia.org/wiki/List_of_IOMMU-supporting_hardw...

Seriously did people even click the link? This is not some insidious full screen advertisement . . . it's windows spotlight.

You were allowing Bing to show you random photos by enabling this feature. If you didn't want that, then disable it . . . give a break. Ads on installers have been common for ages. Ubuntu shows advertisements . . . this is out of control with the fear mongering.

But the enabling by default of advertisements in the OS is certainly something I'm not comfortable with, no matter how pretty the ad is or how simple it is to disable.

Is there a write-up on this that you can recommend?

The Arch docs are super helpful on this, even if you're not running Arch: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVM...

Do you have a link to a recommended tutorial?

The VFIO blog was the best resource I found. Start at the first article (http://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part...) in the how to series, work through that, and then read all the other posts on there. It's also helpful to use https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVM... , as that covers some of the things that the VFIO blog glosses over and it's a little more up-to-date. This is all bleeding edge stuff. For example, VFIO broke in a recent Linux kernel and only got fixed and released in Arch's repos a few days before I started my installation :P

Agreed, this the same info I followed to get Elite Dangerous working in a Windows VM on a Ubuntu host. Feels very nice to have Windows not as the root of a machine.

I've done this too! I was able to do it with my AMD FX-8350 and GA-990FX-UD3, as they both support IOMMU. Passing through my R9 390 to the Windows guest, and the only issue I've encountered the whole time was having to disable hardware acceleration for flash videos.

The whole KVM/Qemu ecosystem is great.

It's so nice for KVM and Xen to support this.

I'd wish virtualbox would support pci pass-through. Virtualbox being the default and most used vm with vagrant it would be awesome to spin up a linux vm with pci pass-through, so you can have a machine learning vm with cuda.

VirtualBox does support PCI passthrough. See https://www.virtualbox.org/manual/ch09.html#pcipassthrough

I have never seen someone who actually got it to work. I read somewhere where it might be possible with a Linux host and a Linux guest. If anyone was ever successful with a windows host and a Linux guest I'd love to read how...

This was the last straw for me. My current chipset doesn't support the VT-D stuff, so it's an excuse to shop around for an upgrade. Figure I get a board that supports the integrated graphics, run Linux on that, and then pass through my GTX980 to a VM.

Thank you very much for this post, this had been one of the things I've wanted to do for so many years but never knew if anyone else found a proper way of doing it, every time I would google this I would end up with bad results.

I just bit the bullet and kept my gaming machine on Windows 10 and built a whole new Debian machine for all my dev work and personal computing.

I would prefer to drop Windows entirely though and was looking forward to SteamOS making strides in this area but it doesn't look too promising.

WINE solves a lot of problems but it's not perfect, and a lot of games still don't work with it according to its compatibility database.

I would really appreciate a write-up, or link to a step-by-step guide on this as well.

I have a 5820K as well, with a GA-X99-UD3P motherboard and 32 gigs of quad-channel DDR4 RAM, and I dual-boot Linux and Windows. Windows just has Steam, and I do all my work on Linux. It would be wonderful to not have to reboot and switches OSes and just run Windows off a VM with full graphics performance! :)

Have your tried Synergy for your mouse/keyboard situation? Another possibility is perhaps a KVM switch that if you plug in two outputs into the same computer that it is recognized as two distinct USB devices. I don't know if it's possible but then you could pass through one to the VM directly.

Synergy is decidedly non-optimal for gaming, unfortunately. Various games try to grab the mouse pointer in different ways and synergy throws a wrench into at least some of them.

you can bind a key to switch to relative mode.

since switching to VM based gaming about 6 months ago I've played and completed: SC2 LOTV, Bioshock Infinite, Crysis 1/2, and The Talos Principle and have not had a single problem.

What is your CS:GO rank?

I haven't played CS since 2001

piggybacking on top comment. I have yet to work through this process, but there's this write-up for doing this in Ubuntu I found a few weeks ago:

http://ubuntuforums.org/showthread.php?t=2266916

Any chance you have a parts list for your build?

My build: http://pcpartpicker.com/p/JVsKYJ

I splurged on an M.2 PCI-E drive. 2 GB/s read!? How could I turn that down? No way I'll ever use that drive to its full potential, but I love the M.2 form factor.

Only thing I'd change is using AIO cooling for the CPU and the S340 instead of the H440 case. I had to use the H440 for the air cooler, and that was not an aesthetically pleasing choice. The H440 also has a number of obnoxious problems.

Not OP but, if you're interested in building your own PC and need some guidance, check out the Build A PC reddit. They're extremely helpful and knowledgeable in there. https://www.reddit.com/r/buildapc

Another similar resource is overclocking https://www.reddit.com/r/overclocking

Yeah this is not a good move; thankfully for now it can be turned off as explained in the text of the article.

I also want to know more about this, care to share it? Thanks!

Have you considered: you don't really need to play video games anyway

It's not something that requires explicit consideration: probably anybody you ask will tell you that no, they don't need to play videogames; people who play them normally do it because they enjoy it, not because they need to.

(It feels weird to have to explain that.)

Have you considered pausing before hitting "reply" and asking yourself: "does this comment add anything? does it have anything intelligent to say?"

You don't really need to do anything that doesn't provide for your economic well-being. Some things a just for fun and that's great!

You don't need to provide for your economic well-being either.

Right, who needs hobbys or fun anyway. /s

For those of us who live fairly far away from our most excellent friends, yes, I need video games to play with them. I look forward to my weekly game sessions to drink a beer and hang out with them. Even something as simple as a board game is well suited to a video game with voice chat, and those often are primarily and only built/tested/supported on Windows, making fpgaminer's notes very welcome.

Nor do you need to be on Hacker News...

You do need to relax and unwind though, and video games are a great way to do that.

No, you don't need to play video games, but they can be a lot of fun.

However, it's important to note that you don't need Windows 10, or Windows at all, to play good video games. All the best games play using emulators, since they all came out in the 70s-90s. And Linux supports emulators like MAME and MESS just fine.

I don't need to play Go, learn guitar, read books, watch movies or travel either, but I still do those things because they're fun.