# {Guide} Create a Gaming Virtual Machine



## dizzy4

Latest News:Alpha 0.0.3: Windows 8 passthough seems to work just fine! A little trickier to install, but hey it's working! Expect better support with a beta version.


Spoiler: Change Log



0.0.3 -- Added preliminary Windows 8 support and instructed to turn off selinux
0.0.2 -- Removed some unnecessary steps, development libraries and tools no longer needed
0.0.1 -- First version of guide, first alpha





Spoiler: Benchmarks etc.



Here is the emulated IDE performance on the Force3 logical volume


and here is my Samsung 470 series with a patriot USB 3.0 enclosure tested on the etron usb3.0 controller that was passed through




My system Hardware is the rig called Test Chamber ITX.

i7-2600 (non-k)
ASrock H67M ITX/HT
Reference HD5850
Xen 4.1.2
Fedora 16
Windows 7 64bit


----------



## evermooingcow

Nice job getting it fully running







+rep
Too bad not many people here seem interested in things like this.

I've been thinking about doing this too but I'm still waiting on the Radeon 7950 or 7870. Really hoping the new cards work.


----------



## Sean Webster

Nice guide. Bookmarked for sure.









One thing though is that the bright green, red and yellow texts hurts my eyes and is slightly hard to read, so i'd suggest you set it black. Or just make them all red and make them bold as well. And maybe use spoilers in your guide to conceal most of it till the user expands it for use? As an example you can refer to my Windows 7 Install guide if it would help.


----------



## dizzy4

Quote:


> Originally Posted by *evermooingcow*
> 
> Nice job getting it fully running
> 
> 
> 
> 
> 
> 
> 
> +rep
> Too bad not many people here seem interested in things like this.
> I've been thinking about doing this too but I'm still waiting on the Radeon 7950 or 7870. Really hoping the new cards work.


They should work no problem. The components within the new cards have improved, but the way they work is pretty much the same so I would be 99% sure they would work just fine.
Quote:


> Originally Posted by *Sean Webster*
> 
> Nice guide. Bookmarked for sure.
> 
> 
> 
> 
> 
> 
> 
> 
> One thing though is that the bright green, red and yellow texts hurts my eyes and is slightly hard to read, so i'd suggest you set it black. Or just make them all red and make them bold as well. And maybe use spoilers in your guide to conceal most of it till the user expands it for use? As an example you can refer to my Windows 7 Install guide if it would help.


Thanks for the advice! I hope this cleans it up a little. Of course I will be cleaning it up slowly as I go and I am hoping to revise the guide and even make this about 4 steps less if I can.


----------



## subassy

Quote:


> Virtualization is the future of computing and is already being used in many capacities today. In writing this guide I hope to educate our community to it's benefits, usage and limitations. The more support it has, the further developed it will become. That is also my goal for this guide; to see it become simplified and offer broader support for hardware and software.


And yet no one seemed enthused about my proposed dedicated VM sub-forum...sorry, still bitter.

Awesome guide btw. If I hadn't been randomly browsing forums I would never have seen it. Too bad there's no way to "sym link" it to the servers forum. I bet a lot of people in there would love this


----------



## lloyd mcclendon

so ... how does it actually perform?

from what i've heard, the fps and such is all pretty decent. the problem is the input thread lags a few ms behind the display ... so your inputs aren't really in sync with what you see on the screen. any thoughts on that?

i didn't know xen had this feature ... will take a look. i wonder when we'll see the same thing on the kvm side. Spice is still quite unstable and so far has been a disappointment, but has some promise.


----------



## dizzy4

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> so ... how does it actually perform?
> from what i've heard, the fps and such is all pretty decent. the problem is the input thread lags a few ms behind the display ... so your inputs aren't really in sync with what you see on the screen. any thoughts on that?
> i didn't know xen had this feature ... will take a look. i wonder when we'll see the same thing on the kvm side. Spice is still quite unstable and so far has been a disappointment, but has some promise.


When the USB controller is passed through to the VM as well there is actually no more lag then there would be natively in windows. Maybe you are thinking of using a virtualized USB port, which I would not suggest for a VM. The performance hit is very slight because the video card is actually being fully controlled by the guest operating system so there is no need for spice at that point.

On another note, this guide is about to get a lot simpler. I am working on a custom Linux Live USB image to boot from that will let anyone try this without installing anything. I am also writing a shell script that will walk users through VM setup and pass the devices through properly while being easy to understand from a beginner's perspective. So look for a huge update in a few days


----------



## Plan9

I absolutely love the concept, but I'm a little confused as to why someone would want to do this as opposed to running Windows natively when you need physical access to the machine anyway.

Something like this would have been handy when I was trying to get Netflix working on Linux, but alas I just couldn't get the performance I wanted (albeit I never tried Xen and had very modest graphics cards so it was all software rendering). I might give Netflix / LoveFilm another try via this method if I get any spare time in the future


----------



## lloyd mcclendon

"why someone would want to do this as opposed to running Windows natively"

Well, obviously _if_ it works just as well there's no need to boot into windows. I never cared for a dual boot, huge waste of time. This way everything is kept in linux, and you have some throw away windows VM for gaming. If the performance is as good as the OP is claming, this is 1000 times better than wine, and 100000 times better than a dual boot. So if it works, i don't know why you wouldn't want to do this. Really this is HUGE news - how many linux users crawl back to windows entirely or dual boot just for gaming. And wine .. it hurts to say this, but it's practically impossible for that project to keep up with the crazyness that the MS developers create.

Apparently KVM does offer the VT-D PCI pass through ... but i'm a little fuzzy on whether or not it actually works right yet. I'll be trying to get one of my gentoo vms with xorg on it to use this... and if that actually works I should be able to get my XP VM showing good FPS as well.

I guess maybe I don't understand this - if I give a windows guest pass through access to my graphics card, and install the windows nvidia drivers package etc - what happens to the host (and the 12 other VMs) using the same graphics card?


----------



## dizzy4

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> "why someone would want to do this as opposed to running Windows natively"
> Well, obviously _if_ it works just as well there's no need to boot into windows. I never cared for a dual boot, huge waste of time. This way everything is kept in linux, and you have some throw away windows VM for gaming. If the performance is as good as the OP is claming, this is 1000 times better than wine, and 100000 times better than a dual boot. So if it works, i don't know why you wouldn't want to do this. Really this is HUGE news - how many linux users crawl back to windows entirely or dual boot just for gaming. And wine .. it hurts to say this, but it's practically impossible for that project to keep up with the crazyness that the MS developers create.
> Apparently KVM does offer the VT-D PCI pass through ... but i'm a little fuzzy on whether or not it actually works right yet. I'll be trying to get one of my gentoo vms with xorg on it to use this... and if that actually works I should be able to get my XP VM showing good FPS as well.


It does solve a lot of headaches, but it can also create some more in the process. This far from being perfected (the guide and the technology), but it really is the future of computing. KVM does offer VT-d passthrough, but is also a type 2 hypervisor which translates to poorer performance than Xen. There is a small performance loss because it still is a VM, but compared to what we usually think of gaming on a VM this is amazing. For instance, your GTX 570s in SLI would still have enough power to play pretty much anything and a 5% loss of performance might be worth it to not have to dualboot or use Wine. Some nVidia cards are reported to work with Xen and VT-d enabled, but others are more tricky (non-reference). The current unstable Xen release is supposed to have more support for nVidia cards.
Quote:


> Originally Posted by *lloyd mcclendon*
> 
> I guess maybe I don't understand this - if I give a windows guest pass through access to my graphics card, and install the windows nvidia drivers package etc - what happens to the host (and the 12 other VMs) using the same graphics card?


The display will go blank because the host no longer owns the resources of the card. This could be fixed by writing a script to reattach the card to the host once the gaming VM shuts down. Another fix is to get a weak little card to run your non-gaming VMs and to keep the host with a dedicated card. This is why having my i7-2600 is so good; The host and other VMs get to use the integrated graphics while the windows VM uses the HD 5850.


----------



## lloyd mcclendon

Quote:


> Originally Posted by *dizzy4*
> 
> The display will go blank because the host no longer owns the resources of the card.


ok i didn't catch that at all. That kind of explains Plan9's post then. Somehow I was expecting the card to be "shared" but obviously that is nonsense. So is it possible to hot detach / reattach a GPU while either system remains running? Reading about this here it sounds like you can do it for some pci devices, but I would think a GPU is pretty fundamental to remain the same while the machine is running. If not you'd need at least two cards.

I'll see if I can get something going and throw my welfare ATI card in there for some testing. Then if that works i'll consider removing the SLI bridge and running them as separate cards (eventually get a 3rd... and 4th card), and also a KVM switch.







I picked up a killawatt meter and this thing only pulls between 400-500W from the wall ... so 4 cards is more feasible than I thought.


----------



## lloyd mcclendon

alright well i tried this out last night ... and no luck. KVM has long had support for PCI pass through. You simply add a "PCI host device" in the VM definition. I had seen this before but never really knew what it was. very cool. This all works by using existing kernel code and theoretically should be* faster than xen which is more userland code. _If_ it would work ...

It all uses the intel_iommu and DMA remapping parts of the kernel. Either this is just really new and still has some bugs to shake out, or it's a real problem of hardware compatibility. Booting with intel_iommu=on caused the marvell sata controller on my board to cause a kernel panic at boot. Moving the plugs for two of my drives to the other controller and disabling this controller got past that and the system boots. However when I try to startx, the nvidia driver won't load... something about DMA remappings and bits already being set.







Either this card / this version of the driver / my config is incorrect. I will try to investigate that a bit further later ..

I think if i could get X to start using these cards, I could easily pass the ATI card I plugged in directly to a VM and it would be perfect. Once I get X started there's a few steps of mapping the busID / IRQ using the pci_stub module ... If that works I'll be dedicating at least one 570 to a windows VM for gaming and ditch this wine crap.


----------



## SoulCleaver

Hi there,

Firstly awesome guide, found it very useful re setup of Xen, there is a few issues I have and am hoping you can help me address them (struggling to find information/guides relevant to this OS/your build):

I am running an i870 (no on chip GPU so figuring this makes a difference tbh)

Trying to start a VM with a hdd image I made (I didnt use partitions as you did due to space issues on my small test drive) I got permissions issues (also got for ISO/CD didn't matter much where I shoved the ISO either

I ran 'setenforce 0' to remove the issue prior to trying passthrough of the GPU etc

When you perform a passthrough of hardware you initially get an error stating PCI-BACK doesn't have/own the hardware, I have found out how to fix this adhoc and that's fine tbh but thought I shoud post the fix for those whom don't know

'sudo lspci' (lists hardware ids) primary gpu should usually list as 01:00.0 my secondary listed as 06:00.0 ans 06:00.1 (sound for secondary card) this was an ATI card, take a note of the ID for the hardware in the above format you will need to unbind this hardware to use it on a VM

'lspci -n' - This gets you the vendor IDs (find the hardware id as per above step in this list and the code listed like xxxx:xxxx is what you seek)

3rdly to unbind the hardware I had to run the following

'echo "xxxx xxxx" > /sys/bus/pci/drivers/pci-stub/new_id
echo "0000:06:00.0" > /sys/bus/pci/devices/0000:06:00.0/driver/unbind
echo "0000:06:00.0" > /sys/bus/pci/drivers/pci-stub/bind'

substitute xxxx xxxx for the values you got from step 2 (xxxx:xxxx format (remove the : and replace with a space as above)

After doing all that I could boot and install an OS (Windows XP and Windows 7 were both setup as testers)

Passed through a dedicated Gigabit NIC and this works fantastically (unbind etc needed as above to make this work however) in either OS

Either OS can see the GPU (tried with an ATI 4350 and a Nvidia 560TI), and it installs the drivers but I get error code 43 (seems common when googling), issue has something to do with patches needing to be installed and this is where I get extremely lost

Any ideas re the patching? Also any tips on how to permanently resolve the other issues (not having to unbind on dom0 boot would be awesome)

BTW I did the unbind etc with the primary card and this caused major issues with the dom0 seems if I ran this via ssh/putty as its the primary GPU it freezes/crashes but can reload virt-manager, though nothing seems to work after this also (any tips here would also be cool)

If I can manage to get this working I can finally wipe and partition my SSD (thus making 2x VMs on it one for me and one for Wife with full GPU passthrough from my pair of 560Ti's

Thanks for all your help in advance and congratulations on an AWESOME guide,


----------



## dizzy4

It sounds like you are having a few hardware quirks, but I am glad to see it working for you! I am working on a configuration and a start-up script that will do all this for the user. Right now, I am about 85% done so stay posted. I am also working on a custom live-usb that will allow this without storage device changes that will include these scripts. SELinux can be a little bit cumbersome and I also usually disable it for use with Xen, so no worries there.

In regards to your PCI-BACK question, I have found a good amount of issues with it and my work-around was to get libvirt to disable it using virt-manager. This can also be done using the virsh console command. The script will take care of this for you, but I will mention how so you can get it up and running in the mean time. virsh nodedev-list will give you a list of all your devices. Find your PCI devices and use this command: virsh nodedev-dettach pci_0000_01_00_0. The device should match what you saw in the list and should work. Notice that "dettach" is misspelled... kind of silly, but that is just how the command is used. The next step would be to do: xm pci-attach win7a 01:00.0. This will attach it to domain 'win7a' and can be done before or after starting the VM. I hope that simplifies your unbind woes.

Nvidia support seems to be somewhat limited now. I hear that some people have had luck using certain reference cards and most of the older 8000 series cards. I am still unsure about patching, but it wouldn't be easy and would most likely require a rebuild of RPMs or a recompile from the source code. Supposedly Xen 4.2 includes the patches, but is still marked as unstable. I will see about including that in the next version of the guide.

I also experience problems when removing the primary video card. For me this is my integrated chip. The OS still seems functional, but will throw the occasional error and sometimes I need to reconnect via SSH. I also have a lot of issues passing through the iGPU to use for another guest. It has do do with the frame buffer of the iGPU being shared with main memory, but 4.1.2 is supposed to be patched to support it. I am spending much of my free time looking into these issues, so stay posted for that too.

P.S. Welcome to OCN!









P.P.S. If anyone wants to help, i would be happy to know how to get my Live-USB to boot Xen before booting into Fedora. This would make it easier for a lot of people who want to try this.

Best,
Dizzy4


----------



## RussianHak

Interesting thread, Booked for sure!


----------



## dizzy4

Small update! I removed unnecessary install components and fixed an issue I found with selinux -- turned it off









Also, I added some preliminary Windows 8 support! I just tried it with the public preview and it works pretty well. Things will only get better, but this is good news









I am also working on the first beta release! It will be huge and make things about 4 steps less and provide a couple scripts that will make things faster and easier. Most importantly, it will allow users to try this without changing their disk drives!


----------



## Peon

Quote:


> Originally Posted by *dizzy4*
> 
> Video card is your choice. I suggest using a newer Radeon for [...] HD audio on the card which means you won't need to pass through the motherboard audio to get sound from the VM. Of course, you could pass motherboard or another discrete Audio device through too.


Could you elaborate a little more on this? AFAIK Nvidia cards also have HD audio, otherwise HDMI wouldn't work properly.

Is it the difference in implementation, in that AMD cards have an mobo-style onboard sound card integrated into them whereas Nvidia cards pass a digital audio stream directly to the receiver, that makes the difference here? If that's the case, I guess Xen can't handle digital audio devices?


----------



## dizzy4

Quote:


> Originally Posted by *Peon*
> 
> Could you elaborate a little more on this? AFAIK Nvidia cards also have HD audio, otherwise HDMI wouldn't work properly.
> Is it the difference in implementation, in that AMD cards have an mobo-style onboard sound card integrated into them whereas Nvidia cards pass a digital audio stream directly to the receiver, that makes the difference here? If that's the case, I guess Xen can't handle digital audio devices?


If what you say is true, the fix might be as simple as passing through the onboard sound as well. If both were assigned to the VM it should work alright. I was also wondering if the virtualized sound could be passed through. Maybe that is where the problem is. Through reading I also found that there are several people trying to patch Xen to make it work and version 4.2 will integrate some of those.

So what I would try would be: Install your domU with virtualized graphics then change your guest config file to include nographic=1. From there restart the system with the card passed through AND the audio passed through. That might force it to work properly.

Another idea I just had is to whether or not passthrough is working for you at all. Have you gotten any device to work in a domU? If you are having trouble binding the device to pci-back there is also a workaround using libvirt that I explained to SoulCleaver.

Best,
Dizzy4


----------



## lloyd mcclendon

in my system here the video and audio parts of the card are separate line items in lspci ...

03:00.0 VGA compatible controller: nVidia Corporation Device 1081 (rev a1)
03:00.1 Audio device: nVidia Corporation Device 0e09 (rev a1)
04:00.0 VGA compatible controller: nVidia Corporation Device 1081 (rev a1)
04:00.1 Audio device: nVidia Corporation Device 0e09 (rev a1)

so you would have to just have 2 pci passthrough devices, 1 for the video @ 03:00.0 and one for the audio @ 03:00.1 ... you could also use a virtual sound card, which will fallback to the onboard one, although this isn't true passthrough so it will lag a few ms behind and your audio will be out of sync.

btw since it looks like kvm is going to be about a year or so behind on this, i did go ahead and get all the xen stuff patched in and installed. i've yet to have the time to give it a try.. i will soon


----------



## dizzy4

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> in my system here the video and audio parts of the card are separate line items in lspci ...
> 03:00.0 VGA compatible controller: nVidia Corporation Device 1081 (rev a1)
> 03:00.1 Audio device: nVidia Corporation Device 0e09 (rev a1)
> 04:00.0 VGA compatible controller: nVidia Corporation Device 1081 (rev a1)
> 04:00.1 Audio device: nVidia Corporation Device 0e09 (rev a1)
> so you would have to just have 2 pci passthrough devices, 1 for the video @ 03:00.0 and one for the audio @ 03:00.1 ... you could also use a virtual sound card, which will fallback to the onboard one, although this isn't true passthrough so it will lag a few ms behind and your audio will be out of sync.
> btw since it looks like kvm is going to be about a year or so behind on this, i did go ahead and get all the xen stuff patched in and installed. i've yet to have the time to give it a try.. i will soon


It's good to see nvidia devices are listed on separate lines -- the same as ATI. That means the guest should not require the onboard audio to be passed through. I saw that one proposed patch actually broke ATI functionality in favor of nvidia. Another thing I read was that they had to be passed as primary video devices. That would mean installing the OS to the domU then redoing the config file as I stated before -- nographic=1. I also heard that the patch tries to load the video bios from the card, so that might be the real issue.

Let me know if the patch works and I will include it in my first beta release.


----------



## buppus

Very interesting thread. I'm a native Windows 7x64 user, but I have a question:

Is this a better solution than Sandboxing/Virtualizing games to keep Windows lean?

I'm absolutely brand new to this but I've been looking at Sandboxie, trying to figure out if I can use it to install Steam/TF2 on my secondary drive [D:] while keeping my brand new SSD Windows7 installation [C:] clean and lean.

Any thoughts?


----------



## dizzy4

Quote:


> Originally Posted by *buppus*
> 
> Very interesting thread. I'm a native Windows 7x64 user, but I have a question:
> Is this a better solution than Sandboxing/Virtualizing games to keep Windows lean?
> I'm absolutely brand new to this but I've been looking at Sandboxie, trying to figure out if I can use it to install Steam/TF2 on my secondary drive [D:] while keeping my brand new SSD Windows7 installation [C:] clean and lean.
> Any thoughts?


This is really not the easiest solution for what you want. This keeps the entire operating system in a virtual machine and needs special instructions to get full performance. What you are doing by using sandboxie is just creating a logical volume on a hard disk that is protected. This gude has less to do with keeping windows lean than it does with running multiple OSes on one box simultaneously.

I think steam is just drag-and-drop with games and there is a manager program somewhere on OCN that helps with it. There also might be a wayt o link steam to the other drive. That's not really my area of expertise, but I am sure someone here can help you. Thanks for checking out my guide


----------



## kamleot

Dizzy4 congratulations for the tutorial, I'll be testing in the coming days, you tested or have any report of any nvidia card that is not "fx 3800, 4800, 5800" work on windows 7?


----------



## dizzy4

Quote:


> Originally Posted by *kamleot*
> 
> Dizzy4 congratulations for the tutorial, I'll be testing in the coming days, you tested or have any report of any nvidia card that is not "fx 3800, 4800, 5800" work on windows 7?


I hear that the earlier cards like the geforce 8000 series work well, but some of the newer cards have issues. I think the best bet is a normal VM install then delete the VM and recreate with nographic=1 in the configuration file. It made Windows 8 work and the passed video card was set as the primary video card. If it does not work, stay tuned for a patched version that will boot from a USB drive.


----------



## nexos

You might want to add a note that UEFI and Xen are incompatible at the moment. I tried UEFI with patched grub-legacy and grub2 and both errored out. I also was unable to get the pass through working without explicitly preventing the kernel modules for the Radeon and HDMI devices. I also have a PS/2 keyboard that is impossible to pass through without weird things happening to the host. The best workaround for that was to use Synergy but I had to adjust the network settings because NAT doesn't allow for host/guest network interactions. But I can report that I have Windows Vista running with full GPU support


----------



## faMine

Excuse me for being noobish, but is this a Virtual Machine within Linux? So your base operating system is Linux with a virtual Windows 7 for gaming?


----------



## dizzy4

Quote:


> Originally Posted by *nexos*
> 
> You might want to add a note that UEFI and Xen are incompatible at the moment. I tried UEFI with patched grub-legacy and grub2 and both errored out. I also was unable to get the pass through working without explicitly preventing the kernel modules for the Radeon and HDMI devices. I also have a PS/2 keyboard that is impossible to pass through without weird things happening to the host. The best workaround for that was to use Synergy but I had to adjust the network settings because NAT doesn't allow for host/guest network interactions. But I can report that I have Windows Vista running with full GPU support


The system I am using right now uses UEFI and there have not been any problems. I do not need to edit my Grub configuration at all other than adding iommu=1 to the Xen multiboot line.... wait did I forget to mention that in the guide? I thought I did. Anyway, I did not need to do anything special for the latest version of the guide to work properly. UEFI included. Are you using the most recent Hypervisor (4.1.2) and do you have the right packages installed? I am using libvirt as a workaround for hot-unplugging PCI devices that can then be passed to the guest.

About the PS/2 keyboard -- I really do not know the status of those devices with passthrough, but I know that USB 1.1 passthrough is possible without VT-d or IOMMU. Glad to hear Synergy is working just fine though!

About NAT -- I have been working on this issue for my next revision of the guide, but I am currently using a custom bridge and disabling the NetworkManager service. This lets the VMs be registered just the like host. I will most likely include a script in the next version, or a hotfix for now. NAT did work for the VMs, but they were assigned incompatible addresses and not all functions worked properly. Steam and Web browsing did







Eventually this will all be handled by the USB boot drive --- that is when I can figure out how to get a proper Grub2 boot menu on a live-cd image.

It is AWESOME that this is working for you too.







Thanks so much for reading and reporting. What are your specific UEFI specs and boot settings?
Quote:


> Originally Posted by *faMine*
> 
> Excuse me for being noobish, but is this a Virtual Machine within Linux? So your base operating system is Linux with a virtual Windows 7 for gaming?


yep


----------



## sks72

So would it be possible to create two virtual machines on a host with 2 GPU's and pass one GPU to each VM? I tried doing this before and failed miserably.


----------



## dizzy4

Quote:


> Originally Posted by *sks72*
> 
> So would it be possible to create two virtual machines on a host with 2 GPU's and pass one GPU to each VM? I tried doing this before and failed miserably.


I do not believe your motherboard supports VT-d, so you would need a different one before attempting this. If I had a larger system (not ITX) I would try it for myself. Currently I am not able to get my iGPU passed through to a guest, but it does work on the dom0 while the radeon powers one VM at a time.


----------



## val-amart

I am going to buy a new pc and i really want to use a 2 or 3 fanless GPUs (in addition to integrated GPU for the host). i was wondering if you happen to know (or if you could test) an SLI or CrossFire configuration in Xen or KVM? It would really make a huge difference for me because otherwise i will need to buy one powerfull noisy GPU, and i'd rather not







Thanks.


----------



## dizzy4

http://www.youtube.com/watch?v=L_g7ZBMWoLk

Here is a video of 2 machines with 2 cards (I think) and it's working alright. Still no word on crossfire and I have no way to test it... ITX board over here









But hey, it's great to see it working on an AMD platform!


----------



## sks72

Quote:


> Originally Posted by *dizzy4*
> 
> I do not believe your motherboard supports VT-d, so you would need a different one before attempting this. If I had a larger system (not ITX) I would try it for myself. Currently I am not able to get my iGPU passed through to a guest, but it does work on the dom0 while the radeon powers one VM at a time.


I was actually testing this out on a 1055T+990FX-UD5 (same board as above video) which has IOMMU support. I could find very little information on Multi-GPU passthrough, and what info I did find didn't elaborate on how to do it.


----------



## dizzy4

Quote:


> Originally Posted by *sks72*
> 
> I was actually testing this out on a 1055T+990FX-UD5 (same board as above video) which has IOMMU support. I could find very little information on Multi-GPU passthrough, and what info I did find didn't elaborate on how to do it.


The way the video is set up would seem pretty straight forward; one GPU per virtual machine. I assume that having two passed through to one machine in itself is not a problem and could be used separately with no issues. The problem could be when they are connected with a crossfire bridge. It might just work, but I know the hypervisor has to collect the interrupts and send them to the right VM. I don't think it would really be a problem, but we won't know until someone tries


----------



## sks72

Sounds like a challenge.









Apparently there are some patches required to make it work though:

http://old-list-archives.xen.org/archives/html/xen-devel/2010-05/msg00441.html

What version of Xen are you using? I was using 4.0.x, there may be some changes in newer version that make this easier.


----------



## dizzy4

Quote:


> Originally Posted by *sks72*
> 
> Sounds like a challenge.
> 
> 
> 
> 
> 
> 
> 
> 
> Apparently there are some patches required to make it work though:
> http://old-list-archives.xen.org/archives/html/xen-devel/2010-05/msg00441.html
> What version of Xen are you using? I was using 4.0.x, there may be some changes in newer version that make this easier.


I am using Xen 4.1.2 which seems like a breeze compared to the patching I have seen in wikis. 4.1.2 has some of the better patches applied and really makes this a lot easier. It tends to just work for older Nvidia cards (pre GTx 200 series) and Radeon HD series cards.

Scripts are coming along too. I am about 75% done (i think).


----------



## nexos

About the UEFI, I don't know what I did differently but it just didn't work. Reinstalled in BIOS mode and it works fine now. I have the exact same motherboard and processor as you. I'm going to put this project on the back burner. I can't get my builtin audio card to work well passed through; it keeps buzzing on the outputs and dropping input. I can 100% confirm that my graphics card works fully passed through.

One other thing I discovered going through the audio passthrough is a limitation in Xen. Only so many devices can be passed through before it runs out of PCI reserved memory space. I had to disable the Xen video device in Device manager to allow enough space for the audio card and
graphics card. This is a problem in Xen's BIOS and is not very high on their list of things to fix.


----------



## dizzy4

Quote:


> Originally Posted by *nexos*
> 
> About the UEFI, I don't know what I did differently but it just didn't work. Reinstalled in BIOS mode and it works fine now. I have the exact same motherboard and processor as you. I'm going to put this project on the back burner. I can't get my builtin audio card to work well passed through; it keeps buzzing on the outputs and dropping input. I can 100% confirm that my graphics card works fully passed through.


That is interesting. is your firmware updated to 1.3? I think it made it a lot easier for me iirc. I will try passing through my audio and see what the deal is for me, but since I have this thing hooked up to a TV for gaming and HTPC stuff I just use HDMI. I passed through the radeon's sound and it works pretty well. Every once in a while it has issues. For instance, with flash player open too long and not outputting audio (hulu) the sound will freeze up and the flash plugin will crash. The good news is that simply ejecting the audio portion with the windows eject hardware icon works. After it ejects it will just reconnect because xen forces it to.
Quote:


> Originally Posted by *nexos*
> 
> One other thing I discovered going through the audio passthrough is a limitation in Xen. Only so many devices can be passed through before it runs out of PCI reserved memory space. I had to disable the Xen video device in Device manager to allow enough space for the audio card and
> graphics card. This is a problem in Xen's BIOS and is not very high on their list of things to fix.


I will look into this too and see if there is a way I can patch it. Do you have shadow memory enabled to be auto configured or do you specify? Setting it to auto works for me, but forcing more might help some. Another thing to try is in your UEFI menu what do you have low MMIO align set to? I set mine to 1024M rather than 64M. It has something to do with IOMMU/VT-d so it might be worth looking into.


----------



## Roman2K

*@dizzy4*
Thank you very much for this neat, clean guide. Excellent concept, running Linux while sandboxing Windows in a mere VM for games. I'm looking to build a new PC to do exactly that now that I know it's possible.

I never liked the idea having a useless IGP in those otherwise fine Intel CPUs. Cannot stand cheapo awkward Virtu either. So much so that I was about to buy an old P67 + 2550K instead of a Z68 or H6x + 2500K. This Linux setup would make the IGP completely relevant again.

Ideally, I'm looking at the following configuration / usage pattern:

Monitor connected to the MB (HD3000).
Boot to Linux running the IGP. Meanwhile, the discrete GPU is off.
When I want to play a game, I start / resume a Windows VM, passing through the discrete GPU, waking it up for the duration of the VM.
Questions:

Is all this possible?
If so, do I get a window with the Windows VM displaying the video output computed by the discrete GPU? That would mean that the HD3000 is proxying the discrete GPU, correct?
Do I need a Z68 or will a H61 do?
Thank you for any help!


----------



## dizzy4

Quote:


> Originally Posted by *Roman2K*
> 
> *@dizzy4*
> Thank you very much for this neat, clean guide. Excellent concept, running Linux while sandboxing Windows in a mere VM for games. I'm looking to build a new PC to do exactly that now that I know it's possible.
> I never liked the idea having a useless IGP in those otherwise fine Intel CPUs. Cannot stand cheapo awkward Virtu either. So much so that I was about to buy an old P67 + 2550K instead of a Z68 or H6x + 2500K. This Linux setup would make the IGP completely relevant again.
> Ideally, I'm looking at the following configuration / usage pattern:
> 
> Monitor connected to the MB (HD3000).
> Boot to Linux running the IGP. Meanwhile, the discrete GPU is off.
> When I want to play a game, I start / resume a Windows VM, passing through the discrete GPU, waking it up for the duration of the VM.
> Questions:
> 
> Is all this possible?
> If so, do I get a window with the Windows VM displaying the video output computed by the discrete GPU? That would mean that the HD3000 is proxying the discrete GPU, correct?
> Do I need a Z68 or will a H61 do?
> Thank you for any help!


You are on the right track for sure. You are right that it can work, but unfortunately not with a LGA1155 'K' processor at this time. The problem is that intel has not included the proper instruction set (VT-d) in the 2600k and 2500k chips. The good news is that the 3930k and 3960x C2 revision chips have it enabled, but they don't have onboard GPUs. The most common option is a non-overclocking i5-2400, i5-2500 or i7-2600 chip. The next step is finding a motherboard with VT-d enabled in the BIOS. Your best bet for intel is ASrock. Most of their boards that can support VT-d have it in the bios. Another good chance would be Zotac. I am not sure about Z77 boards, but they are less likely to support it.

If you don't care about overclocking, an H67 would do just fine, but I have not seen an H61 with the feature enabled.

The HD2000 will not proxy anything and you will have to use the output from the physical card itself. The VM will not boot in a window, but rather have full hardware acceleration to it's own monitor (or another input on the one you are using).

Hope that helped. I really hope to see better support on enthusiast chips in the future.


----------



## Roman2K

Thank you very much for your thorough, helpful answer!









I'm staying with tried and true Sandy Bridge for now: Q67 + i5 2500. Mostly because finding a micro-ATX motherboard with VT-d (for certain) isn't easy. I found the Intel DQ67SW and the Gigabyte GA-Q67M-D2H-B3, as they clearly state the feature as present in their specs page or the manual.

Do you know if the GTX 680 works via IOMMU in Xen? Same question for the 580?


----------



## dizzy4

Quote:


> Originally Posted by *Roman2K*
> 
> Thank you very much for your thorough, helpful answer!
> 
> 
> 
> 
> 
> 
> 
> 
> I'm staying with tried and true Sandy Bridge for now: Q67 + i5 2500. Mostly because finding a micro-ATX motherboard with VT-d (for certain) isn't easy. I found the Intel DQ67SW and the Gigabyte GA-Q67M-D2H-B3, as they clearly state the feature as present in their specs page or the manual.
> Do you know if the GTX 680 works via IOMMU in Xen? Same question for the 580?


There are patches to enable support, but I am not sure which ones are needed and there are several that have been proposed. I am working (slowly) on getting a patched install .iso for you Nvidia users. Hopefully within a week I will have a more streamlined guide and within a month the patches and scripts will be available as well. ASrock H67 and some Z68/Z77 boards also have it.

I am always glad to help


----------



## Roman2K

Thanks for the motherboard suggestions. In the end, I bought the Gigabyte Q67.

Regarding the graphics card, I decided to play safe and go with an AMD (ASUS 7970 DCII) after seeing your comment, and given the need for flimsy patches and headaches, with nVidia cards.

It's very kind of you to put the effort into creating a turn-key solution for setting up a gaming VM. Personally, I don't like binary / generic solutions so I don't think I'll use it. I always compile packages from scratch and run hand-optimized kernels (Gentoo Linux distribution)... One has to satisfy their OCD







. Plus, Xen is part of the mainline kernel as of v2.6.39 which is fantastic. That way, we can always run the very latest kernel. I would gladly contribute to your project if there's any way I can help, though.


----------



## Roman2K

Someone successfully passed through his 6770 to a KVM container under Fedora 16:

YouTube video
Instructions
Awesome!


----------



## dizzy4

Quote:


> Originally Posted by *Roman2K*
> 
> Someone successfully passed through his 6770 to a KVM container under Fedora 16:
> 
> YouTube video
> Instructions
> Awesome!


Yeah that is cool! I almost went the KVM route for this guide (and my home system) but the Xen Hypervisor is a lot more efficient. Look at the CPU score







On Xen that would be at least a 7.6 on WEI. I know it's not the best benchmark, but ken does a lot better job with resources. The process looks a lot easier with KVM -- well at least until I get my Xen USB stick working.

Speaking of that, do you know how to get a proper grub2 bootloader on a fredora spin? I assume I would have to modify the plymouth initrd.gz.


----------



## Roman2K

What score are you talking about? I'm sorry, I don't follow.

Recently, I read this article and thought Xen was on par with KVM. Have you tried KVM yet, to compare?

Doesn't Fedora 16 already install GRUB 2?

If so, you can perform a regular install of Fedora with the stick as the target. It will create partitions and install GRUB.

Or, install on another media and when you achieved a final version, manually partition the stick, mount them in, say, /mnt/usb, use grub-install on the USB device node (/dev/sdX) like you would on a regular hard disk and then "(cd / && tar cp) | (cd /mnt/usb && tar x)".

I can't go into further details for now as I don't have a proper PC to experiment on while waiting for parts.


----------



## dizzy4

Quote:


> Originally Posted by *Roman2K*
> 
> What score are you talking about? I'm sorry, I don't follow.
> Recently, I read this article and thought Xen was on par with KVM. Have you tried KVM yet, to compare?
> Doesn't Fedora 16 already install GRUB 2?
> If so, you can perform a regular install of Fedora with the stick as the target. It will create partitions and install GRUB.
> Or, install on another media and when you achieved a final version, manually partition the stick, mount them in, say, /mnt/usb, use grub-install on the USB device node (/dev/sdX) like you would on a regular hard disk and then "(cd / && tar cp) | (cd /mnt/usb && tar x)".
> I can't go into further details for now as I don't have a proper PC to experiment on while waiting for parts.


I have read articles that also say Xen is a lot slower. There were even some that showed Xen on top. That review is also outdated and is still on Xen 3.x while the current Xen release is 4.1.2 You also have to keep in mind that Xen is a type 1 hypervisor and KVM is a type 2. In general, type 1 are more efficient and give better results when used properly. Xen also does Paravirtual machines better than KVM if I recall. I know that WEI isn't the best benchmark, but it still measures the CPU and all I know is that what I saw was a lower CPU score than mine (using the same processor).

Moving on to Grub2... The problem is that I want the .iso to include the proper boot loader so that anyone can use this technology. I could get it working for myself, but I want it to be ready for anyone to download the .iso, use the utility to put it on a USB stick and just boot it. I Think the key is in the fedora spin's .ks file. There is most likely a way to script it to use a full-featured boot loader in the boot image rather than the stock fedora live version.


----------



## WoodiE

I'm looking to build a new system using an Ivy Bridge i7-3770 cpu. Anyone know of a site or list, listing motherboards that support VT-d?


----------



## dizzy4

Quote:


> Originally Posted by *WoodiE*
> 
> I'm looking to build a new system using an Ivy Bridge i7-3770 cpu. Anyone know of a site or list, listing motherboards that support VT-d?


I'm not sure about a full listing, but I know that most ASrock 1155 boards offer it. The best way is to download a copy of the manual and just search for VT-d in the PDF.


----------



## buggystick

Great guide! Thanks for all the work putting it together. I'm so close that I can almost taste success, but as of yet I have been unable to get the VM to start with the GPU attached. I'm able to get to the point where 'xm pci-list-assignable-devices' shows the devices I'm wanting to assign to the guest, but every time I try starting the VM I get an error saying "Error: pci: PCI Backend and pci-stub don't own device xxxx:xx:xx.x".

I've tried using both pci-stub and xen-pciback, but am still met with the same error. I can confirm with lspci that they are in fact owned by pci-stub or pciback. Any ideas?

MB: Asus Crosshair Formula V
CPU: FX-8120
dom0 GPU: GTX 280
domU GPU: GTX 590
OS: Fedora 16
Xen: 4.1.2

I've also tried giving the 280 to domU, thinking that there might be issues with dual GPU cards, but I still get the same error.


----------



## dizzy4

Quote:


> Originally Posted by *buggystick*
> 
> Great guide! Thanks for all the work putting it together. I'm so close that I can almost taste success, but as of yet I have been unable to get the VM to start with the GPU attached. I'm able to get to the point where 'xm pci-list-assignable-devices' shows the devices I'm wanting to assign to the guest, but every time I try starting the VM I get an error saying "Error: pci: PCI Backend and pci-stub don't own device xxxx:xx:xx.x".
> I've tried using both pci-stub and xen-pciback, but am still met with the same error. I can confirm with lspci that they are in fact owned by pci-stub or pciback. Any ideas?
> MB: Asus Crosshair Formula V
> CPU: FX-8120
> dom0 GPU: GTX 280
> domU GPU: GTX 590
> OS: Fedora 16
> Xen: 4.1.2
> I've also tried giving the 280 to domU, thinking that there might be issues with dual GPU cards, but I still get the same error.


I know that there are a lot more issues with the Nvidia cards (GTx 200 series and newer), but it seems like you are having an issue actually getting it attached. I had a lot of trouble assigning and binding to pciback and the pci-stub, but libvirt actually does a good job at this. Do you have libvirt installed? I recommend that you do so and then run the following like I suggested to other posters:

Code:



Code:


virsh nodedev-list | grep pci
virsh nodedev-dettach pci_xxxx_xx_xx_xx
xm pci-list-assignable-devices
xm pci-attach <domain-name> xx:xx.x
xm pci-attach <domain-name> xx:xx.x


----------



## buggystick

Quote:


> Originally Posted by *dizzy4*
> 
> I know that there are a lot more issues with the Nvidia cards (GTx 200 series and newer), but it seems like you are having an issue actually getting it attached. I had a lot of trouble assigning and binding to pciback and the pci-stub, but libvirt actually does a good job at this. Do you have libvirt installed? I recommend that you do so and then run the following like I suggested to other posters:


That is actually the method I used to detatch. However, the xm pci-attach fails

Code:



Code:


# xm pci-attach win7a 0a:00.0
Error: pci: device 0000:0a:00.0 has been attached!
Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func> [virtual slot]

Insert a new pass-through pci device.

Here is what xm pci-list shows:

Code:



Code:


# xm pci-list-assignable-devices
0000:0a:00.0
0000:0a:00.1
0000:09:00.0
0000:09:00.1
0000:07:00.0
0000:08:00.0
0000:08:02.0




Spoiler: And lspci -v



Code:



Code:


07:00.0 PCI bridge: nVidia Corporation Device 05b9 (rev a3) (prog-if 00 [Normal decode])
        Flags: fast devsel
        Bus: primary=07, secondary=08, subordinate=0a, sec-latency=0
        I/O behind bridge: 0000b000-0000bfff
        Memory behind bridge: f4000000-f70fffff
        Prefetchable memory behind bridge: 00000000c0000000-00000000d9ffffff
        Capabilities: [40] Power Management version 3
        Capabilities: [60] Express Upstream Port, MSI 00
        Capabilities: [a0] Subsystem: nVidia Corporation Device 064e
        Kernel driver in use: pci-stub
        Kernel modules: shpchp

08:00.0 PCI bridge: nVidia Corporation Device 05b9 (rev a3) (prog-if 00 [Normal decode])
        Flags: fast devsel
        Bus: primary=08, secondary=09, subordinate=09, sec-latency=0
        Memory behind bridge: f6000000-f70fffff
        Prefetchable memory behind bridge: 00000000d0000000-00000000d9ffffff
        Capabilities: [40] Power Management version 3
        Capabilities: [60] Express Downstream Port (Slot+), MSI 00
        Kernel driver in use: pci-stub
        Kernel modules: shpchp

08:02.0 PCI bridge: nVidia Corporation Device 05b9 (rev a3) (prog-if 00 [Normal decode])
        Flags: fast devsel
        Bus: primary=08, secondary=0a, subordinate=0a, sec-latency=0
        I/O behind bridge: 0000b000-0000bfff
        Memory behind bridge: f4000000-f50fffff
        Prefetchable memory behind bridge: 00000000c0000000-00000000c9ffffff
        Capabilities: [40] Power Management version 3
        Capabilities: [60] Express Downstream Port (Slot+), MSI 00
        Kernel driver in use: pci-stub
        Kernel modules: shpchp

09:00.0 3D controller: nVidia Corporation Device 1088 (rev a1)
        Subsystem: nVidia Corporation Device 0868
        Flags: bus master, fast devsel, latency 0, IRQ 32
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at d0000000 (64-bit, prefetchable) [size=128M]
        Memory at d8000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at f7000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: pci-stub
        Kernel modules: nouveau

09:00.1 Audio device: nVidia Corporation GF110 High Definition Audio Controller (rev a1)
        Subsystem: nVidia Corporation Device 0868
        Flags: fast devsel, IRQ 33
        Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: pci-stub
        Kernel modules: snd-hda-intel

0a:00.0 VGA compatible controller: nVidia Corporation Device 1088 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: nVidia Corporation Device 0868
        Flags: bus master, fast devsel, latency 0, IRQ 34
        Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=128M]
        Memory at c8000000 (64-bit, prefetchable) [size=32M]
        I/O ports at b000 [size=128]
        Expansion ROM at f5000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: pci-stub
        Kernel modules: nouveau

0a:00.1 Audio device: nVidia Corporation GF110 High Definition Audio Controller (rev a1)
        Subsystem: nVidia Corporation Device 0868
        Flags: fast devsel, IRQ 35
        Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: pci-stub
        Kernel modules: snd-hda-intel


----------



## nezorf

This is exactly what what I have been getting...

[[email protected]~]# virsh nodedev-dettach pci_0000_07_00_0
Device pci_0000_07_00_0 dettached

[[email protected] ~]# virsh nodedev-dettach pci_0000_07_00_1
Device pci_0000_07_00_1 dettached

[[email protected] ~]# xm pci-list-assignable-devices
0000:07:00.0
0000:07:00.1

[[email protected] ~]# xm pci-attach havoc 07:00.1
Error: Failed to parse pci device: 07:00.1
Usage: xm pci-attach [-o|--options=] [virtual slot]

I am starting to wonder if it has something to do with my motherboard.. I have even tried another release with SL6.2.
using http://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/

This worked great as xen-pciback.hide actually is built into the kernel, but again the ATI card did not display any video...

Device supports FLReset from what i read here...

Code:



Code:


07:00.0 VGA compatible controller: ATI Technologies Inc Barts XT [ATI Radeon HD 6800 Series] (prog-if 00 [VGA controller])
        Subsystem: XFX Pine Group Inc. Device 3107
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Region 2: Memory at f7820000 (64-bit, non-prefetchable) [disabled] [size=128K]
        Region 4: I/O ports at a000 [disabled] [size=256]
        Expansion ROM at f7800000 [disabled by cmd] [size=128K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ [B]FLReset-[/B]
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 00000000fee00338  Data: 0000
        Capabilities: [100] Vendor Specific Information <?>
        Capabilities: [150] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
        Kernel modules: radeon


----------



## dizzy4

@buggystick That error shows that your device is already attached to a VM. The way things work out is that once bound to a VM, it will be included in the configuration (by libvirt) that is used whenever the VM is started. So in theory if the stub and pciback are working properly it will boot and be passed through just fine. Your issue may be something new that is unknown because Nvidia passthrough support is still very unstable, but it looks like you have done everything right.

@nezorf I am not sure I can help you a whole lot more because you are using a different distribution of redhat. The only thing I would suggest is connecting to the VM using VNC (this can be done with virt-manager) and then installing drivers. By default, the primary video device is the emulated one, but the radeon should show up in the device list. If you install the CCC drivers and reboot without connecting VNC to the VM it should then select the radeon for video output. Another thing to try if that does not work would be to delete the VM (xm delete), then edit the config file and set nographic=1. Then xm new and xm start . That should also set the radeon to the primary output and is how I was able to get Win8 consumer preview to work.

Hope it helps!


----------



## nezorf

So after much work. I have it working







Thanks Dizzy4!!!!!!

I think my issue was 2 things... First install the CCC drivers.. I did not do that at first.... and the no graphic option seemed to kick me past qemu screen.

Now to get some audio working... that still did not show up and possibly try loading OSX up on another one...

Saw a video here : http://www.youtube.com/watch?v=Q7v-5xSlgdk of some guy doing it. So i know it must be possible... Sort of end config would also would include a distro with lifespan then F16.

Again this is just a huge win thanks!


----------



## powerhouse

Dizzy4: Thanks for the guide.

I saw that some people managed to get it work, but I failed miserably. Actually, I failed long before even getting to your instructions, with the "simple" Fedora install. I couldn't get Fedora 16 working in my machine, then after some tries I managed to get Fedora 17 installed. But the install was so corrupted that I eventually gave up on Fedora.

I understand you chose Fedora for the out-of-box LVM support. I installed LMDE 12 (Linux Mint Debian Edition) now and it works out of the box. The Xen package also installed almost fine - it boots but there is one error line that I believe needs to be dealt with (some PCI error).

Before trying to fix things, do you have any experience or suggestions for getting the following hardware work with Xen and VGA passthrough (any Linux, though Linux Mint would be great):

ASUS Sabertooth X79 board (set to use UEFI, but it's also possible to use legacy BIOS setting)
i7 3930K processor with VT-d support (enabled in MB Bios)
PNY Quadro 600 GPU (Nvidia)
32 Gig memory
120 GB Sandisk Extreme SSD for OS and domU
WD 2000 Green hard disk for data
A bunch of other disks I will transfer from my old PC

Fedora 16 didn't work neither in Legacy BIOS mode nor in UEFI mode. The install disk did boot though, but the installation continued rebooting right after kernel selection.

How will the Nvidia Quadro 600 card work in your setup - any experience? I would want to install the proprietary Nvidia driver instead of the Nouveau OSS driver, as it provides better GPU support.

I hope to get it work somehow.


----------



## powerhouse

One more question: As mentioned before, I installed LMDE 12 but on a regular ext4 partition (/ and /home, /boot is on ext3). LVM2 is also working, including under Xen, but I use it for data drives and plan to use it for a Windows 7 domU.

Now the question: Is there a compelling reason to put the OS/Xen dom0 (LMDE) / and /home directories on LVM, or can I use my setup and just create the domU on the LVM partition?


----------



## powerhouse

Actually, I have yet another question: Do you use the dom0 Xen kernel as your day to day Linux environment, or do you install Linux onto a domU, side by side with your Windows domU?

I'm asking that since I thought to use the dom0 as my regular Linux desktop for everyday applications and the domU for Windows applications only. However, I can imagine to use a slim dom0 to run Linux as domU as well. What is your suggestion on that?


----------



## nezorf

I went through a bit of thinking about this last night... To use disk images or LVM. I went with LVM. My reasons being is it's one less layer that needs to be written to or read from.. Unless you are just passing raw disk partitions to the domU's. Though if you are looking to do thin provisioning the disk images could allow for that... That has screwed me in the past(i.e ran out of space). So allocating up front I think just makes sense...


----------



## nezorf

Quote:


> "I installed LMDE 12 (Linux Mint Debian Edition) now and it works out of the box. The Xen package also installed almost fine - it boots but there is one error line that I believe needs to be dealt with (some PCI error).
> Before trying to fix things, do you have any experience or suggestions for getting the following hardware work with Xen and VGA passthrough (any Linux, though Linux Mint would be great):


What are the errors? With this Distro I wonder if the first it has xen pciback built into the kernel or just a module or even at all? I have never used MINT so i am unsure.. I tried quite a few Redhat "Like" distro's with varying results.

Another thing to note is some ASUS before in the xen docs were thought not to work with xen.
Quote:


> For ASUS Desktop Boards, these have VT-d support enabled, but Asus does NOT support Linux, so you are on you're own with any Linux or Xen issues like broken BIOSes:


found at http://wiki.xensource.com/xenwiki/VTdHowTo

Possibly that is outdated information.


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> Dizzy4: Thanks for the guide.
> I saw that some people managed to get it work, but I failed miserably. Actually, I failed long before even getting to your instructions, with the "simple" Fedora install. I couldn't get Fedora 16 working in my machine, then after some tries I managed to get Fedora 17 installed. But the install was so corrupted that I eventually gave up on Fedora.
> I understand you chose Fedora for the out-of-box LVM support. I installed LMDE 12 (Linux Mint Debian Edition) now and it works out of the box. The Xen package also installed almost fine - it boots but there is one error line that I believe needs to be dealt with (some PCI error).
> Before trying to fix things, do you have any experience or suggestions for getting the following hardware work with Xen and VGA passthrough (any Linux, though Linux Mint would be great):
> ASUS Sabertooth X79 board (set to use UEFI, but it's also possible to use legacy BIOS setting)
> i7 3930K processor with VT-d support (enabled in MB Bios)
> PNY Quadro 600 GPU (Nvidia)
> 32 Gig memory
> 120 GB Sandisk Extreme SSD for OS and domU
> WD 2000 Green hard disk for data
> A bunch of other disks I will transfer from my old PC
> Fedora 16 didn't work neither in Legacy BIOS mode nor in UEFI mode. The install disk did boot though, but the installation continued rebooting right after kernel selection.
> How will the Nvidia Quadro 600 card work in your setup - any experience? I would want to install the proprietary Nvidia driver instead of the Nouveau OSS driver, as it provides better GPU support.
> I hope to get it work somehow.


Is your 3930k a C2 stepping? If not, VT-d is out of the question and will not work because it doesn't actually have the instructions. Furthermore, you are trying to use an Nvidia card, for which support is very limited. The driver does not matter at all if you cannot get the hardware passed through. Have you installed libvirt? Try attaching the video card and then connecting with VNC. Then check to see if it is in lspci. That will let you know if you have everyhting right so far.


----------



## powerhouse

@dizzy4: Thanks for the fast reply!

i7 3930K - it's the C2 stepping version with VT-d support. Here a shortened output of xm dmesg:

(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN) - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN) -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 32 KiB.
(XEN) VMX: Supported advanced features:
(XEN) - APIC MMIO access virtualisation
(XEN) - APIC TPR shadow
(XEN) - Extended Page Tables (EPT)
(XEN) - Virtual-Processor Identifiers (VPID)
(XEN) - Virtual NMI
(XEN) - MSR direct-access bitmap
(XEN) - Unrestricted Guest
(XEN) EPT supports 1GB super page.
(XEN) EPT supports 2MB super page.
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging detected.
(XEN) Brought up 12 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN) Xen kernel: 64-bit, lsb, compat32
(XEN) Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x193b000

libvirt is installed, too.

lspci gives me this (shortened):
01:00.0 VGA compatible controller: nVidia Corporation GF108 [Quadro 600] (rev a1)
01:00.1 Audio device: nVidia Corporation GF108 High Definition Audio Controller (rev a1)

I used virt-manager to create a Windows 7 install.
By the way, I couldn't start libvirtd (...not found)??? It also doesn't show in lsmod | grep libvirt* . Is there anything missing?

What do you mean by "Try attaching the video card and then connecting with VNC"?


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> lspci gives me this (shortened):
> 01:00.0 VGA compatible controller: nVidia Corporation GF108 [Quadro 600] (rev a1)
> 01:00.1 Audio device: nVidia Corporation GF108 High Definition Audio Controller (rev a1)
> I used virt-manager to create a Windows 7 install.
> By the way, I couldn't start libvirtd (...not found)??? It also doesn't show in lsmod | grep libvirt* . Is there anything missing?
> What do you mean by "Try attaching the video card and then connecting with VNC"?


It all looks good to me, except libvirtd not running and DMA not being enabled. You might need direct memory access for nvidia due to video bioses etc.

As I said before, nvidia support is very limited and you may not be able to get it working until patches are applied. there are several nvidia passthrough patches for xen right now, but they will be sorted out in 4.2. Current release is 4.1.2.

You are running a debian based distribution right? I know that libvirt is targeted toward fedora based systems using KVM, but it works with Xen as well and should be ble to work with debian too. I would suggest looking up how to install it properly on a debian system.

First of all, you need to get the card detached from your dom0. If you only have one video card, you will need to do this using SSH from another machine. This is what I use libvirt for. After you get libvirt working you should be able to bind the card to pci-back. You can also try hiding the card at boot by editing GRUB. You might need to set the nographic=1 option in your vm config file to get it to load the nvidia card as primary.

What I sugegsted with VNC was attach, but leave disabled the Nvidia card and use emulated graphics to see if the card shows up in the VM hardware list. This is lspci for a linux domU or the device manager for a windows domU.


----------



## powerhouse

Hello dizzy...:

Thanks again for your reply!

I installed Fedora 16 as domU, just to see if it works. Well it does, but as a fully virtualized install (I believe so, as virt-manager doesn't give me an option for paravirtualization). I believe also that the standard setup for virt-manager is to use VNC - at least it is selected (the other option is SPICE).

DMA enable: I will have to look in the Xen setup. I also wondered about this.

libvirtd not running: I don't know why I can't find it. It's installed, according to synaptic.

Debian: Yes, I use a Debian installation. It's LMDE (Linux Mint Debian Edition). Works without a problem, at least I can easily install it and the Xen kernel boots into the graphics screen. Unlike Fedora 16 or 17, which was a nightmare getting anything working. So I'd rather stick with Linux Mint or Debian.

"First of all, you need to get the card detached from your dom0. If you only have one video card, you will need to do this using SSH from another machine. This is what I use libvirt for. After you get libvirt working you should be able to bind the card to pci-back. You can also try hiding the card at boot by editing GRUB. You might need to set the nographic=1 option in your vm config file to get it to load the nvidia card as primary."

I first need to get libvirtd working. Well, I have no clew on how to do this, but I'll be searching. I hope it will work on LMDE.

I haven't checked yet what shows up with lspci under Fedora 16 domU.

I would try Fedora 16 or 17 again but I can't figure out how to get this working. Even when I got Fedora 17 "working" it was totally bunkers. Do I have such a weird hardware (I posted the hardware specs)? LMDE doesn't seem to have a problem with it. Go figure.

I'll look into the setup files for Xen to see if I can fix DMA and libvirtd.


----------



## powerhouse

nezorf: Thanks for the reply!

Re errors: I had to make some links to get virt-manager to install domU guests (the LMDE Debian edition doesn't use /urs/lib64... but /usr/lib/... instead, and there is also a .../xen-default link instead of a .../xen directory under /usr/lib/...). So two ln -s fixed this issue.

As I posted before, I still can't load libvirtd. I need to look into that. I know libvirt is installed (checking via synaptic).

I'm using a Debian version of Linux (Linux Mint Debian Edition or LMDE) so things will probably look a little different from Fedora. I had a hard time getting Fedora 17 installed (16 didn't work), and then had serious problems making it work - I gave up. So I really hope I can get VGA passthrough for a Win 7 install on LMDE work.

Re ASUS: I didn't have many options to choose an X79 motherboard. The Asus Sabertooth X79 is quite a nice board technically speaking, the other option being Gigabyte. The Gigabyte had not so favorable reports regarding BIOS support etc., though my old PC has a Gigabyte board that just works fine (really fine). The Asus Sabertooth has excellent cooling and also the BIOS setup feature is state of the art. I do not think I have a BIOS or motherboard compatibility issue, at least not with LMDE.

EDIT June 9, 2012: I had to return my PC after I discovered hardware issues (RAM issues). They are testing it now. Any issues I reported are most likely due to hardware problems that would show up randomly in the shape of boot errors, fs errors, file corruptions, and segfaults. Often enough everything seemed OK until some segfault creeped up, or a file error.


----------



## vesslan

Hi, Dear Dizzy, according to your guide, we made a try, and we added a video card of NV 9400 GT as the attached monitor input.
But we encountered a problem , could you help us?
Now, we can do the Step1~6 successfully, in Step7 we can see the card in Win7 device manager, then we downloaded the newest driver for HD 5850 and install it. However , when restart win7 we cannot enter the win7 OS again (the "start windows" will keep unmoving and silent status for ever) while in safe mode we can get to it and we can see the card HD 5850. How can we do?

1 we set nographic=1
2 we set gfx_passthru=0
3 experiment platform
i7-2600 (non-k)
ASUS p8 H67-M ITX/HT
XFX HD5850 for vga passthru, nv 9400 gt for monitor input in dom0
Xen 4.1.2
Fedora 16
Windows 7 64bit


----------



## tavi

Quote:


> Yeah that is cool! I almost went the KVM route for this guide (and my home system) but the Xen Hypervisor is a lot more efficient. Look at the CPU score On Xen that would be at least a 7.6 on WEI. I know it's not the best benchmark, but ken does a lot better job with resources. The process looks a lot easier with KVM -- well at least until I get my Xen USB stick working.


Actually I only passed 2 cpu cores to the kvm virtual machine, so 6.9 is decent. The first time I did the vga passthrough witk kvm I only was concerned with what worked, what didn't and what was the proper way to do this, I didn't care that much about performance.

Now, with Fedora 17, that solves (I think) all the problems I had regarding to USB and audio passthrough, I tryed to concentrate more on the aspects of performance and benchmarking. I passed 8 cores and my HD6770 to the VM with 4GB of ram and did some (I hope) proper benchmarks.

Most of them are >90% of native Windows 7 performance, with the exception of 2 gaming benchmarks Crysis and STALKER Call of Prypiat which are, well, way less then the rest of the results, this is the first time I did serious benchmarking so I might have done something wrong. I did played the Crysis Demo with all setting at max, and it was ok and I do have to say that the benchmarking tool for Crysis was working a bit strange so I would take those result with a pich of salt.

You should also check this out.


----------



## Pip Boy

sorry if its been mentioned but what's the difference between VT-X and VT-D ? Im in the process of buying a laptop and the processor of my choice is only VT-X (Core i7-3610QM) if its important for virtualisation to be greater than 90% efficiency then il wait for the Core i7-3720QM to be available for the laptop as that supports both.

(before anyone asks, initially i will use windows for a gaming machine but ultimately will always tend to run a laptop on Linux even if i have dual boot hence asking the question)


----------



## powerhouse

VT-d is a must, otherwise you can't connect to your VGA from within the Xen domU. In AMD slang it's called IOMMU.

VT-d and VT-X are two different things. Most (all?) new processors support VT-x which helps virtualization. However, the tutorial here uses VGA passthrough to connect the virtualized guest (domU) directly to the GPU. This requires VT-d (IOMMU) to work (it must also be enabled in the BIOS, which usually isn't by default).

VT-d is NOT supported by all Intel CPUs - in fact most don't support it. You can check on the Intel website to see which CPU supports it and which not. In some cases VT-d may be supported by the motherboard (usually older motherboards and CPUs).

You should also check that your laptop/motherboard actually allows you to enable VT-d (i.e. has VT-d support). So, having a VT-d enabled CPU in a laptop that doesn't allow you to enable VT-d won't help you either. The laptop/motherboard manufacturer may issue a new BIOS firmware that enables this feature. Check the laptop manual under BIOS settings for a VT-d option, and/or the BIOS release notes of the manufacturer. A VT-d option must be explicitly mentioned!


----------



## Roman2K

Quote:


> Originally Posted by *powerhouse*
> 
> In AMD slang it's called IOMMU.


AMD's IOMMU specification used to be called IOMMU, it's now called *AMD-Vi*.


----------



## DiNet

Quote:


> Originally Posted by *dizzy4*
> 
> Dizzy's Guide to Creating a Gaming VM
> Version: 0.0.3
> *Virtualization is the future of computing and is already being used in many capacities today. In writing this guide I hope to educate our community to it's benefits, usage and limitations. The more support it has, the further developed it will become. That is also my goal for this guide; to see it become simplified and offer broader support for hardware and software.*
> *Some Benefits:*
> 
> Reset the VM without having to hard reset the system
> Uninterrupted sessions of other VMs
> Ability to run the latest games on a VM
> Being able to use all that extra Memory and those Cores
> Dizzy4


Nice effort. Strange guide tho. Reset VM is a + somehow? I can as easy run uninterrupted VM's while my base OS is windows... Ability to run latest games on VM is same as ability to run latest games on my PC. Last point is even more out of the line...

The real advantage is having one very beefy server-like system that can give fully working game system over RDP to all my devices. For instance I really hate heavy laptops and don't see any reason to buy laptop packed with ssd, i7 and discrete graphics. On other side I'll have to still be in lan because synchronous connection is way over "gaming" laptops price.
Another huge advantage would be in family+kids environment.

Average consumer and virtualization is a waste of time.
Vmware > citrix btw


----------



## Roman2K

*@DiNet*
Wrapping Windows in a mere VM brings all kinds of advantages over polluting a host with it.

With such a configuration, you get the cleanliness and infinite flexibility of Linux for everything. Popular games are missing due to developers leaning toward the more popular OS (Windows). Not anymore







.

Now, Windows can be seen as throw-away launched just for the duration of a gaming session. All in a tight container (domU or KVM guest). None of its crap leaks out (messy installs, dirty voodoo-y registry, uncontrollable leftovers, you know the deal).

Not to mention copy-on-write snapshots and clones, easy single-file backups, flexible network configs, memory compression, ballooning and deduplication, etc...


----------



## DiNet

Dual boot.
What you describe is Linux guy that wants to play windows games.
Same can be achieved with dual boot. You can install windows, standard software that you need + whatever else you might think, make image of partition and just use it whenever you want fast clean install.

I'm not huge Linux fan, especially for desktop. Flaws are from managing system to software naming. My favorite is guide on installing network printer... I think it's good in terms of creating Sys-admin jobs and thats about it. Windows is simplifying their desktop and server OS that leads only to reduced need of IT staff overall. So me, being sys-admin, want to push Linux in company that I work.

Back on topic. If pc is intended for gaming with hardware for it, OS installed on it will be windows. Why do I need Linux if I want to game on that PC anyway? File backups are easy as hell on windows. You don't need any third party software or MS courses to use them, as opposed to most Linux tasks.

I'll try to make example of what I meant in my previous post.
Couple with 3 kids household. 5 low-mediocre PC's or laptops for day-to-day tasks and web browsing + media center + "smart house".
Server with xeons maybe even, top gpu and tons of ram to host VM's for playing latest games.
Again this does not come in favor of LInux on it since I can put on it ESXi that will use 100mb ram. And provide all that is described in topic?

I can see this as a way to use Windows as it would be "freeware" software tho. Instead of buying and activating it I'll go re-create VM or revert it to earlier state


----------



## lloyd mcclendon

in case you missed it, the entire point of this thread was to avoid the hassle of a dual boot AND run linux as your primary OS.

Your opinion of whether or not that is the best way to go is just noise to those of us who are actually following this thread for technical information, so please, move along. You are not going to convince anyone reading here, it is wasted cycles on both ends.


----------



## Roman2K

*@DiNet*
Quote:


> Same can be achieved with dual boot. You can install windows, standard software that you need + whatever else you might think, make image of partition and just use it whenever you want fast clean install.


Like lloyd mcclendon said, the whole point of this thread is to avoid dual boot, for obvious reasons.
Quote:


> I'm not huge Linux fan, especially for desktop. Flaws are from managing system to software naming. My favorite is guide on installing network printer... I think it's good in terms of creating Sys-admin jobs and thats about it. Windows is simplifying their desktop and server OS that leads only to reduced need of IT staff overall. So me, being sys-admin, want to push Linux in company that I work.


Linux gives you total control over your hardware (finely tunable kernel, proc + sysfs, whole host of commands). There are also GUIs like on Windows, exposing a fraction of the possibilities in an easy-to-use interface. The difference is Linux gives you the choice between the two, or even to combine both. As a result, it's extremely versatile. It runs the biggest supercomputers in the world just as it does a BeagleBone. Same kernel, same everything-as-a-file model. That's the beauty of it. You may not get it until you try it.
Quote:


> You don't need any third party software or MS courses to use them, as opposed to most Linux tasks.


As long as you use Linux like Windows (GUI), you don't need courses to use the former either. The difference is Linux gives you access to what's underneath the cover, and to not have a cover in the first place.
Quote:


> File backups are easy as hell on windows.


Obviously... If you're referring yo what I called "easy single-file backups", I meant the Windows disk image as a single file: easy to backup, snapshot, clone (possibly copy-on-write).
Quote:


> Couple with 3 kids household. 5 low-mediocre PC's or laptops for day-to-day tasks and web browsing + media center + "smart house".
> Server with xeons maybe even, top gpu and tons of ram to host VM's for playing latest games.
> Again this does not come in favor of LInux on it since I can put on it ESXi that will use 100mb ram. And provide all that is described in topic?


I like this model of thin clients. ESXi though... seriously? Proprietary = thanks but no thanks. And, it's old news.
Quote:


> Originally Posted by *lloyd mcclendon*
> 
> in case you missed it, the entire point of this thread was to avoid the hassle of a dual boot AND run linux as your primary OS.
> Your opinion of whether or not that is the best way to go is just noise to those of us who are actually following this thread for technical information, so please, move along. You are not going to convince anyone reading here, it is wasted cycles on both ends.


+1


----------



## PappaSmurfsHarem

I'm a little confused, and perhaps if I read this thread more in-depth I would have found my answer.

So how does video output work on this?

Assuming you get it setup so that the integrated is set for the Host and the discrete is set for the VM. Do you need two monitors? One to connect to the integrated to manage the host and the other to see the VM's output?

Or does the VM loop back through the host similar to how Lucid Logix works on some motherboards?

//edit

I suppose it is irrelevant though as my 2500K does not support it... What a bunch of wish wash... 2500 = VT-d... 2500K= NO VT-d....


----------



## Roman2K

*@PappaSmurfsHarem*
In case you still care despite your -K processor, the VM takes control of the graphics card and the output goes to a monitor plugged into it. If you only have one monitor that has several inputs, you can plug both graphics cards in to it and switch between the host and the VM but you can't have both outputs show at the same time.


----------



## PappaSmurfsHarem

Quote:


> Originally Posted by *Roman2K*
> 
> *@PappaSmurfsHarem*
> In case you still care despite your -K processor, the VM takes control of the graphics card and the output goes to a monitor plugged into it. If you only have one monitor that has several inputs, you can plug both graphics cards in to it and switch between the host and the VM but you can't have both outputs show at the same time.


I figured, so theres no way to control the VM from the host other then setting up an VNC server on the VM once it is running then VNC'ing to the VM?


----------



## Roman2K

Quote:


> Originally Posted by *PappaSmurfsHarem*
> 
> I figured, so theres no way to control the VM from the host other then setting up an VNC server on the VM once it is running then VNC'ing to the VM?


You can connect from the host to VM via RDP (run by the VM) or VNC (run by the host in QEMU for KVM or the hypervisor for Xen -- I think). With VNC, since the server is run by the host, there's nothing to do in the VM to set it up. A simple -vnc :0 (qemu-kvm option) or "vnc = 1" (Xen config. file) is enough.

*EDIT:* Third option (the default, actually, IIRC): let QEMU show the output to an SDL app (X client).


----------



## Xaero252

The real benefit of this will come when we allow parallel access to the GPU over VT-d.
In this case, you don't relinquish control of the GPU to the guest operating system, instead the GPU's resources are shared between the host and guest operating systems (or, if you fancy, a secondary GPU can be given to the guest, and primary can be given to the host, and the secondary GPU can render to the primary GPU's framebuffer) Some really smart X11 developer has already done this, although only in a laboratory environment, and both "guests" were seperate X sessions, the two X sessions were able to trade control of the GPU back and forth with ease, and with multiple GPU, load sharing was easily achieved as well. The same is possible with Virtual machines via VT-d however there is still much work to be done on it, iirc some early form of this type of behavior is available now, though I'm not certain how well it works.


----------



## Roman2K

Quick update: after switching from a dying ASUS 7970 DCII to a reference 7970 (Sapphire), GPU passthrough works out of the box with QEMU + KVM (no workarounds, no tricks).

All I needed:

a VT-d capable motherboard: Gigabyte GA-Z77MX-D3H
a VT-d capable CPU: i7-2600
an AMD graphics card: reference 7970*
Kernel with KVM and IOMMU enabled
qemu-kvm (fork of QEMU, soon to be merged upstream) compiled from Git as of two weeks ago
...and simply pass the following option to qemu-kvm: -device pci-assign,host=00:01.0

Where 00:01.0 is the BDF of the graphics card's VGA function (.0) on my particular system (use lspci to determine these). One can also pass through the HDMI audio function (.1).

The main GPU is the HD2000, controlled by Linux. When booting the Windows VM, its video output comes out to a secondary input of my monitor. When the VM boots up, the VNC output shows the Windows logo frozen after a second or two. I can also hear the fan slow down. Sign that the Windows AMD driver has taken over. Which means, it's time to switch inputs on the monitor and Windows is there, fully 3D accelerated!

In the end, there's really nothing to it. Almost turn key







.

* Not sure about the "reference" requirement: the ASUS DCII was freezing after a minute once passed through, but after weeks of struggle, I eventually found out it also artifacted on bare metal sometimes as soon as the Windows logo while booting. Pretty sure about the AMD requirement, as AMD employees like Joerg Roedel have actively contributed to QEMU for PCI passthrough to work.

---

Regarding doing the same thing with Xen, Michael Mair-Keimberger has accomplished it successfully as well, see his blog entry.


----------



## lloyd mcclendon

^ awesome, great to hear. I really need to get some IOMMU friendly hardware...

have you had much of a chance to use it yet? If so, how's the performance of everything?


----------



## Roman2K

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> have you had much of a chance to use it yet? If so, how's the performance of everything?


I don't have numbers yet, sorry. However, so far, I've played CoD:Black Ops and Medal of Honor (I should finally be playing BF3 soon, still downloading), tried out VLC (hardware video decoding) and WebGL games on Chrome. All perfectly smooth, I cannot tell the difference compared to bare metal. And that's with "only" 4 virtual CPUs, 4 GB of RAM, disk image on a Btrfs compressed (LZO) subvolume.


----------



## RocketW0rm

First of all, Thank you dizzy, for your formidable guide









This weekend i successfully planted a win7ult on my xen.

My Hardware:
Mainboard: Gigabyte 990FXA-UD3
Processor: Phenom II 955 BE
Ram: 16GB 1333 DDR3
Gfx #1: Radeon HD5450 (small passive cooled just for Dom0)
Gfx #2: Radeon HD7850

Here are my impressions:

Had to overcome some oddities with pciback settings, it didn't work til i compiled them into the kernel. (Maybe a AMD chipset thing, you may add this to your guide, dizzy)
After that it was easy to pass the 7850 to my win7.

What i noticed, i can reboot my domU several times (if i xm delete/xm new it each time), but then it gets really sluggish. I think there may be some memoryleak or not released interrupts. After a Dom0 reboot all is fine again.

Next thing is to get audio running. I can't pass the mainboard audio to the DomU, because xen is complaining that it doesnt work without another pci device, as it does for my old Conexant TV card. Is there any qm-emu audio driver which could be used to pass audio streams back to the main audio driver (like Virtualbox is doing it)?

As for performance, played a few hours Burnout Paradise, no problems, no hangs or anything. I'll post HeavenDX this evening.
Btw, did anyone check the throughput difference between HD image file and physical access to HD yet?


----------



## wTheOnew

Quote:


> Originally Posted by *RocketW0rm*
> 
> Btw, did anyone check the throughput difference between HD image file and physical access to HD yet?


I have, and the results seem a little unreasonable:



That's significantly faster than it was as a native Win 7 install.


----------



## RocketW0rm

Nice one









Did my HeavenDX comparision now.

Bare: with 16GB, didnt want to remove Memory, but i don't think it has too much impact.
Win7 HVM: 4GB and 8GB

Quite nice speed. Though i've seen some corrupted screens now and then. Just single frames complete white. I wonder where that comes from.


----------



## wTheOnew

Do you have the vcpus pinned to actual cpus? I had some weird issues until I pinned all of my VMs to specific cores. Did you install Catalyst Control Center or just the driver? I've heard of issues resulting from having CCC installed...

3DM Vantage comparison: http://3dmark.com/compare/3dmv/4190200/3dmv/4111825

I selected the closest system to my VM that I could find. The graphics scores are almost identical and the CPU score difference can probably be explained by the 3570K's higher clock. It's very much a negligible difference.


----------



## RocketW0rm

Quote:


> Originally Posted by *wTheOnew*
> 
> Do you have the vcpus pinned to actual cpus? I had some weird issues until I pinned all of my VMs to specific cores. Did you install Catalyst Control Center or just the driver? I've heard of issues resulting from having CCC installed...


No, don't have pinned the cpus yet. And yes i have installed the CCC.

Tried now with pinned cpus and removed the CCC. Here is the result.
The max FPS is still ~10% lower, but the overall score is better now (1182 -> 1217







).


----------



## stussyking

dizzy,
could u post a link to the live linux usb ?
I tried many times but only to have succeeded once and cant replicate it .
problem with detaching the vga hdmi sound and usb controller ..

cpu : xeon e3 1235
board : asus c206 p8b ws
gpu : hd5670


----------



## jdrews

Thanks Dizzy for a great guide! I got it working on my system. Here are my specs.

Fedora 17 x86_64
Xen 4.1.2
i7 3770
ASRock Z77 Extreme4
Sapphire AMD Radeon HD 6450 (passed through to a Win7 VM)

I can only get it to work if I manually run a small script that does the following:

Code:



Code:


#!/bin/bash

#01:00.0 VGA compatible controller: ATI Technologies Inc NI Caicos [AMD RADEON HD 6450]
virsh nodedev-dettach pci_0000_01_00_0

#01:00.1 Audio device: ATI Technologies Inc NI Caicos HDMI Audio [AMD RADEON HD 6450]
virsh nodedev-dettach pci_0000_01_00_1

#00:1d.0 USB Controller: Intel Corporation Panther Point USB Enhanced Host Controller #1 (rev 04) -- 2x grey 2.0 ports on back.
virsh nodedev-dettach pci_0000_00_1d_0

xm start win777

What I mean is that my xen-pciback.hide isn't functioning properly. So I have to dettach from dom0 after boot. i.e. this doesn't work in /etc/sysconfig/grub:
GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M xen-pciback.hide=(0000:01:00.0)(0000:01:00.1)(0000:00:1d.0)"

So I tried running that script in rc.local (rc.local apparently runs before xend), and I also dumped it in /etc/xen/auto (looks like this folder just takes sxp type files). At this point I can get it to work, but only if I manually run the script. I'm probably making this more complicated than it needs to, but I suppose I could mod my little script to loop and watch for a started xend. Hmm...

Also I'm seeing a drastic performance hit on my Crucial M4 128GB SSD when inside my little VM. It's running on a SATA2 port so maybe I'll try the SATA3


----------



## edalbkrad

is fedora an absolute requirement or can we use Mint debian instead?

I want to try running 3 instances of starcraft broodwar on my sig rig.
If this works i can have a LAN party with only 1 rig!


----------



## cedivad

I've just finished this. I didn't followed this howto (that seems to be well done), I didn't used a GUI and I did many things differently, but it works. It just took me one day, i'm surprised of that.


----------



## wTheOnew

Quote:


> Originally Posted by *edalbkrad*
> 
> is fedora an absolute requirement or can we use Mint debian instead?
> I want to try running 3 instances of starcraft broodwar on my sig rig.
> If this works i can have a LAN party with only 1 rig!


With enough time and effort anythings possible.







Fedora's not a requirement, but it will make the install a little different if you use a different distro. I'm running Debian Squeeze.


----------



## spikexp

Anybody who manage to do it can post a gaming benchmark?


----------



## wTheOnew

Quote:


> Originally Posted by *spikexp*
> 
> Anybody who manage to do it can post a gaming benchmark?


I could give Heaven a try tonight. What game do you want me to try? Preferably something free.









I posted a link to my 3DMark Vantage score on the last page.


----------



## spikexp

Quote:


> Originally Posted by *wTheOnew*
> 
> I could give Heaven a try tonight. What game do you want me to try? Preferably something free.
> 
> 
> 
> 
> 
> 
> 
> 
> I posted a link to my 3DMark Vantage score on the last page.


Didn't saw the 3dmark vantage, pretty cool.

I have no idea for a free game bench...
I would like to see some real in-game performance on heavy game.


----------



## wTheOnew

Quote:


> Originally Posted by *spikexp*
> 
> Didn't saw the 3dmark vantage, pretty cool.
> I have no idea for a free game bench...
> I would like to see some real in-game performance on heavy game.


Played some Half Life 2 last night on it and it never dropped below 140fps.







Guess I should install BF3...

Heaven was 10-90fps at 1280x1024. Forget what the score was. This isn't my main box so it has to deal with the old monitor for now.


----------



## XenosEnder

I apologize ahead of time if this is a noob idea/question.

Hypothetically - Assuming some small modifications and building the perfect VM gaming machine(multiple gpus etc) .....could you.....

Run essentially a networked gaming rig?

Have 2 maybe more? (how many more in theory?)

networked peripherals....

Sidebar: Is there software out there that can split your peripheral inputs mice, keyboard, monitors, controllers. (not like a KVM) That can essentially allocate certain peripheral inputs to designated VMs.

- Similar to fullscreen mode in gaming... each separate display (VM) is run full screen and has separate inputs. Diagram

I am not a programmer, but it seems like if there wasn't it could be created.

Now I have been a PC gamer my entire life. I have owned consoles but they receive maybe 10% of time average and 30% on a new release.

One example of why I see value here.
Lan's
Why do they suck?
-who wants to lug around their tower/peripherals?
-take down- setup - take down
- You cant even console lan anymore because all games are xbox live only (etc)
- Expensive when you think .... for every person there they have 1k ish in electronics......
- I could go on but I am sure your thinking what im thinking....

How we could bring lans back....
- To go to a lan....... all you need is peripheral inputs and in my mind...... a display device with bluetooth or Intel WIDI - you dont even need to lug a monitor -
- Lan center? Setup a few of these bad boys and all you need is a bunch of displays

Anyways. I am a noob. This is probably impossible. But please ANYONE AND EVERYONE comment/critique/critisize/ explain why it could work/ why it couldnt work. ANYTHING

Haha thanks. Appreciate your guys time!

TL; DR Networked Gaming Station Viable>?


----------



## wTheOnew

If I get what you mean, the short answer is yes.

You can pass through USB controllers. So say you had 4 video cards and 4 USB controllers (each with 2 USB ports)(very common for motherboards to have at least this). You could pass through a video card and USB controller to each VM. Plug a keyboard/mouse or controller for each VM into their respective USB ports. Voila, 4 gaming machines running off of one bad-ass computer. Keep in mind that you'll probably need at least a quad-core i7. (One physical core and one logical pinned per VM with dom0 floating across all of them). 8-core Sandy Xeon would be better...

The easiest way to get sound would be by using HDMI from the video cards for both video and sound.

You can have them all share a single bridged network adapter.

You can also pass through USB devices by ID instead of just passing the controller through, but it's harder to set up and IMO not as convenient as saying these are VM1's USB ports, these are VM2's, etc.


----------



## powerhouse

I'm back in the game of setting up a Xen system with VGA passthrough (had hardware/memory problems).

Here is what I've installed so far:

Linux Mint 13 64bit Mate desktop with Xen hypervisor.

My hardware:
Asus Sabertooth X79
i7 3930K with VT-d enabled !!!
32GB RAM
PNY Quadro 600 graphics card (Nvidia chipset)
120GB SSD drive for Linux and (future) Windows 7 Ult VM, with / and /home (and the VMs) using LVM.
2x WD2000EARX 2TB drives for data, using LVM
3-4 more HDs to add once everything is running.
DVD r/w

My questions:

1. Any chance of getting my Quadro 600 GPU work with VGA passthrough?
2. I don't really need a 2nd GPU but I also spend 50% of the time in Linux and would it be better to let the dom0 have its own GPU (instead of running Windows VM to access Linux via VNC or similar)?
3. Can I boot the Xen dom0 (my Linux desktop) and have access to the VGA, then start the Windows domU and switch the VGA to the Windows guest? And vice versa, exit Windows and go back to Linux dom0 so that it takes over control of the VGA?
4. If it would be better to have 2 VGA cards, can I mix Nvidia and AMD graphics cards? Of what I'm reading here, AMD seems to have better support for VGA passthrough and I would then get a card that's known to work with VGA passthrough. On the other hand, I much prefer to pass through my Nvidia Quadro card to Windows because of its enhanced graphics properties for photo editing.

I know this how-to was written for Fedora, but I have a real hard time getting along with Fedora. Some here have succeeded using Debian, I wonder how much similarity there would be to Linux Mint 13 Maya which is based on Ubuntu 12.04 with Mate desktop? I could also install Linux Mint LMDE which is Debian by nature. Any suggestions?

I've read different approaches to getting VGA passthrough work, and am confused about how to make it work. Your help is appreciated.


----------



## wTheOnew

1. The Quatro is probably going to suck to get working. (I'm an ATi person so this is hearsay) From what I've seen it's significantly harder to get nVidia adapters passed through. Some work as easy as ATi cards and some are probably going to be more effort than they're worth... Xen Wiki At the bottom of that they have a few links for nVidia cards that look promising. Nothing specific to the Quatro, but it's a starting point.

2. Your preference. I have the IGP dedicated to dom0 and I can't remember the last time I've had that monitor on. Mostly do stuff through putty or vnc, so it's not like you need one.

3. Short answer, no.

4. You can mix adapters. Don't expect to get multiple adapters working in a single VM. ATi is nice because they can be passed-through like any other PCI device. Once you figure out how to hide any PCI device from dom0 it's simple to get an ATi card passed through and working.

Go with whatever disto you're most comfortable with. It's probably going to take a couple installs to get it just right so don't worry if you screw something up along the way. I know it took me a while to get everything working right.


----------



## powerhouse

@wTheOnew: Thanks for your quick reply!

Your answer to 3. is a little bummer. With one GPA this would mean that I have to run my VM (Win7) all the time, even when I'm using Linux (via VNC, for example). Or can I just close Windows and attach the GPA to dom0?

My CPU does not have a GPU.

As to 1.: I know that the faster Nvidia Quadro cards (Quadro 4000, for example) are tested for VGA passthrough. I was hoping they use the same or similar design and software for the Quadro 600. I will try to find some info on the Internet.


----------



## Swiss

Got this working a few days ago. So far I have been doing all of my gamming (Mainly TF2s new MVM mode) on this.

As requested here are some benchmarks of the system.

Host System:
[email protected] (8 threads)
16GB DDR3 2133 quad channel
2x Radeon 5850 1GB
Fedora 64bit with Xen 4.1

Gaming VM:
5 threads
4GB
1x Radeon 5850
Win7 64bit





*HyperPI*
4 threads (4 core processor) 1M digits
Average time 10.964
Best time 10.89
Worst time 11.031


*HL2 ep2*

playing first few minutes of game

Max graphics. 8xAA
@2560x1440
av 90
min 48 (at bridge)
max 190-ish

@1920x1080
av 105
min 70
max 300 (game limit)

*3dmark 11* Performance
@defaults (720)
Score: P3863
Graphics: 3563
Physics: 7293
Combined: 3649

Graphics test 1: 18.75fps
G test 2: 18.14
G test 3: 22.41
G test 4: 9.52
Physics Test: 23.16
Combined: 16.98

*3dmark 11 (8 threads)* Performance
P3909
graphics 3558
physics 8804
combined 3508

I watched a Starcraft 2 player play on this system on the dell 27" @ 2560x1440, all max graphics settings and he could not discern a difference between the VM and native.

Will post some more game benches, Native Benches for comparison and benches of 2 VMs running the same game on this box when I get back from holiday.

enjoy


----------



## wTheOnew

Quote:


> Originally Posted by *powerhouse*
> 
> @wTheOnew: Thanks for your quick reply!
> Your answer to 3. is a little bummer. With one GPA this would mean that I have to run my VM (Win7) all the time, even when I'm using Linux (via VNC, for example). Or can I just close Windows and attach the GPA to dom0?
> My CPU does not have a GPU.
> As to 1.: I know that the faster Nvidia Quadro cards (Quadro 4000, for example) are tested for VGA passthrough. I was hoping they use the same or similar design and software for the Quadro 600. I will try to find some info on the Internet.


You don't need to be running the Win 7 VM all the time, but you wont be able to easily switch between the Win 7 DomU and the Linux Dom0 on the same GPU. I think it's possible to hot-passthrough devices, but it's no where near as easy as hiding them at boot time. Into the land of things I haven't done so I'm just going by what I've read along the way.


----------



## powerhouse

@wTheOnew: thanks for the reply! I thought so.

I'm now considering getting a second graphic adapter, and I'm thinking about an AMD card.

I've been checking the AMD lineup and found a number of cards with high bit depth of 30 bit I believe (i.e. 10 bit per color channel). But I read about difficulties getting them working in that high-bit mode even under Windows. I was hoping to use a lower end AMD Radeon card for the dom0 (Linux) and try to pass through the Nvidia card to the domU Win7. I know that Nvidia can be tricky to pass through, and I've also read in forums that newer drivers may cause problems. Hope to get it working.

I'm also not so sure how Xen or Linux will take the two different cards and drivers altogether. But if I use the Nvidia exclusively for the Win7 domU - assuming I get it working - I don't see a reason why not.

Any experience with mixing Nvidia with AMD graphic adapters? I know it won't work under the same domU, but i need one for Linux dom0 and one (the Nvidia hopefully) for Win7 domU. The Nvidia has the advantage of being very easy on my PSU, only 30W, but it does a good job at photo editing.


----------



## powerhouse

@Swiss: Congratulations, this looks like a great setup.

I'm very much interested in how you got it working, since I have similar hardware. Here my hardware specs:

i7 3930K with VT-d support (Sandybridge-E)
Asus Sabertooth X79 board with VT-d support enabled in BIOS
PNY Quadro 600 graphics adapter - with Nvidia chip (here we differ)
32GB Kingston 1600MHz RAM
120GB SSD
2x 2TB WD drives
3 or 4 other drives I still need to install

I'd much prefer to use Linux Mint 13 Mate as my Linux desktop - have been using LM for 5-6 years now. I had installed Fedora 16 and 17 but it gives me fits. It's rather hard for me to get used to it.

I have successfully installed LM13 Mate and the Xen hypervisor with / and /home on LVM using ext4, and have some 60GB spare on my SSD for the Win7 domU. Both the regular LM13 and the Xen hypervisor boot.

Boot time with the Xen hypervisor is substantially longer than just LM13. The latter takes about 20 seconds to get a full desktop, the Xen hypervisor takes 2 minutes to boot.

I am considering getting a second graphics adapter, perhaps an AMD Radeon like yours. However, I would much prefer to use the Nvidia for the domU since it I know it works well with Win7 in high bit depth mode (more than 8 bit per color channel). My screen supports 12 bit per channel and has a wide color gamut for photo editing, which is what I want to do under Windows.

If Adobe would issue Photoshop and Lightroom for Linux, I'd happily dump Windows. But as it is, I can either get a Mac or a Windows PC to do professional photo editing.

I don't need super graphics performance (for games), but I must have direct access to the graphics adapter in order to use my screen calibration hardware/software which uploads the color correction curve into my screen via the DVI link. I'll also need to pass through some USB ports, but I hope that this is the easiest part. The screen comes with 2 DVI connectors that I can select via buttons (I believe).

If Linux Mint doesn't work, I would grudgingly accept Fedora 17 in the hope I could get a second domU with Linux Mint working. But I'd rather use the dom0 as my Linux desktop, if possible, or alternatively have a small hypervisor installed and run 2 domU for Win and Linux.

Any suggestions or instructions on how to make it work are welcome.


----------



## powerhouse

@Swiss: With regard to your disk performance index under Windows, are you using LVM? If yes, are you running a regular hard drive (not a SSD)?

Just curious cause I thought LVM would improve disk performance and somewhere I even saw that it might be better than on native Windows 7 partitions.

While disk performance is not paramount for me, it could slow down my work flow for photo editing/RAW conversion, with typical file sizes of some 20 MB to several 100s of MB.


----------



## wTheOnew

Quote:


> Originally Posted by *powerhouse*
> 
> @wTheOnew: thanks for the reply! I thought so.
> I'm now considering getting a second graphic adapter, and I'm thinking about an AMD card.
> I've been checking the AMD lineup and found a number of cards with high bit depth of 30 bit I believe (i.e. 10 bit per color channel). But I read about difficulties getting them working in that high-bit mode even under Windows. I was hoping to use a lower end AMD Radeon card for the dom0 (Linux) and try to pass through the Nvidia card to the domU Win7. I know that Nvidia can be tricky to pass through, and I've also read in forums that newer drivers may cause problems. Hope to get it working.
> I'm also not so sure how Xen or Linux will take the two different cards and drivers altogether. But if I use the Nvidia exclusively for the Win7 domU - assuming I get it working - I don't see a reason why not.
> Any experience with mixing Nvidia with AMD graphic adapters? I know it won't work under the same domU, but i need one for Linux dom0 and one (the Nvidia hopefully) for Win7 domU. The Nvidia has the advantage of being very easy on my PSU, only 30W, but it does a good job at photo editing.


Once you've hidden the nVidia card with pciback from dom0 it'll be like it's not even there. No drivers will get loaded for it or anything. The Win7 domU will have direct control over it using Windows drivers. So it shouldn't be a problem.


----------



## CyberInferno

Thanks so much for this guide. I'm converting my desktop to a hypervisor to run a couple servers, and I'd like the option to play games on it without having to dual-boot (and consequently shut down the running servers). I'll report back with my results!


----------



## N0BOX

Wow! I've been looking for exactly this since I built my current desktop. I specifically looked for a CPU/Motherboard that both support VT-d, only to find out that the only VMWare product that supports VT-d is ESXi, and it is impossible to view a VM's screen using the ESXi host's monitor.

Just for clarification's sake, is the Fedora install the "host OS", or is it just installed so that it can be turned into some sort of Xen bare-metal hypervisor? I know next to nothing about Xen, other than the fact that my Linode runs on a Xen server. :/


----------



## powerhouse

Quote:


> Originally Posted by *wTheOnew*
> 
> Once you've hidden the nVidia card with pciback from dom0 it'll be like it's not even there. No drivers will get loaded for it or anything. The Win7 domU will have direct control over it using Windows drivers. So it shouldn't be a problem.


Thanks for your reply!

The trouble is hiding the damn PCI device (VGA). My Linux Mint edition (LM13 Maya Mate 64bit) doesn't have xen-pciback compiled into the kernel. So I tried different ways of loading the nodule at boot, with no success (it's loaded, but way after the nouveaux Nvidia driver takes control).

Here is what I tried:

1. Put xen-pciback.hide=... in grub.conf as boot parameter
2. Put xen-pciback into /etc/modules.cfg to load
3. Installed xen-pciback into the initram file system
4. Tried insmod and modprobe in /boot/grub.cfg to load the module before the nouveaux driver

No matter what I do the nouveaux Nvidia driver is loaded and the VGA card is bound to dom0. I also can't remove the device from the PCI port once it's assigned to the nouveaux driver.

The PCI parameters are correct as per lspci.

Here is the /boot/grub.cfg file:


Spoiler: grub.cfg



#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
set have_grubenv=true
load_env
fi
set default="0"
if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi

function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}

function recordfail {
set recordfail=1
if [ -n "${have_grubenv}" ]; then if [ -z "${boot_once}" ]; then save_env recordfail; fi; fi
}

function load_video {
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
}

insmod lvm
insmod part_msdos
insmod ext2
set root='(lm13-root)'
search --no-floppy --fs-uuid --set=root f9e21c53-7594-4d48-a8fe-64e7912cbe6a
if loadfont /usr/share/grub/unicode.pf2 ; then
set gfxmode=auto
load_video
insmod gfxterm
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
set locale_dir=($root)/grub/locale
set lang=en_US
insmod gettext
fi
terminal_output gfxterm
if [ "${recordfail}" = 1 ]; then
set timeout=-1
else
set timeout=10
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/06_mint_theme ###
set menu_color_normal=white/black
set menu_color_highlight=white/light-gray
### END /etc/grub.d/06_mint_theme ###

### BEGIN /etc/grub.d/10_linux ###
function gfxmode {
set gfxpayload="$1"
if [ "$1" = "keep" ]; then
set vt_handoff=vt.handoff=7
else
set vt_handoff=
fi
}
if [ ${recordfail} != 1 ]; then
if [ -e ${prefix}/gfxblacklist.txt ]; then
if hwmatch ${prefix}/gfxblacklist.txt 3; then
if [ ${match} = 0 ]; then
set linux_gfx_mode=keep
else
set linux_gfx_mode=text
fi
else
set linux_gfx_mode=text
fi
else
set linux_gfx_mode=keep
fi
else
set linux_gfx_mode=text
fi
export linux_gfx_mode
if [ "$linux_gfx_mode" != "text" ]; then load_video; fi
menuentry 'Linux Mint 13 MATE 64-bit, 3.2.0-27-generic (/dev/sda1)' --class linuxmint --class gnu-linux --class gnu --class os {
recordfail
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
linux /vmlinuz-3.2.0-27-generic root=/dev/mapper/lm13-root ro quiet splash $vt_handoff
initrd /initrd.img-3.2.0-27-generic
}
menuentry 'Linux Mint 13 MATE 64-bit, 3.2.0-27-generic (/dev/sda1) -- recovery mode' --class linuxmint --class gnu-linux --class gnu --class os {
recordfail
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
echo 'Loading Linux 3.2.0-27-generic ...'
linux /vmlinuz-3.2.0-27-generic root=/dev/mapper/lm13-root ro recovery nomodeset
echo 'Loading initial ramdisk ...'
initrd /initrd.img-3.2.0-27-generic
}
submenu "Previous Linux versions" {
menuentry 'Linux Mint 13 MATE 64-bit, 3.2.0-23-generic (/dev/sda1)' --class linuxmint --class gnu-linux --class gnu --class os {
recordfail
gfxmode $linux_gfx_mode
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
linux /vmlinuz-3.2.0-23-generic root=/dev/mapper/lm13-root ro quiet splash $vt_handoff
initrd /initrd.img-3.2.0-23-generic
}
menuentry 'Linux Mint 13 MATE 64-bit, 3.2.0-23-generic (/dev/sda1) -- recovery mode' --class linuxmint --class gnu-linux --class gnu --class os {
recordfail
insmod gzio
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
echo 'Loading Linux 3.2.0-23-generic ...'
linux /vmlinuz-3.2.0-23-generic root=/dev/mapper/lm13-root ro recovery nomodeset
echo 'Loading initial ramdisk ...'
initrd /initrd.img-3.2.0-23-generic
}
}
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/10_lupin ###
### END /etc/grub.d/10_lupin ###

### BEGIN /etc/grub.d/20_linux_xen ###
submenu "Xen 4.1-amd64" {
menuentry 'LinuxMint GNU/Linux, with Xen 4.1-amd64 and Linux 3.2.0-27-generic' --class linuxmint --class gnu-linux --class gnu --class os --class xen {
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder iommu=1 iommu=pass-through dom0_mem=2048M xen-pciback.hide=(01:00.0)(01:00.1)
echo 'Loading Linux 3.2.0-27-generic ...'
module /vmlinuz-3.2.0-27-generic placeholder root=/dev/mapper/lm13-root ro quiet splash
echo 'Loading initial ramdisk ...'
module /initrd.img-3.2.0-27-generic
}
menuentry 'LinuxMint GNU/Linux, with Xen 4.1-amd64 and Linux 3.2.0-27-generic (recovery mode)' --class linuxmint --class gnu-linux --class gnu --class os --class xen {
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder iommu=1 iommu=pass-through dom0_mem=2048M xen-pciback.hide=(01:00.0)(01:00.1)
echo 'Loading Linux 3.2.0-27-generic ...'
module /vmlinuz-3.2.0-27-generic placeholder root=/dev/mapper/lm13-root ro single
echo 'Loading initial ramdisk ...'
module /initrd.img-3.2.0-27-generic
}
menuentry 'LinuxMint GNU/Linux, with Xen 4.1-amd64 and Linux 3.2.0-23-generic' --class linuxmint --class gnu-linux --class gnu --class os --class xen {
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder iommu=1 iommu=pass-through dom0_mem=2048M xen-pciback.hide=(01:00.0)(01:00.1)
echo 'Loading Linux 3.2.0-23-generic ...'
module /vmlinuz-3.2.0-23-generic placeholder root=/dev/mapper/lm13-root ro quiet splash
echo 'Loading initial ramdisk ...'
module /initrd.img-3.2.0-23-generic
}
menuentry 'LinuxMint GNU/Linux, with Xen 4.1-amd64 and Linux 3.2.0-23-generic (recovery mode)' --class linuxmint --class gnu-linux --class gnu --class os --class xen {
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
echo 'Loading Xen 4.1-amd64 ...'
multiboot /xen-4.1-amd64.gz placeholder iommu=1 iommu=pass-through dom0_mem=2048M xen-pciback.hide=(01:00.0)(01:00.1)
echo 'Loading Linux 3.2.0-23-generic ...'
module /vmlinuz-3.2.0-23-generic placeholder root=/dev/mapper/lm13-root ro single
echo 'Loading initial ramdisk ...'
module /initrd.img-3.2.0-23-generic
}
}
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
linux16 /memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
insmod part_msdos
insmod ext2
set root='(hd0,msdos1)'
search --no-floppy --fs-uuid --set=root 6e90997b-fe9a-4bd9-b34d-ae5bf1e1792b
linux16 /memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f $prefix/custom.cfg ]; then
source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###


----------



## RocketW0rm

Hi powerhouse,

i had to compile the kernel to get pciback running (fedora 16/17). It didnt work for me at all as module, but after i compiled it into the kernel all is fine.
running a win7, xp and even a 98SE as hvm nearly without a problem. only radeon drivers glitch on my 2nd monitor sometimes.


----------



## powerhouse

I made it ! (Well, almost - I need a second DVI cable.)

Bought a cheapo AMD 6450 graphics card, installed it in the first VGA slot and moved my Nvidia Quadro 600 into the second slot.

Now I can use Linux Mint on dom0 and Windows 7 on domU each using its own graphics card.

Will write more once everything is cleaned up.

Thanks for all your help and encouragement!


----------



## powerhouse

I was celebrating too soon :-(

I bought a cheap AMD 6450 to be used for dom0, in the hope I can get my Nvidia Quadro 600 card to work with passthrough.

Boot dom0, able to detach the Nvidia card and attach it to pciback. Start the domU via virt-manager, installation of Win7 went fine.

Win7 recognizes the Nvidia card and shows it in the device manager. I installed the latest Nvidia driver (305.93), but no matter what I did to the win7.cfg file under /etc/xen, it wouldn't activate the card (error 43 in Windows - yellow triangle in device manager).

Next I replaced the Windows Nvidia driver to version 275.89 (latest 275.x driver) but it's just the same.

Then I added the "passthrough=1" option to the /etc/module "xen-pciback" line to use passthrough instead of vpci (see http://wiki.xen.org/wiki/Xen_PCI_Passthrough). After rebooting my PC I got to the graphical user login, entered credentials and the computer froze with a black screen and the cursor in the middle - only hard reset helped (ssh didn't work as well).

So right now the domU gets a different (virtual) PCI ID, as I'm using vpci.

Can anyone who made this VGA/PCI passthrough work let me know if vpci works too? Run

Code:



Code:


dmesg | grep pciback

Look at the first line of the output and let me know if you see vpci or pass.

Any suggestions? Do I need to compile Xen with patches to make it work, or even compile a new kernel?

Can anyone with an Nvidia card share his/her /etc/xen/win7.sxp or .cfg config file? Or is it pointless in trying to make the Nvidia work?


----------



## SCAndroid

This thread...
I've been thinking about doing this for 2 months now and never found any info on it. And here it is, all neatly bundled into 1 handy thread.
I've even been planning a new rig to use as VMhost with workstation and gaming VMs.

Small question though, and I'm not sure if it has been brought up before. I was thinking of ArchLinux to use as dom0 as it's so very lightweight. What are your ideas on that? (I also have way more hands on experience with Arch than with Fedora, I made the switch aeons ago)


----------



## powerhouse

Quote:


> Originally Posted by *SCAndroid*
> 
> This thread...
> I've been thinking about doing this for 2 months now and never found any info on it. And here it is, all neatly bundled into 1 handy thread.
> I've even been planning a new rig to use as VMhost with workstation and gaming VMs.
> Small question though, and I'm not sure if it has been brought up before. I was thinking of ArchLinux to use as dom0 as it's so very lightweight. What are your ideas on that? (I also have way more hands on experience with Arch than with Fedora, I made the switch aeons ago)


I'm not an ArchLinux guy, but their wikis and help docs are among the best. Isn't ArchLinux based on Fedora? If my memory doesn't fail me, it should be. So most of what's been said here should work.

One warning though: Make sure your hardware supports VGA passthrough. I am still trying to passthrough a Nvidia Quadro 600 card and it's a pain in the neck. You need to check the following hardware components for compatibility:

CPU - as for Intel CPUs, they MUST support VT-d !!! i7 non-K processors are supported, as well as the 3930K (which I have) with C2 stepping. Same for i5 processors. The newest Ivybridge processors are also supposed to support VT-d, but check before you buy! AMD has a different name for it (can't remember), again not all CPUs support it.

Motherboard - must have a VT-d or equivalent option for AMD (IOMMU ?) in the BIOS setup. Even with VT-d or IOMMU support in BIOS, some are more buggy than others. Check this thread or others for proven support / VGA passthrough success. A good place to start is looking at http://wiki.xen.org/wiki/VTdHowTo, but it's not up-to-date and the CPU stuff (see Intel CPUs) is misleading (for example, an i7 2600K CPU won't have VT-d - see above). But the best is to read through this thread for proven solutions!

Graphics adapter: On-chip graphics adapters on Intel i7 CPUs such as a i7 2600 (non-K !!!) should be supported. For discrete graphics cards, AMD looks like the better choice for out-of-the-box support, but again, go for proven solutions (=cards). Nvidia-based graphics cards are difficult. AFAIK only the Nvidia Quadro 4000 and 3000 are officially supported. Also, Nvidia doesn't support Linux and open source, though their Linux proprietary drivers are often better than the AMD counterpart and they don't drop support so fast (I've had Nvidia cards working with proprietary drivers for many years, different cards too, and they are still supported). With regard to VGA passthrough, AMD seems the better choice right now. Important: Search the Internet or this thread for proven hardware that works !!!

A last note: There are a lot of hardware pieces that need to fit together to make it work. As for VT-d (Intel slang) or IOMMU, this is essential. VT-d or better IOMMU is a memory controller function which is either handled in the motherboard chipset (usually older systems) or nowadays with the Intel Sandybridge, Sandybridge-E, or Ivybridge in the CPU itself. So before you buy anything, make absolutely sure it will work with VGA passthrough.


----------



## powerhouse

I made it !!!

The Nvidia Quadro 600 card didn't work with VGA passthrough, at least not without patching/compiling a new kernel / xen.

I replaced the Quadro 600 for a Quadro 2000, reinstalled everything, made some adjustments, and re-installed Windows 7 Ultimate. Within Windows I installed the Nvidia Quadro 2000 driver (had Windows search for it) and rebooted. Voila - it worked without a problem.

Windows performance index is *5.8* (lowest value = storage). I will prepare a detailed how-to for other Linux Mint 13 Mate users (should work on other Linux Mint 13 versions as well as Ubuntu 12.04 as well).

The critical part is the hardware !!! Here my current list:

i7 3930K CPU with C2 stepping
Asus Sabertooth X79 with 1203 BIOS
PNY Quadro 2000 (Nvidia) graphics adapter with "multi-OS" support used with Windows
AMD 6450 graphics card used with Linux / dom0
KVM USB switch to switch keyboard and mouse between two USB host adapters
120 GB SSD drive for /, /boot, /home, and /win7, where /, /home, and /win7 reside on LVM2 formatted LVs
Several regular hard drives

I will prepare a detailled how-to and post it here or elsewhere with a link from here.

Thanks for all your support and encouragement!

EDIT: I was exaggerating with the Windows Experience Index - it was only 5.8 (not 6.8) using the HVM qemu-dm that comes as standard (fixed it in the text as well). But watch for my next post!


----------



## powerhouse

I didn't like the low storage performance index I got from Windows - only 5.8 for an SSD based storage (phy:/path/to/lvm/device...).

So today I installed the GPLPV driver from here: http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers

This is the result:



That was worth it







. Use the "signed" drivers for anything newer than Windows XP (e.g. Vista, Win 7 or Win 2008).


----------



## ghormoon

Hi,

I'm just trying to install this on my laptop and I've got a problem.
I don't know why, but when I start the VM, it "runs" for about 40s and poweroffs. Nothing is displayed anytime. Am I doing something wrong? Iso of the bootcd is set ...

another problem is that I had to comment out ethernet, I'm not able to set up the virtual bridge correctly, it dodn't set up automatically (I'm using Debian stable), but that would make sense to solve only when I'm able to boot at least something ...

If you have any ideas, let me know









Thanks in advance,
Ghor


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> Hi,
> I'm just trying to install this on my laptop and I've got a problem.
> I don't know why, but when I start the VM, it "runs" for about 40s and poweroffs. Nothing is displayed anytime. Am I doing something wrong? Iso of the bootcd is set ...
> another problem is that I had to comment out ethernet, I'm not able to set up the virtual bridge correctly, it dodn't set up automatically (I'm using Debian stable), but that would make sense to solve only when I'm able to boot at least something ...
> If you have any ideas, let me know
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks in advance,
> Ghor


You have to give a little more information. What laptop? Which processor (exactly)? Which graphic chip(s)?

But before you go any further, the following is a minimum requirement for VGA and/or PCI passthrough to work:

1. CPU with VT-d support !!! For example, a i7 2600 CPU will work, but NOT the i7 2600K. If you have an Intel CPU, you can check it here: http://ark.intel.com/. For example an Intel i5 2500T should work (if the rest of the points here are met). My kids have a Dell 15R (n5110) with a i5 2450M CPU which doesn't support VT-d







.

2. A motherboard with VT-d option in the BIOS. That will be even harder to find, especially on many lousy laptop BIOSes. It also depends on the BIOS release - often manufacturers add VT-d support only at a later stage in their BIOS. Last not least, some BIOSes with VT-d option are buggy. For a laptop user this will be a real challenge (if #1 above is met).

3. A graphics adapter that supports VGA passthrough. If the above two seem like obstacles, be prepared. This is the most tricky of all. But, there are also good news: The new on-chip Intel graphics adapters are supported as of Xen hypervisor 4.1.2 (or 4.2, I can't remember), as well as many AMD graphics adapters. If you have an Nvidia graphics adapter (as I have), there are three options/possibilities:

a) You got a "multi-OS" (Nvidia terminology) graphics adapter (unlikely in a laptop) and it's smooth rolling! The Quadro series of professional graphics workstation graphics adapters from Quadro 2000 on upwards (not the Quadro 600 which I tried) will work, as well as some other (usually high-end) "multi-os" adapters from Nvidia.

b) You got another Nvidia graphics chip or adapter where you can use the David Techer web site: http://www.davidgis.fr/blog/index.php?Xen and http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through, the latter with instructions on how to compile the patches into the Xen hypervisor.

c) If a) and b) don't work, you're probably out of luck. Some explanations for that, or help for choosing the right hardware, can be found here: http://wiki.xen.org/wiki/XenVGAPassthrough.

4. If you're looking for better Linux graphics acceleration on laptops with two graphics chips using Nvidia "Optima" technology, try bumblebee for Nvidia chips. But that doesn't give you any Windows games or other, only native Linux apps that you want to use with accelerated Nvidia graphics (instead of the Intel CPU built-in GPU that's usually no good for more demanding graphics tasks). I tried it on my kids laptop and it works great. I also love the technology behind bumblebee - this is the future: using VDI technology for remote desktop to the actual CPU/GPU on the computer (I wish there were host providers that did that, like Stephen Perlman's onlive http://www.onlive.com/).

I should meet with Steve, I have some nice ideas on how to develop his technology to make it THE (r)evolution!

Hope the above helps.


----------



## ghormoon

laptop HP Elitebook 8560p
CPU: Intel i7-2620M
GPU: ATI HD 6470M

1) should be ok
2) should be ok
3) I'm passing the discrete card, the integrated HD3000 in CPU is diasbled and not possible to allow in bios








4) I want Windows 7 with grapphics (because of games) and debian running at the same time (X11 SSH passthrough)

Btw, I'm using Xen 4.0, could it be the problem?


----------



## wTheOnew

Anyone noticing odd/bad performance with a bridged network adapter? If I pass through a NIC it performs fine, but if I use a bridged adapter I get some pretty wild fluctuations in speed and latency. Not a big deal, I wanted dedicated adapters for each VM anyway... Well, if I install more than two at a time I can't passthrough any of them.







It's always something.
Quote:


> Originally Posted by *powerhouse*
> 
> I didn't like the low storage performance index I got from Windows - only 5.8 for an SSD based storage (phy:/path/to/lvm/device...).
> So today I installed the GPLPV driver from here: http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers
> This is the result:
> 
> That was worth it
> 
> 
> 
> 
> 
> 
> 
> . Use the "signed" drivers for anything newer than Windows XP (e.g. Vista, Win 7 or Win 2008).


Nice. Good to see you got everything up and running.









Always seems easy going into it, followed by a couple weeks of pure suck, followed by an extreme feeling of accomplishment when the darn things finally works right.


----------



## Swiss

@powerhouse Sorry for the late reply. Have been (and still are) traveling. Atm I'm running these VMs without LVM (as .img) off an external USB3 mechanical. Will be trying LVM when I get back. Your solution with the GPLPV driver looks good!

@wTheOnew I saw terrible network issues running d3 over bridged network. Team Fortress 2 has been consistantly flawless however.

When I get back I will be attempting a dual seat setup. Does anyone have other benchmarks they'd like to see also??


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> laptop HP Elitebook 8560p
> CPU: Intel i7-2620M
> GPU: ATI HD 6470M
> 1) should be ok
> 2) should be ok
> 3) I'm passing the discrete card, the integrated HD3000 in CPU is diasbled and not possible to allow in bios
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 4) I want Windows 7 with grapphics (because of games) and debian running at the same time (X11 SSH passthrough)
> Btw, I'm using Xen 4.0, could it be the problem?


1. If you have VT-d enabled in the BIOS (in my BIOS it was disabled by default), it may work. Not all BIOSes are equal, though. Check for newer BIOS versions and what they additional features they have (or bug fixes). In my case there are newer BIOS versions available, but they address Win8 compatibility. They also do not allow a roll-back. So I don't even try.
2. CPU is with VT-d support, according to Intel.
3. ATI 6470M - I don't know but chances are it will work. It's a pity you can't use the HD3000, it would help.

4. The xen.org page has information about the newer hypervisor versions. In my case, 4.1.2 worked without patches.

The newer 4.2 hypervisor should bring further improvements in VGA passthrough. It may be worth downloading and compiling it. My 4.1.2 hypervisor still uses xm as toolstack, the 4.2 hypervisor will have xl which is a lot different under the surface. Most how-tos I've found are based on xm, and since xl is not complete in 4.1.2 I went with xm. However, it will be deprecated later. 4.2 should still support it, I think.

If the latest Debian doesn't offer a 4.1.2 hypervisor package then I would go for the 4.2 release and compile it (there aren't yet any packages of it, as far as I remember, but it should be pretty close to release).

How far did you get?

In your case, once you pass through the graphic card, you won't have a screen for Linux. You need to use VNC or remote desktop from another computer on the network to be able to do anything until Windows is installed and running.

I have shared my config files and more details on another blog site under replies (go down to the bottom): http://gro.solexiv.de/2012/08/pci-passthrough-howto/#comment-29. You may have a look at them and adjust them - bear in mind that I'm passing through my secondary graphics adapter, so you need to adapt the PCI IDs to your hardware (lspci can help).

I am using the xen-pciback module and haven't compiled xen-pciback into the kernel. Which means my Xen hypervisor and kernel will boot first using the screen for Linux and sometime in the boot process it will detach my secondary display adapter from Linux. In your case it would produce a blank screen sometime during the boot process, and you would need some access from another machine to log in and start Linux. So in your case i would not insert the pciback shell script in the inet.d process but manually activate the script when you are set up and connected to your laptop from another machine.

Another thing to watch out for is that pci passthrough is using vpci as default in Ubuntu based distributions and I believe in Debian too. So you should load the xen-pciback module as "xen-pciback passthrough=1" in your /etc/modules file. Please have a look at the link above for more details on the setup. Bear in mind that your configuration files may differ because of different hardware and Debian distribution - I use Linux Mint 13 Mate 64bit.

You will also need to check your dmesg output when you're in the dom0:

Here are some useful terminal commands to check your hardware and setup:

lspci # to check your PCI IDs
lspci -vv # to get more details
dmesg # to see the boot messages and debug output
dmesg | grep XEN # to narrow the output to the XEN related stuff
xm list # to list your current domains - only useful when running your domU
xm dmesg # your XEN messages

Check the http://wiki.xen.org/wiki/Xen_PCI_Passthrough and http://wiki.xen.org/wiki/XenVGAPassthrough pages for more information and instructions.

Hope it helps.


----------



## powerhouse

Quote:


> Originally Posted by *wTheOnew*
> 
> Anyone noticing odd/bad performance with a bridged network adapter? If I pass through a NIC it performs fine, but if I use a bridged adapter I get some pretty wild fluctuations in speed and latency. Not a big deal, I wanted dedicated adapters for each VM anyway... Well, if I install more than two at a time I can't passthrough any of them.
> 
> 
> 
> 
> 
> 
> 
> It's always something.
> Nice. Good to see you got everything up and running.
> 
> 
> 
> 
> 
> 
> 
> 
> Always seems easy going into it, followed by a couple weeks of pure suck, followed by an extreme feeling of accomplishment when the darn things finally works right.


I haven't played much with my Windows domU, so I don't know how it performs on local network drives. Internet speed is perfect - just like on Linux (I got a 10/1.5Mb Internet link). If i get to it later, I will try some local network file transfers to see how it fares.

I strongly recommend to install the GPLPV drivers in Windows. Here is the download link for "signed" drivers: http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers. Use the recommended installation. I had no luck installing all of them. There should be also a network driver. In my case it improved the Windows Experience Index for the hard drive substantially (from 5.8 to 7.8 of a maximum of 7.9). So maybe the network driver works similar.

Else I would check the /etc/network/interfaces file and edit if necessary. I removed the network manager package as it interfered with my manual setup in the interfaces file. Then edit or create /etc/resolv.conf as follows:

# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND - YOUR CHANGES WILL BE OVERWRITTEN
nameserver your.primary.DNS.server # e.g. 8.8.8.8
nameserver your.secondary.DNS.server

Hope it helps.

Thanks for the thumbs up. Yes, its as you described it.


----------



## powerhouse

Quote:


> Originally Posted by *Swiss*
> 
> @powerhouse Sorry for the late reply. Have been (and still are) traveling. Atm I'm running these VMs without LVM (as .img) off an external USB3 mechanical. Will be trying LVM when I get back. Your solution with the GPLPV driver looks good!
> @wTheOnew I saw terrible network issues running d3 over bridged network. Team Fortress 2 has been consistantly flawless however.
> When I get back I will be attempting a dual seat setup. Does anyone have other benchmarks they'd like to see also??


Yes, I'm really happy with the GPLPV driver. LVM seems to be a good option. I tried to get a 3.5TB logical volume with stripe working on the Windows domU, but the bloody NTFS wouldn't allow more than 2TB. Actually, it even complains at 2TB and I had to reduce it to somewhere 1.5TB. I wonder if VDH in Windows would allow me to user larger virtual drives. For now I guess I will stick with what works with NTFS - I have no experience with VDH. Just in case I installed the packages for VDH under Linux, to be able to use it if and when needed.


----------



## ghormoon

powerhouse:

I've been thinking about trying out 4.2 because of simplier pci passthrough (It took me about 5 hours to make working, I also had it only as a module) but quited it because i'm not sure what all i'd need to compile and such, i'm using stable debian and i'm a linux noob.

Now I'm in state that xen is able to start the VM (pci devices are linked to pciback) when I comment out network (I'll set it up when I'll know it works at least somehow)

Btw, I'm passing the VGA like PCI device, does it matter?

I'm logged to the xen from my desktop (SSH) so I see what's going on









When I start the VM, it runs for about 30-40s and turns off, nothing on the screen, so I don't know what's going on. I need to install the OS somehow







(I've got mounted iso, I've been trying using CD directly and then changed to iso, but obviously that was not the problem







)

Thanks,
Ghor


----------



## wTheOnew

Quote:


> Originally Posted by *ghormoon*
> 
> powerhouse:
> I've been thinking about trying out 4.2 because of simplier pci passthrough (It took me about 5 hours to make working, I also had it only as a module) but quited it because i'm not sure what all i'd need to compile and such, i'm using stable debian and i'm a linux noob.
> Now I'm in state that xen is able to start the VM (pci devices are linked to pciback) when I comment out network (I'll set it up when I'll know it works at least somehow)
> Btw, I'm passing the VGA like PCI device, does it matter?
> I'm logged to the xen from my desktop (SSH) so I see what's going on
> 
> 
> 
> 
> 
> 
> 
> 
> When I start the VM, it runs for about 30-40s and turns off, nothing on the screen, so I don't know what's going on. I need to install the OS somehow
> 
> 
> 
> 
> 
> 
> 
> (I've got mounted iso, I've been trying using CD directly and then changed to iso, but obviously that was not the problem
> 
> 
> 
> 
> 
> 
> 
> )
> Thanks,
> Ghor


If you have something like the following in your configuration you should be able to VNC into the VM at 192.168.1.x:5910 to setup the OS:

Code:



Code:


vnc=1
vncdisplay="10"
vnclisten="192.168.1.x"
vncpasswd=''


----------



## powerhouse

@wTheOnew: I checked the network speed with bridged network, here the results on my local network:

1. File transfer over 100Mb wired connection (via Gigabit router):
a) Windows domU: ~2.5MB/s
b) Linux dom0: ~2.3MB/s

2. File transfer over 1Gb wire link (via router):
a) Windows domU: Couldn't test it yet.
b) Linux dom0: 4.7MB/s

This is no good !!! Fast Ethernet should deliver somewhere around 12MB/s (megabyte), Gigabit Ethernet around 5-10x as much, depending on how good the NICs are on both end and on the router in between.

I haven't got the time now to compare that with my Macbook transfer speed, but there is certainly a lot of room for improvement here.

Have you checked the wiki.xen.org site? IIRC, there is a network problems wiki. Let me know if you find a solution (other than a second NIC). I won't be able to deal with this until Thursday.


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> powerhouse:
> I've been thinking about trying out 4.2 because of simplier pci passthrough (It took me about 5 hours to make working, I also had it only as a module) but quited it because i'm not sure what all i'd need to compile and such, i'm using stable debian and i'm a linux noob.
> Now I'm in state that xen is able to start the VM (pci devices are linked to pciback) when I comment out network (I'll set it up when I'll know it works at least somehow)
> Btw, I'm passing the VGA like PCI device, does it matter?
> I'm logged to the xen from my desktop (SSH) so I see what's going on
> 
> 
> 
> 
> 
> 
> 
> 
> When I start the VM, it runs for about 30-40s and turns off, nothing on the screen, so I don't know what's going on. I need to install the OS somehow
> 
> 
> 
> 
> 
> 
> 
> (I've got mounted iso, I've been trying using CD directly and then changed to iso, but obviously that was not the problem
> 
> 
> 
> 
> 
> 
> 
> )
> Thanks,
> Ghor


I wish I could say that I got something in only 5 hours - it took me nearly two months to get everything figured out (well, almost, network speed on local LAN sucks at the moment). So you are in good shape now.

Try VNCviewer or similar to connect to your laptop from another PC on your network. Make sure you enable VNCviewer access before you run the Windows guest.


----------



## ghormoon

I'm using SSH and X11 passthough when I'm connecting to the dom0 from another machine ... or you mean I should log on the guest system from VNC? Is there any chance that it will display more than the notebook display?








I'll try it out in the morning


----------



## wTheOnew

^ Yeah, connect to the guest with VNC.
Quote:


> Originally Posted by *powerhouse*
> 
> @wTheOnew: I checked the network speed with bridged network, here the results on my local network:
> 1. File transfer over 100Mb wired connection (via Gigabit router):
> a) Windows domU: ~2.5MB/s
> b) Linux dom0: ~2.3MB/s
> 2. File transfer over 1Gb wire link (via router):
> a) Windows domU: Couldn't test it yet.
> b) Linux dom0: 4.7MB/s
> This is no good !!! Fast Ethernet should deliver somewhere around 12MB/s (megabyte), Gigabit Ethernet around 5-10x as much, depending on how good the NICs are on both end and on the router in between.
> I haven't got the time now to compare that with my Macbook transfer speed, but there is certainly a lot of room for improvement here.
> Have you checked the wiki.xen.org site? IIRC, there is a network problems wiki. Let me know if you find a solution (other than a second NIC). I won't be able to deal with this until Thursday.


The signed drivers definitely helped the speed over the bridged adapter. It used to wildly fluctuate between 1MB/s and 20MB/s, now it's a solid 50MB/s when transferring files. It still adds a bit of latency.

Current file transfer averages and pingtest averages:
Asus GigE NIC: 70MB/s, 9ms
Netgear GigE NIC: 69MB/s, 9ms
Bridged (Realtek) GigE NIC: 50MB/s, 21ms


----------



## ghormoon

damn, now it won't even start and I'd swear I didnt' change anything except the vnc ...

in cat /var/log/xen/qemu-dm-win7.log i see this:
seems like some sound problem but iv don't know how to tell it to not use sound (like i did with ethernet) ...

oss: Could not initialize DAC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
oss: Could not initialize DAC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
audio: Failed to create voice `adlib'
oss: Could not initialize DAC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
oss: Could not initialize DAC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
audio: Failed to create voice `gus'
oss: Could not initialize ADC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
oss: Could not initialize ADC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
audio: Failed to create voice `ac97.pi'
oss: Could not initialize DAC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
oss: Could not initialize DAC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
audio: Failed to create voice `ac97.po'
oss: Could not initialize ADC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
oss: Could not initialize ADC
oss: Failed to open `/dev/dsp'
oss: Reason: No such file or directory
audio: Failed to create voice `ac97.mc'
xs_read(/local/domain/0/device-model/2/xen_extended_power_mgmt): read error
bind() failed

edit:
when I comment out the vnc it starts for a while like before ...


----------



## wTheOnew

Quote:


> Originally Posted by *ghormoon*
> 
> damn, now it won't even start and I'd swear I didnt' change anything except the vnc ...
> in cat /var/log/xen/qemu-dm-win7.log i see this:
> 
> bind() failed
> edit:
> when I comment out the vnc it starts for a while like before ...


could you copy/paste your config file? Doesn't matter if the vnc part is still commented out or not.


----------



## ghormoon

kernel = "hvmloader"
builder='hvm'
memory = 3072
name = "win7"
vcpus=3
pae=1
acpi=1
apic=1
#vif = [ 'type=ioemu, bridge=xenbr0' ]
disk = [ 'phy:/dev/sda2,sda,w' , 'file:/win7recovery.iso,sdc:cdrom,r' ]
device_model = 'qemu-dm'
boot="dc"
sdl=0
opengl=1
vnc=1
vncdisplay="10"
vnclisten="192.168.1.42"
vncpasswd='42'
stdvga=0
#nographic=1 #!!! only uncomment this if you are using win8 or are trying to get a nvidia card to work
serial='pty'
tsc_mode=0
soundhw='all'
usb=1
usbdevice='mouse'
gfx_passthru=0
pci=[ '01:00.0', '01:00.1' , '00:1a.0' , '00:1b.0' , '00:1d.0' , '24:00.0' , '24:00.1' , '24:00.2' , '25:00.0' , '26:00.0' ]

for reference lspci:

00:00.0 Host bridge: Intel Corporation Sandy Bridge DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Sandy Bridge PCI Express Root Port (rev 09)
00:16.0 Communication controller: Intel Corporation Cougar Point HECI Controller #1 (rev 04)
00:16.3 Serial controller: Intel Corporation Cougar Point KT Controller (rev 04)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 04)
00:1a.0 USB Controller: Intel Corporation Cougar Point USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation Cougar Point High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 1 (rev b4)
00:1c.1 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 2 (rev b4)
00:1c.2 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 3 (rev b4)
00:1c.3 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 4 (rev b4)
00:1c.7 PCI bridge: Intel Corporation Cougar Point PCI Express Root Port 8 (rev b4)
00:1d.0 USB Controller: Intel Corporation Cougar Point USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Cougar Point LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation Cougar Point 6 port SATA AHCI Controller (rev 04)
01:00.0 VGA compatible controller: ATI Technologies Inc NI Seymour [AMD Radeon HD 6470M]
01:00.1 Audio device: ATI Technologies Inc Device aa98
24:00.0 FireWire (IEEE 1394): JMicron Technology Corp. IEEE 1394 Host Controller (rev 30)
24:00.1 System peripheral: JMicron Technology Corp. SD/MMC Host Controller (rev 30)
24:00.2 SD Host controller: JMicron Technology Corp. Standard SD Host Controller (rev 30)
25:00.0 Network controller: Intel Corporation 6000 Series Gen2 (rev 34)
26:00.0 USB Controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)


----------



## powerhouse

I think ssh -X11 ... should work from a remote computer, else try VNC. If you do the VGA passthrough during boot your laptop screen will go blank. So you need to connect to your laptop remotely to log in to your Linux dom0 ssh -X11 or probably even without X11 should allow you to do that. Search for headless Linux login or similar to find a way to boot into the Linux dom0 and from there try to run your domU.

I've been using both virt-manager (GUI) and xm create ... to start the domU. Maybe check the instructions of the original poster as he describes VGA passthrough using the primary display adapter like you will be doing. Make sure your router doesn't block port 5900 on the LAN, or any other port. Usually it shouldn't.

Using virt-manager with ssh -X11 gives you a GUI and VNC display on the remote console in the X11 window. You need to install the virt-manager prior to this. I'm on my iPhone now and can't check this.

In general there are two ways to get this working. One way is to detach the primary display adapter during the boot process - there are different stages at which this can be triggered. In any case it should be before the xend damon takes control (using xm, xl doesn't use xend).
Again, check the OP instructions and/or the link I posted in a previous post.

As for VGA versus PCI passthrough the instructions are the same if you use an AMD GPU or a multi-os Nvidia card. BUT there quite some differences when using Nvidia GPUs unless you got a multi-os card. With AMD cards - and if my memory doesn't fail me - you should be able to run virt-manager and get a VNC connection to your domU.

So in essence you should be able to see your Windows installer loading (within the X11 window when using ssh -X11 and virt-manager. I've been using this method on a
MacBook and it works - the X11 screen gives me the virt-manager window with the VNC display inside it (the VNC is used on the laptop to see the Windows domU screen, while you still have access to Linux via command line: use the "&" option at the command line to make virt-manager run in the background i.e. virt-manager &, then switch to the X11 window).

As mentioned before, post your /etc/xen/win.cfg file or whatever name you gave to your config file.


----------



## ghormoon

I've alrady set up the x11 passthrough, it works, the problem is that when I start the VM (both commandline or virt-manager) the machine turns off in a while and I don't see anything on the screen, so I can't start win installation ... cfg fiel is in my previous post.


----------



## powerhouse

I forgot to me to mention the second way, or actually another way:

1) detach your VGA right at the start of the boot process using a grub command in the / etc/default/grub file. In Ubuntu or Linux Mint you then need to run update-grub, probably the same in Debian. But you need xen-pciback compiled into the kernel.

2) load the xen-pciback module in /etc/modules. Run the pciback shell script (see link in a previous post) to detach the graphics adapter before xend.

3) same as 2) but run the pciback script whenever you feel like within Linux.

In any of the options above you need to run e.g. xm create ... to start the Windows domU, or xm new and virt-manager to start the domU. From here on the config file for your domU takes control. For the installation a remote access is required.

As mentioned before post your domU config file.


----------



## ghormoon

I'm deattaching the VGA in init.d script, because that i have kernel that has xen-pciback as module and i've never compiled kernel, so it would end in a disaster









I've posted the config in this post: http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/130#post_18141770 ... or do you mean different config?
Btw that version is that one, which doesn't start (that audio problem, most propably because of that vnc wants to pass audio somehow), without the vnc settings it does and shuts down in a while. Isn't there any way to use notebook display for installing?


----------



## vesslan

When I used Xen 4.1.2 kernel and fedora16 do vga pass through,ATI graphics can be successful,But NVIDIA graphics card unsuccessful.I used ATI graphics card includes HD5850&HD6850,used NVIDIA graphics card includes 9400GT 9800GT GTX550TI GTX470 GTX580.
And I tested fedora17 and xen 4.1.2,ATI graphics can be successful,NVIDIA can not.
I see xen 4.2 have better support of NVIDIA graphics card.But, now if use command in Linux OS installed xen just connected 4.1.2 version.
I saw EnMing Zhang about NVIDIA graphics passthrough documents, and according to the steps tested,but unsuccessful.
Are there someone have successful NVIDIA graphics card vga passthrough on fedora16 or 17 and xen 4.1.2 ? (VM OS is win7 64bit).If there is ,can you tell me how to operate,I want to know steps.


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> I'm deattaching the VGA in init.d script, because that i have kernel that has xen-pciback as module and i've never compiled kernel, so it would end in a disaster
> 
> 
> 
> 
> 
> 
> 
> 
> I've posted the config in this post: http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/130#post_18141770 ... or do you mean different config?
> Btw that version is that one, which doesn't start (that audio problem, most propably because of that vnc wants to pass audio somehow), without the vnc settings it does and shuts down in a while. Isn't there any way to use notebook display for installing?


Sorry I missed your post (config file).

Here are some comments:

disk = [ 'phy:/dev/sda2,sda,w' , 'file:/win7recovery.iso,sdc:cdrom,r' ]

I couldn't get Windows installer to start with "sda" setting. I think it complained about missing driver disk or so. Try to change your above line to read:

disk = [ 'phy:/dev/sda2,hda,w' , 'file:/win7recovery.iso,hdc:cdrom,r' ]

I think you have far too many PCIs passed through. Start with the basic ones you need:

Instead of: pci=[ '01:00.0', '01:00.1' , '00:1a.0' , '00:1b.0' , '00:1d.0' , '24:00.0' , '24:00.1' , '24:00.2' , '25:00.0' , '26:00.0' ]

Try: pci=[ '01:00.0', '01:00.1' , '00:1d.0' ] # replace 00:1d.0 with 00:1a.0 if it doesn't work

01:00.0 is for the graphics adapter, 01.00.1 is for the audio on that graphics adapter when using an HDMI link. 00.1a.0 is your Intel CPU integrated USB host #2 (the CPU has two, the other one is 00.1d.0). That's where your keyboard / mouse should be connected to. Check the dmesg output to find your mouse and keyboard - note the PCI ID(s).

You first want to try to install the Windows domU, with or without sound support. Once can run Windows in domU, shut it down and add additional PCIs you want to pass through, for sound, USB etc. devices.

Don't try to pass through your SATA controller 24:00.x !!! You may be trying to take away your hard drive from dom0, which is not a good idea. There is no need to pass through the SATA controller to Windows.

The same goes for your network controller 25:00.0. If you take it away from the Linux dom0, you have no way to access your machine during the Windows installation.

Once you changed your Win7 cfg file and perhaps rebooted, check the output of the following command when using your ssh remote session:

xm pci-list-assignable-devices

It should list the 3 PCI IDs you specified, if they have been successfully detached !!! If there is nothing displayed, check your pciback shell script or config file to make sure it contains only the three PCI IDs you listed in the win7.cfg file, but using the domain ID in front of the PCI ID like here:

0000:01:00.0

(See your pciback script for exact syntax!)

So, when you don't get any PCIs listed with the xm pci-list... command, you need to run the pciback script to detach the PCIs. If the script doesn't give you an error, check again with xm pci-list-assignable-devices. Do they show up now?

Important: The above can only be done from a remote PC via ssh or VNC, as it will disconnect your keyboard/mouse and screen from the Linux dom0 so you won't see anything nor be able to use your mouse/keyboard.

Let's start with the above suggestions and go from there.


----------



## powerhouse

Quote:


> Originally Posted by *vesslan*
> 
> When I used Xen 4.1.2 kernel and fedora16 do vga pass through,ATI graphics can be successful,But NVIDIA graphics card unsuccessful.I used ATI graphics card includes HD5850&HD6850,used NVIDIA graphics card includes 9400GT 9800GT GTX550TI GTX470 GTX580.
> And I tested fedora17 and xen 4.1.2,ATI graphics can be successful,NVIDIA can not.
> I see xen 4.2 have better support of NVIDIA graphics card.But, now if use command in Linux OS installed xen just connected 4.1.2 version.
> I saw EnMing Zhang about NVIDIA graphics passthrough documents, and according to the steps tested,but unsuccessful.
> Are there someone have successful NVIDIA graphics card vga passthrough on fedora16 or 17 and xen 4.1.2 ? (VM OS is win7 64bit).If there is ,can you tell me how to operate,I want to know steps.


I don't use Fedora, but Linux Mint 13 (Ubuntu / Debian based distro) with Xen hypervisor 4.1.2 and a Nvidia Quadro 2000 graphics card as secondary graphics adapter for my Win7 64 Pro domU.

Nvidia officially supports what they call "multi-OS" on some of their workstation graphics cards. With a multi-OS specified card it should be smooth sailing. They are designed for exactly that application, that is VGA passthrough. These cards aren't cheap, though.

Many/most of the Nvidia Quadro cards offer multi-os. But not all of them - I had a Quadro 600 card that didn't work and is NOT specified to support multi-OS. Check the Nvidia website. I think some Tesla or so cards also support multi-OS. I don't know how these cards fare with games, as my Quadro 2000 gives rather mediocre results for graphics under Windows (7.0 experience index, the lowest of all). They are designed for professional graphics workstations running CAD-CAM, video or photo editing software, not for games. Unless Nvidia expands the multi-OS support to "gamer" graphics cards, this business will probably go to AMD.

There are some other Nvidia cards that have been reported to support VGA passthrough using patches. But as you mentioned, it may or may not work. My Quadro 2000 gave me instant success with no patches or hypervisor or kernel compilation at all - just straight out of the box.

The only thing I had to fix was to blacklist the nouveau Nvidia driver that Linux automatically loaded.

Ah, by the way, I use a cheap AMD graphics card for my Linux dom0 using the proprietary AMD driver. Unfortunately the Nvidia proprietary driver doesn't work under Xen (at least not out-of-the-box), only nouveau. And the nouveau driver isn't that great of a performer, though it usually works well with basic applications and without 3D etc. gimmicks.


----------



## vesslan

Quote:


> Originally Posted by *powerhouse*
> 
> I don't use Fedora, but Linux Mint 13 (Ubuntu / Debian based distro) with Xen hypervisor 4.1.2 and a Nvidia Quadro 2000 graphics card as secondary graphics adapter for my Win7 64 Pro domU.
> Nvidia officially supports what they call "multi-OS" on some of their workstation graphics cards. With a multi-OS specified card it should be smooth sailing. They are designed for exactly that application, that is VGA passthrough. These cards aren't cheap, though.
> Many/most of the Nvidia Quadro cards offer multi-os. But not all of them - I had a Quadro 600 card that didn't work and is NOT specified to support multi-OS. Check the Nvidia website. I think some Tesla or so cards also support multi-OS. I don't know how these cards fare with games, as my Quadro 2000 gives rather mediocre results for graphics under Windows (7.0 experience index, the lowest of all). They are designed for professional graphics workstations running CAD-CAM, video or photo editing software, not for games. Unless Nvidia expands the multi-OS support to "gamer" graphics cards, this business will probably go to AMD.
> There are some other Nvidia cards that have been reported to support VGA passthrough using patches. But as you mentioned, it may or may not work. My Quadro 2000 gave me instant success with no patches or hypervisor or kernel compilation at all - just straight out of the box.
> The only thing I had to fix was to blacklist the nouveau Nvidia driver that Linux automatically loaded.
> Ah, by the way, I use a cheap AMD graphics card for my Linux dom0 using the proprietary AMD driver. Unfortunately the Nvidia proprietary driver doesn't work under Xen (at least not out-of-the-box), only nouveau. And the nouveau driver isn't that great of a performer, though it usually works well with basic applications and without 3D etc. gimmicks.


Yes, I know Tesla and Quadro maybe support pass-thou,but they need to run Xenserver (It's virtual machine system of Citrix ).
On its website wrote only Platinum Edition and Enterprise Edition supports vga pass-through,and supports Tesla&Quadro. But they are not free.
I downloaded Advanced Edition,version is 6.0,it's Xen hypervisor is 4.1.2. So,I think Xenserver maybe modified Linux kernel to support NVIDIA graphics cards.
In before I mentioned documents,EnMing Zhang also modified memory in config files and kernel files,he tested successful NVIDIA 9400GT,and he says the earlier version compatibility. I think accordance he's documents maybe 8800GT and 8600GT maybe success.
But so far, I only know that he doesn't use XenServer success.
You said I also installed patches and driving test, and the Linux built-in drivers unload, but without success。 Did you succeed whit Quadro 2000?


----------



## powerhouse

Quote:


> Originally Posted by *vesslan*
> 
> Yes, I know Tesla and Quadro maybe support pass-thou,but they need to run Xenserver (It's virtual machine system of Citrix ).
> On its website wrote only Platinum Edition and Enterprise Edition supports vga pass-through,and supports Tesla&Quadro. But they are not free.
> I downloaded Advanced Edition,version is 6.0,it's Xen hypervisor is 4.1.2. So,I think Xenserver maybe modified Linux kernel to support NVIDIA graphics cards.
> In before I mentioned documents,EnMing Zhang also modified memory in config files and kernel files,he tested successful NVIDIA 9400GT,and he says the earlier version compatibility. I think accordance he's documents maybe 8800GT and 8600GT maybe success.
> But so far, I only know that he doesn't use XenServer success.
> You said I also installed patches and driving test, and the Linux built-in drivers unload, but without success。 Did you succeed whit Quadro 2000?


My system is up and running Xen hypervisor 4.1.2 with Linux Mint 13 Mate 64bit dom0 and Windows 7 64bit Pro as domU.

The Quadro 2000 works perfect! I did not use any patches or kernel builds. Aside from some minor fixes in Linux Mint (missing links) for some xen packages there was nothing out of the ordinary that I had to do.

I never tried the commercial Xenserver and don't see a need for it now.

By the way, I load xen-pciback as module with the passthrough=1 option (edit /etc/modules file) and run a pciback script to detach/attach the PCIs.

In Windows 7 I installed the GPLPV disk driver - works wonders.

In summary, I can confirm that the Nvidia Quadro 2000 works well with VGA passthrough.


----------



## cladus

Hello,
Is it possible to dedicate one of my GPU's to a VM and for the rest of my VM's to share a Intel HD Graphics?


----------



## powerhouse

Quote:


> Originally Posted by *cladus*
> 
> Hello,
> Is it possible to dedicate one of my GPU's to a VM and for the rest of my VM's to share a Intel HD Graphics?


Sure you can. But your other guests will get a virtualized HD graphics adapter, that means not native performance.

You would use the Intel HD GPU without passthrough for your dom0 and other guest VMs (domUs). Then you passthrough your secondary, dedicated graphics adapter to the domU you want, for example a Windows 7 domU. That Windows domU would have exclusive and direct/native access to the secondary graphics adapter.


----------



## Killhouse

I do cross-platform development for very "heavy" tasks - they require maximum memory and maximum CPU resources.

I can take the hit from the extra overhead of dom0/xen, but I have a question:
Can I pass through ALL cores and ALL memory, and let Xen handle the resources efficiently? Or is that just not possible?

Also, I only have one GPU. I understand the complications, but the end result should be that the active OS has control of the GPU and hence the display - in such a manner that I can flick from VM to VM to dom0 and back in one keypress? Correct?

EDIT: I assume the same applies to passing through the USB controller so that my KB/mouse just switches over too.

Very nice guide, I look forward to trying it.

I am just about to set up my PC with an LVM partition on the SSD; and swap, a backup OS and data storage on RAID1 with my HDDs.


----------



## cladus

Thank you for that powerhouse,
would any one mind telling me what hardware I'm going to need, I will have 2 VM's with 2 different dedicated graphics and 3VM's running normally.
I have found the i7-3770 Processor and Intel® Desktop Board DZ77GA-70K both with vt-d and vt-x support but are they powerful enough for what I'm planning, And how much ram should I get I was thinking 32GB but I don't know if its too much or too less.
Price Range: $1000-$3000
Thank You


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> I do cross-platform development for very "heavy" tasks - they require maximum memory and maximum CPU resources.
> I can take the hit from the extra overhead of dom0/xen, but I have a question:
> Can I pass through ALL cores and ALL memory, and let Xen handle the resources efficiently? Or is that just not possible?
> Also, I only have one GPU. I understand the complications, but the end result should be that the active OS has control of the GPU and hence the display - in such a manner that I can flick from VM to VM to dom0 and back in one keypress? Correct?
> EDIT: I assume the same applies to passing through the USB controller so that my KB/mouse just switches over too.
> Very nice guide, I look forward to trying it.
> I am just about to set up my PC with an LVM partition on the SSD; and swap, a backup OS and data storage on RAID1 with my HDDs.


1. You need to leave some resources for Xen. You can't use the entire memory for your domU, nor all CPU resources. As to CPU resources, the granularity is measured in virtual CPUs. In my case with a 6-core i7 3930K CPU, I have 12 threads and ergo 12 VCPUs. I dedicated 2 VCPUs to the dom0/Linux, the rest for the Windows domU.

Xen offers lots of ways to tweak performance. And then there are the PV (para-virtualization) drivers which gave me a great performance boost for disk access under Windows 7 (look for "signed" GPLPV drivers). As for memory, I have 32GB in total and my dom0 with full-blown Linux Mint desktop works happily with 2GB, even less.

If you don't care for a graphical dom0 Linux desktop, you can also install the Xen hypervisor with a minimal dom0 Linux - there should be a readymade ISO available on the Xen website.

2. In my opinion, one GPU is a pain in the neck (but that's very subjective). You can't just switch from one domU to the other (while the first is still running), without causing havoc when switching the GPU to another system. Think about, it would be like pulling your graphics card out of a running Windows system (or Linux). I don't think that this works nicely.

If you don't have special graphics requirements for multiple guests and/or the dom0, get a cheap AMD graphics adapter (I bought a 6450, but thats really bottom line even with proprietary drivers), plug it into the first PCIe x16 slot and put your main GPU into the second slot. This way you have a dedicated graphics card for your dom0, and a dedicated one for your main guest system (e.g. Windows). Many modern boards allow you to install a third adapter, if needed.

3. USB passthrough is way easier. For one thing, USB was designed to be hot-pluggable. Combined with second GPU, you can get a USB KVM and switch your mouse / keyboard between two different USB hosts (one passed thru to the guest) and that works fine. Your screen should have two connection ports so you can run two cables, one from each GPU. Your screen should also allow you to switch between the inputs, or automatically switch over when a port becomes active.

For external USB disks use a dedicated, passed-through USB port (host actually) for your domU guest.

4. If you prefer a Ubuntu-based (or perhaps Debian) system, have a look at my how-to: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It's a little more complicated than this how-to, though.

5. LVM is the way to go for Xen. Your setup is similar to mine, though I use LVM striped on my HDDs to get RAID0 performance and use LVM snapshots and backup (for example rsync) for backup purposes. I find LVM very flexible and I often find myself in need for more disk space. As mentioned before, use the GPLPV drivers for Windows - it boosted my SDD based Windows Experience index for disk performance from 5.8 to 7.8 (7.9 being the maximum). I can't complain.

Important: Make sure you have the right hardware - VTd in both CPU (or motherboard) and in the motherboard BIOS settings is a must. Don't even think about passing through a graphics adapter that's not been confirmed to work. AMD GPUs should be the saver road (again check your specific model), though my Nvidia Quadro 2000 "multi-OS" GPU works out of the box with Xen hypervisor 4.1.2.

Good luck!


----------



## powerhouse

Quote:


> Originally Posted by *cladus*
> 
> Thank you for that powerhouse,
> would any one mind telling me what hardware I'm going to need, I will have 2 VM's with 2 different dedicated graphics and 3VM's running normally.
> I have found the i7-3770 Processor and Intel® Desktop Board DZ77GA-70K both with vt-d and vt-x support but are they powerful enough for what I'm planning, And how much ram should I get I was thinking 32GB but I don't know if its too much or too less.
> Price Range: $1000-$3000
> Thank You


According to http://ark.intel.com/products/65719/Intel-Core-i7-3770-Processor-8M-Cache-up-to-3_90-GHz the Intel i7-3770 has VT-d. Here are some observations:

a) 4 cores / 8 threads - this translates into 8 virtual CPUs or vcpus in Xen. If I understand you correctly, you are planning to run 5 VMs (domU in Xen talk) plus the obligatory dom0. That doesn't leave many options. You will be able to dedicate 2 vcpus per guest for two guests, and one vcpu for the remaining domUs and dom0. It may be enough, but that depends on the applications.

Some questions:
Can you explain what you try to build?
Will all VMs run concurrently, or do you plan to run them when needed and turn them off when not in use?

b) Memory is limited to 32GB. For normal desktop tasks and even combined with games this should give you plenty of memory. dom0 (control domain = Linux) will need some 2GB, even with a full desktop system like Fedora or Ubuntu (unless you do some memory intensive tasks on it). A regular Windows 7 desktop with office applications, browser and email should need more than 4GB. A gaming desktop may need a little more, but 8GB should be enough. Only if you plan to do video or photo editing with large RAW files (uncompressed high-resolution images), CAD-CAM, software compilations, or similar memory intensive stuff will you be able to use more than 8GB of memory. But those last applications will also require a lot of CPU power to run efficient, in which case your CPU may be underpowered.

c) Memory bandwidth is "only" 25GB/s.

d) The 3770 has an internal HD 4000 graphics adapter, which can be used for the dom0. On the downside, it supports 1x PCIe x16, or 2x PCIe x8, which means if you plug in two discrete graphics adapters they will run at x8 speed instead of x16. I have no idea how that impacts graphics performance, but I guess it will.

In comparison, the i7-3930K I use has 6 cores / 12 threads, can address a total of 64GB memory, has a bigger internal cache (12MB versus 8MB), and double the memory bandwidth (50GB/s). A C2 stepping version of it supports VT-d. It supports two PCIe x16 cards. BUT, it doesn't have an internal GPU.

Depending on what you want to do, you may want to consider the i7 3930K.

Intel boards are generally considered to have good BIOS versions and support. Still, check in the forums or on the Internet to see if it's been verified to work with PCI/VGA passthrough.

32GB should be enough memory, but the best is to list the applications or VMs you want to run and add up their memory requirement. See above. One remark on memory: memory kits are matched kits. So if you buy a 16GB kit and want to add another 16GB later, it may not work even with the same exact memory kit. My 3930K and Asus X79 motherboard combo were quite tricky about memory. And it gets more tricky the more memory you have.

Last not least the graphics cards: If you want Nvidia, as of now, only the Nvidia "multi-OS" graphics cards are designed to work with VGA passthrough. There are hypervisor patches etc. for other (perhaps many) Nvidia cards, but that isn't for everyone (I tried and failed with a Quadro 600 card). AMD, on the other hand, has reportedly better VGA graphics passthrough support, but make sure to get cards that have been verified by other users.

Hope this helps.


----------



## Killhouse

Quote:


> Originally Posted by *powerhouse*
> 
> 1. You need to leave some resources for Xen. You can't use the entire memory for your domU, nor all CPU resources. As to CPU resources, the granularity is measured in virtual CPUs. In my case with a 6-core i7 3930K CPU, I have 12 threads and ergo 12 VCPUs. I dedicated 2 VCPUs to the dom0/Linux, the rest for the Windows domU.
> Xen offers lots of ways to tweak performance. And then there are the PV (para-virtualization) drivers which gave me a great performance boost for disk access under Windows 7 (look for "signed" GPLPV drivers). As for memory, I have 32GB in total and my dom0 with full-blown Linux Mint desktop works happily with 2GB, even less.
> If you don't care for a graphical dom0 Linux desktop, you can also install the Xen hypervisor with a minimal dom0 Linux - there should be a readymade ISO available on the Xen website.
> 2. In my opinion, one GPU is a pain in the neck (but that's very subjective). You can't just switch from one domU to the other (while the first is still running), without causing havoc when switching the GPU to another system. Think about, it would be like pulling your graphics card out of a running Windows system (or Linux). I don't think that this works nicely.
> If you don't have special graphics requirements for multiple guests and/or the dom0, get a cheap AMD graphics adapter (I bought a 6450, but thats really bottom line even with proprietary drivers), plug it into the first PCIe x16 slot and put your main GPU into the second slot. This way you have a dedicated graphics card for your dom0, and a dedicated one for your main guest system (e.g. Windows). Many modern boards allow you to install a third adapter, if needed.
> 3. USB passthrough is way easier. For one thing, USB was designed to be hot-pluggable. Combined with second GPU, you can get a USB KVM and switch your mouse / keyboard between two different USB hosts (one passed thru to the guest) and that works fine. Your screen should have two connection ports so you can run two cables, one from each GPU. Your screen should also allow you to switch between the inputs, or automatically switch over when a port becomes active.
> For external USB disks use a dedicated, passed-through USB port (host actually) for your domU guest.
> 4. If you prefer a Ubuntu-based (or perhaps Debian) system, have a look at my how-to: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It's a little more complicated than this how-to, though.
> 5. LVM is the way to go for Xen. Your setup is similar to mine, though I use LVM striped on my HDDs to get RAID0 performance and use LVM snapshots and backup (for example rsync) for backup purposes. I find LVM very flexible and I often find myself in need for more disk space. As mentioned before, use the GPLPV drivers for Windows - it boosted my SDD based Windows Experience index for disk performance from 5.8 to 7.8 (7.9 being the maximum). I can't complain.
> Important: Make sure you have the right hardware - VTd in both CPU (or motherboard) and in the motherboard BIOS settings is a must. Don't even think about passing through a graphics adapter that's not been confirmed to work. AMD GPUs should be the saver road (again check your specific model), though my Nvidia Quadro 2000 "multi-OS" GPU works out of the box with Xen hypervisor 4.1.2.
> Good luck!


Huge help, thank you very much! + rep!

It's a shame about the CPU thing. I will have to change how much resources I want to ship around to different VMs depending on which VM I'm testing on... that's a bit of a pain but oh well.

GPU: I'll give it a go and see what happens, I think the 4870 is compatible - BIOS is checked for IOMMU and virtualization and the Phenom should take it.

Had to abandon the RAID, I couldn't get the drivers installed into Linux for a HW RAID, and cant install backup Windows on a software OS (belonging to Linux) so I'm doing it RAID-less.

Thanks again, will update with any developments.


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> Huge help, thank you very much! + rep!
> It's a shame about the CPU thing. I will have to change how much resources I want to ship around to different VMs depending on which VM I'm testing on... that's a bit of a pain but oh well.
> GPU: I'll give it a go and see what happens, I think the 4870 is compatible - BIOS is checked for IOMMU and virtualization and the Phenom should take it.
> Had to abandon the RAID, I couldn't get the drivers installed into Linux for a HW RAID, and cant install backup Windows on a software OS (belonging to Linux) so I'm doing it RAID-less.
> Thanks again, will update with any developments.


1. When you shut down a domU the CPU resources are freed. So you could configure multiple domUs with "oversubscription" of CPU time, as long as they don't run concurrently. You could even have two different domU.cfg files with different CPU and RAM parameters for the same domU, though Windows might complain about licenses at it sees different hardware when running with the different configs.

But yes, dynamic assignment of resources would be better.

2. Hm, are you running Xen? I suggest you try LVM. There is a command for lvcreate that essentially gives you RAID1. LVM has also performance advantages with Xen. Don't forget to install the "signed" GPLPV driver under Linux that you can find here: http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers.

I haven't played with different "disk" formats under Xen, just the plain phy:/path/to/lvm/volume. In this case, when using LVM, you just create a logical volume and don't format it (no mkfs). When installing Windows it will create an NTFS formatted virtual storage "container" (that's probably not the right word, but I hope you understand) which is not visible to Linux. Under Linux you use the kpartx command to find and register the domU virtual partitions on the LV, then you can mount and access them under Linux/LVM. It will see NTFS formatted drives. For Windows the whole LVM stuff is transparent and it only sees the NTFS drives. Mounting these virtual drives under Linux while running the domU should be done in read-only mode. I still have to try it with a running Windows domU, to see that it works flawlessly, but I don't see why.

Yeah, I would like to hear how this develops.


----------



## Killhouse

Quote:


> Originally Posted by *powerhouse*
> 
> 1. When you shut down a domU the CPU resources are freed. So you could configure multiple domUs with "oversubscription" of CPU time, as long as they don't run concurrently. You could even have two different domU.cfg files with different CPU and RAM parameters for the same domU, though Windows might complain about licenses at it sees different hardware when running with the different configs.
> But yes, dynamic assignment of resources would be better.
> 2. Hm, are you running Xen? I suggest you try LVM. There is a command for lvcreate that essentially gives you RAID1. LVM has also performance advantages with Xen. Don't forget to install the "signed" GPLPV driver under Linux that you can find here: http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers.
> I haven't played with different "disk" formats under Xen, just the plain phy:/path/to/lvm/volume. In this case, when using LVM, you just create a logical volume and don't format it (no mkfs). When installing Windows it will create an NTFS formatted virtual storage "container" (that's probably not the right word, but I hope you understand) which is not visible to Linux. Under Linux you use the kpartx command to find and register the domU virtual partitions on the LV, then you can mount and access them under Linux/LVM. It will see NTFS formatted drives. For Windows the whole LVM stuff is transparent and it only sees the NTFS drives. Mounting these virtual drives under Linux while running the domU should be done in read-only mode. I still have to try it with a running Windows domU, to see that it works flawlessly, but I don't see why.
> Yeah, I would like to hear how this develops.


1. I will only be running 1 domU at a time, but would have preferred if it could take over 3.8 cores rather than 3. I assume it will complain if I put "4" there!

2. I wasn't clear there. I have LVM on my SSD where dom0 and domUa/b/c will be installed but I have 2x1TB drives that I wanted to put in RAID1. I also wanted to install a backup OS (separate from dom0) onto those drives so that in case of my SSD dying I had something to boot to - this has been a lifesaver in the past. Maybe I could make the 900gb data storage on those with an LVM RAID, I hadn't considered that; but I will do some form of software backup instead.

3. The issue I'm having is with pciback. I have some error suggesting that my graphics card (and probably the other PCI devices) aren't attached to pciback or pcistub. I'm not really sure how to do this and the guide seems to skip a lot in the SSH/pciback section that I must be missing. Any ideas?


----------



## cladus

Quote:


> Originally Posted by *powerhouse*
> 
> According to http://ark.intel.com/products/65719/Intel-Core-i7-3770-Processor-8M-Cache-up-to-3_90-GHz the Intel i7-3770 has VT-d. Here are some observations:
> a) 4 cores / 8 threads - this translates into 8 virtual CPUs or vcpus in Xen. If I understand you correctly, you are planning to run 5 VMs (domU in Xen talk) plus the obligatory dom0. That doesn't leave many options. You will be able to dedicate 2 vcpus per guest for two guests, and one vcpu for the remaining domUs and dom0. It may be enough, but that depends on the applications.
> Some questions:
> Can you explain what you try to build?
> Will all VMs run concurrently, or do you plan to run them when needed and turn them off when not in use?
> b) Memory is limited to 32GB. For normal desktop tasks and even combined with games this should give you plenty of memory. dom0 (control domain = Linux) will need some 2GB, even with a full desktop system like Fedora or Ubuntu (unless you do some memory intensive tasks on it). A regular Windows 7 desktop with office applications, browser and email should need more than 4GB. A gaming desktop may need a little more, but 8GB should be enough. Only if you plan to do video or photo editing with large RAW files (uncompressed high-resolution images), CAD-CAM, software compilations, or similar memory intensive stuff will you be able to use more than 8GB of memory. But those last applications will also require a lot of CPU power to run efficient, in which case your CPU may be underpowered.
> c) Memory bandwidth is "only" 25GB/s.
> d) The 3770 has an internal HD 4000 graphics adapter, which can be used for the dom0. On the downside, it supports 1x PCIe x16, or 2x PCIe x8, which means if you plug in two discrete graphics adapters they will run at x8 speed instead of x16. I have no idea how that impacts graphics performance, but I guess it will.
> In comparison, the i7-3930K I use has 6 cores / 12 threads, can address a total of 64GB memory, has a bigger internal cache (12MB versus 8MB), and double the memory bandwidth (50GB/s). A C2 stepping version of it supports VT-d. It supports two PCIe x16 cards. BUT, it doesn't have an internal GPU.
> Depending on what you want to do, you may want to consider the i7 3930K.
> Intel boards are generally considered to have good BIOS versions and support. Still, check in the forums or on the Internet to see if it's been verified to work with PCI/VGA passthrough.
> 32GB should be enough memory, but the best is to list the applications or VMs you want to run and add up their memory requirement. See above. One remark on memory: memory kits are matched kits. So if you buy a 16GB kit and want to add another 16GB later, it may not work even with the same exact memory kit. My 3930K and Asus X79 motherboard combo were quite tricky about memory. And it gets more tricky the more memory you have.
> Last not least the graphics cards: If you want Nvidia, as of now, only the Nvidia "multi-OS" graphics cards are designed to work with VGA passthrough. There are hypervisor patches etc. for other (perhaps many) Nvidia cards, but that isn't for everyone (I tried and failed with a Quadro 600 card). AMD, on the other hand, has reportedly better VGA graphics passthrough support, but make sure to get cards that have been verified by other users.
> Hope this helps.


Most of the VM's will run at the same time, I think I will get the 3930k and Intel DX79TO Mobo, Can you also tell me what is C2 Stepping?
I will be using AMD for the graphics.
Here is a detail of what each VM will do:
VM1:
Fast Gaming
Fast Sony Vegas Rendering
Fast Photoshop
Youtube, Web and Email.

VM2:
Fast Gaming
Youtube, Web and Email.

VM3:
Youtube, Web and Email.

VM4:
Youtube, Web and Email.

VM5:
Fast calculations - Will not be used allot
Youtube, Web and Email.

how much ram will I need?


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> 1. I will only be running 1 domU at a time, but would have preferred if it could take over 3.8 cores rather than 3. I assume it will complain if I put "4" there!
> 2. I wasn't clear there. I have LVM on my SSD where dom0 and domUa/b/c will be installed but I have 2x1TB drives that I wanted to put in RAID1. I also wanted to install a backup OS (separate from dom0) onto those drives so that in case of my SSD dying I had something to boot to - this has been a lifesaver in the past. Maybe I could make the 900gb data storage on those with an LVM RAID, I hadn't considered that; but I will do some form of software backup instead.
> 3. The issue I'm having is with pciback. I have some error suggesting that my graphics card (and probably the other PCI devices) aren't attached to pciback or pcistub. I'm not really sure how to do this and the guide seems to skip a lot in the SSH/pciback section that I must be missing. Any ideas?


1. With a multi-threaded CPU a 4-core processor would give you 8 threads. You could assign 7 vcpus to the guest and leave 1 vcpu for the host, but that may be borderline. In a 7:1 setup you could increase the dom0 queue priority, giving it precedence over CPU time when needed.

2. If RAID1 doesn't work, use LVM with mirror (similar to RAID1). Well, I'm actually a little confused now. It seems you are trying to access a Linux software RAID via Windows? I doubt this works, neither will a natively installed Windows be able to read LVM volumes.

Here is what I would consider:

a. Run your SSD (/dev/sda) with LVM for everything except /boot (and /efi if you got it - though I discourage using UEFI unless really needed). Once everything works, boot your PC with a Linux live USB stick and backup the entire SSD to disk of the same or larger size using dd if=/dev/sda of=/dev/backup_disk or into an image file with dd if=/dev/sda of=backup.img which must reside on another disk. That will take quite some time, but with the first option you would have a bootable, fully installed disk that you can just swap for your SSD; with the second option you would need to replace the SSD for a new one of the same or larger size and install the disk image via dd command (Linux live USB stick).

b. To use Windows with Linux software RAID or LVM, Windows must be installed as domU. In this case it runs on top of the hypervisor / Linux and they will take care of it. Xen will create some storage container (I don't know how it's called) that will reside on your LVM volume(s). I've tried to format an LVM striped volume (similar to RAID0) with NTFS and specified it using the phy:/dev... option in the domU configuration file, but this gave me some strange results. Windows would see it as a drive, but with multiple partitions, some formatted, some not. I abandoned this attempt.

To access the Windows file system under Linux (dom0) you need to use the kpartx command.

c. As to the data drives: You could run software RAID1 combined with LVM, or just LVM with mirror. I think hardware RAID is generally discouraged. For example, if your computer or specifically the hardware RAID controller fails on you, you would have to get the exact same computer/controller to restore your information, as each manufacturer may implement RAID in a different way. With Linux software RAID or LVM you won't have this issue, as it's either backwards compatible (if you use a newer version) or you can just keep one or two USB sticks with the Linux live system in case you need it.

As mentioned under b., if you use these data drives from within a Windows domU, you can mount/access them under Linux via kpartx.

Other backup possibilities with LVM are snapshots, which can be done on a live system. You need a snapshot volume that must be as big or bigger as the amount of data you expect to be written/changed during the live backup. Just for clarity: a snapshot by itself doesn't provide a backup!!! Once the snapshot is created, you need to back it up to another drive/media or remote location (rsync ?, which of course works locally as well). To do the actual backup use dd or better perhaps rsync, or a backup utility under Linux. At the end you must remove the snapshot. It's best to read up on LVM and snapshots to get a better understanding.

To sum up the backup strategy:
- RAID1 or LVM mirror will give you online backup of your data. The caveat is that when you accidentally delete some file(s) they are gone.
- With LVM snapshot and backup you can create copies of your data manually or at given times (via a cron job). Some backup utilities like rsync don't need to copy the whole data each time, but will backup only the files that have changed. There are also backup utilities that keep copies of the original files so in case you deleted some files you can recover them even after you did a backup.

I would go with a combination of both RAID1 or LVM mirror and some snapshot/backup scheme with history, with the backup preferably on external media.

3. Yeah, pciback takes some time to figure out. Let's start with this command:

xm pci-list-assignable-devices

If you don't get a list of devices that can be assigned, then they haven't been detached yet. A short how-to on this and VGA passthrough (for Ubuntu-based systems) is http://gro.solexiv.de/2012/08/pci-passthrough-howto/, a longer one with more details (again for Ubuntu-based systems) is mine here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&p=629268#p629268. In my how-to see part 3 steps 3-6.

Be aware that if you use Fedora, you may need to make some changes or use different commands (to update your init.d). But the idea is the same.

Here in essence:
a) If xen-pciback isn't compiled into the kernel, you need to load it. Under Ubuntu/Debian systems you can use the /etc/modules file and add "xen-pciback passthrough=1" (without quotes) to load the pciback module during boot.
b) You need to detach the PCI devices from their drivers to make them assignable later on. In both how-tos this is done by the pciback script. It unbinds the driver from it's PCI device, adds a new slot to the PCI backend list, and binds it to the new slot. Note the long PCI IDs with domain, e.g. 0000:01:00.0.
c) Once the PCI device has been bound to pciback, it becomes assignable. You can use the command above (xm pci-list...) to see the assignable devices and check that all is OK.

From there on you should be able to execute xm create /etc/xen/your_domU.cfg, or xm new /etc/xen/your_domU.cfg followed by virt-manager (in both cases pointing to your domU configuration file).

Good luck!


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> 1. I will only be running 1 domU at a time, but would have preferred if it could take over 3.8 cores rather than 3. I assume it will complain if I put "4" there!
> 2. I wasn't clear there. I have LVM on my SSD where dom0 and domUa/b/c will be installed but I have 2x1TB drives that I wanted to put in RAID1. I also wanted to install a backup OS (separate from dom0) onto those drives so that in case of my SSD dying I had something to boot to - this has been a lifesaver in the past. Maybe I could make the 900gb data storage on those with an LVM RAID, I hadn't considered that; but I will do some form of software backup instead.
> 3. The issue I'm having is with pciback. I have some error suggesting that my graphics card (and probably the other PCI devices) aren't attached to pciback or pcistub. I'm not really sure how to do this and the guide seems to skip a lot in the SSH/pciback section that I must be missing. Any ideas?


Re 3. I forgot to mention: I believe the tutorial at the start of this thread uses pciback compiled into the kernel. Or virt-manager (see part 7 of the tutorial) is able to do the job on its own. I actually was a little confused when I saw that I had to add devices in virt-manager that were already listed, but obviously this does the trick of detaching the devices. Have you tried this?


----------



## jp777cmoe

Can you make a VM with Windows XP installed and do this guide?


----------



## powerhouse

Quote:


> Originally Posted by *cladus*
> 
> Most of the VM's will run at the same time, I think I will get the 3930k and Intel DX79TO Mobo, Can you also tell me what is C2 Stepping?
> I will be using AMD for the graphics.
> Here is a detail of what each VM will do:
> VM1:
> Fast Gaming
> Fast Sony Vegas Rendering
> Fast Photoshop
> Youtube, Web and Email.
> VM2:
> Fast Gaming
> Youtube, Web and Email.
> VM3:
> Youtube, Web and Email.
> VM4:
> Youtube, Web and Email.
> VM5:
> Fast calculations - Will not be used allot
> Youtube, Web and Email.
> how much ram will I need?


Hello cladus.

1. Intel i3930K or i3960X C2 stepping - older batches of these processors - designated C1 stepping - had VT-d disabled. The new batches starting somewhere around the end of January/February 2012 are C2 stepping types. On the box you should see this code: SR0KY.

2. The 3930K should give better performance over the i7-3770, at the cost of higher power consumption (130W versus 70W). You need good cooling for it. I am not familiar with this Intel motherboard, but generally Intel is supposed to offer good VT-d support. Search the Internet to make sure it does.

3. AMD graphics: go with cards that have been proven to work. Make sure your chassis has plenty of cooling to deal with 2 graphics cards and a powerful CPU that produces a lot of heat.

4. VMs: You're trying to do a lot with your PC







.

VM1: 4 vcpus should be the minimum. Photoshop can use quite some CPU power, particularly when working on RAW camera files or doing sharpening etc. on high resolution images. I'm not familiar with games, but I guess that 4 vcpus should do, combined with a strong graphics card.

VM2: Start with 2 vcpus and see how it works.

VM3-VM4: 1 vcpu is hopefully enough, make it 2 if you don't run all VMs concurrently.

VM5: 2 vcpus or more if you don't run all VMs. You'll be using the CPU for the calculations.

dom0: 2 vcpus - you don't want to throttle it down and it needs to handle a lot with all those VMs. You might want to give dom0 a higher processor / queue priority.

5. Memory:
4-6GB for each VM should be enough. For VM1 (Photoshop) and perhaps VM2 you might give it a little more, say 8-10GB. dom0 should be OK with 2GB. You won't be able to use all memory for dom0 and VMs, as Xen has to set aside some of it.

Check the games you want to play and their memory / CPU requirements. I believe 32GB will do.

Get a large SSD (250GB at least), or perhaps 2. I use my SSD for the OSes and programs and use two LVM striped (like RAID0) hard drives for data, as well as some others for backup.

Windows 7 takes a lot of disk space. How many Windows VMs you plan to run?

For the things you do on VM3-5 you could use Linux. It uses much less resources both regarding drive space and CPU. There are PV packages for several Linux distributions that might work well. However, I don't have any idea how well you can mix HVM and PV domUs under Xen. In case of using PV domUs, you might need to give your dom0 more resources (vcpus). HVM means fully virtualized, PV is para-virtualized meaning the domU uses drivers under the dom0.

I really like your idea and hope this works out for you.

Remark: Please take my recommendations with a grain of salt. I have never done what you are trying to do here, and I have no idea how this will work. I'm only running a dom0 and one VM (domU). Though I definitely played with the idea of remote desktops.

Ah, how do you use your VMs? Via remote desktop I guess?

If you want to use VM3-5 via remote desktop, will you use thin clients?


----------



## ghormoon

Quote:


> Originally Posted by *powerhouse*
> 
> Sorry I missed your post (config file).
> Here are some comments:
> disk = [ 'phy:/dev/sda2,sda,w' , 'file:/win7recovery.iso,sdc:cdrom,r' ]
> I couldn't get Windows installer to start with "sda" setting. I think it complained about missing driver disk or so. Try to change your above line to read:
> disk = [ 'phy:/dev/sda2,hda,w' , 'file:/win7recovery.iso,hdc:cdrom,r' ]
> I think you have far too many PCIs passed through. Start with the basic ones you need:
> Instead of: pci=[ '01:00.0', '01:00.1' , '00:1a.0' , '00:1b.0' , '00:1d.0' , '24:00.0' , '24:00.1' , '24:00.2' , '25:00.0' , '26:00.0' ]
> Try: pci=[ '01:00.0', '01:00.1' , '00:1d.0' ] # replace 00:1d.0 with 00:1a.0 if it doesn't work
> 01:00.0 is for the graphics adapter, 01.00.1 is for the audio on that graphics adapter when using an HDMI link. 00.1a.0 is your Intel CPU integrated USB host #2 (the CPU has two, the other one is 00.1d.0). That's where your keyboard / mouse should be connected to. Check the dmesg output to find your mouse and keyboard - note the PCI ID(s).
> You first want to try to install the Windows domU, with or without sound support. Once can run Windows in domU, shut it down and add additional PCIs you want to pass through, for sound, USB etc. devices.


did not help at all ...
Quote:


> Originally Posted by *powerhouse*
> 
> Don't try to pass through your SATA controller 24:00.x !!! You may be trying to take away your hard drive from dom0, which is not a good idea. There is no need to pass through the SATA controller to Windows.
> The same goes for your network controller 25:00.0. If you take it away from the Linux dom0, you have no way to access your machine during the Windows installation.
> Once you changed your Win7 cfg file and perhaps rebooted, check the output of the following command when using your ssh remote session:
> xm pci-list-assignable-devices
> It should list the 3 PCI IDs you specified, if they have been successfully detached !!! If there is nothing displayed, check your pciback shell script or config file to make sure it contains only the three PCI IDs you listed in the win7.cfg file, but using the domain ID in front of the PCI ID like here:
> 0000:01:00.0
> (See your pciback script for exact syntax!)
> So, when you don't get any PCIs listed with the xm pci-list... command, you need to run the pciback script to detach the PCIs. If the script doesn't give you an error, check again with xm pci-list-assignable-devices. Do they show up now?


sata is 00:1f.2, 24:... is for firewire and such, 25:... is wifi (i'm using ethernet for management, 00:19.0)
I have init.d script to deattach the devices, it doesn't start automatically, dunno why, but when I run it manually, I see the devices as assignable
Quote:


> Originally Posted by *powerhouse*
> 
> Important: The above can only be done from a remote PC via ssh or VNC, as it will disconnect your keyboard/mouse and screen from the Linux dom0 so you won't see anything nor be able to use your mouse/keyboard.
> Let's start with the above suggestions and go from there.


I'm logging through putty and xming all the time ...
When I allow vnc in the config file, it gives me errors, I got rid of some by diabling sound, anyway, now I'm stuck on

xs_read(/local/domain/0/device-model/4/xen_extended_power_mgmt): read error
bind() failed

(in /var/log/xen/qemu-dm-win7.log)


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> did not help at all ...
> sata is 00:1f.2, 24:... is for firewire and such, 25:... is wifi (i'm using ethernet for management, 00:19.0)
> I have init.d script to deattach the devices, it doesn't start automatically, dunno why, but when I run it manually, I see the devices as assignable
> I'm logging through putty and xming all the time ...
> When I allow vnc in the config file, it gives me errors, I got rid of some by diabling sound, anyway, now I'm stuck on
> xs_read(/local/domain/0/device-model/4/xen_extended_power_mgmt): read error
> bind() failed
> (in /var/log/xen/qemu-dm-win7.log)


Go with the minimum of devices to pass through, that is only the ones you need for installation to succeed! You can always add them later when Windows is running.

Have you run update-rc.d or whatever you need to run to activate your pciback script? Seems like it's not executed at boot. Of course, check to see that the script has been made executable.

Another possible cause is that the xen-pciback module is loaded after the pciback script execution. In my case inserting xen-pciback passthrough=1 into the /etc/modules file worked.

I also disabled sound (actually something I still have to make work under Windows, but I don't need it now).

Now to the problem at hand:

xs_read(/local/domain/0/device-model/4/xen_extended_power_mgmt): read error bind() failed (in /var/log/xen/qemu-dm-win7.log)

Possible causes/solutions:
1. You try to passthrough a PCI device that's supposed to have extended power management. Find out which PCI device it is and if not really needed for installation, remove the PCI ID from both the pciback script and the /etc/xen/win7.cfg (or whatever you named it) domU config file.

2. Extended power management doesn't work in your machine, or with the specific device. There should be a way to disable extended power management, but I can't remember how. Having searched for this error, I'm not sure this is the reason why your domU doesn't start. It seems to be a common error and probably can be ignored.

I assume you copied your Windows DVD to an .img file in a folder on your hard drive? If not, this could be the culprit. You can use dd if=/dev/cdrom of=/path/to/your/win7.img file. And add this image as cdrom to your win7.cfg file using the phy:/.... option.


----------



## ghormoon

I've beem trying only 01:00.0 and 01:00.1 and got that ... and yes, I DDed the dvd into iso ...

is there any way to force xen to ignore the error and start the VM?


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> did not help at all ...
> sata is 00:1f.2, 24:... is for firewire and such, 25:... is wifi (i'm using ethernet for management, 00:19.0)
> I have init.d script to deattach the devices, it doesn't start automatically, dunno why, but when I run it manually, I see the devices as assignable
> I'm logging through putty and xming all the time ...
> When I allow vnc in the config file, it gives me errors, I got rid of some by diabling sound, anyway, now I'm stuck on
> xs_read(/local/domain/0/device-model/4/xen_extended_power_mgmt): read error
> bind() failed
> (in /var/log/xen/qemu-dm-win7.log)


Can you post your new win7.cfg file?


----------



## ghormoon

kernel = "hvmloader"
builder='hvm'
memory = 3072
name = "win7"
vcpus=3
pae=1
acpi=1
apic=1
#vif = [ 'type=ioemu, bridge=xenbr0' ]
disk = [ 'phy:/dev/sda2,hda,w' , 'file:/win7recovery.iso,hdc:cdrom,r' ]
device_model = 'qemu-dm'
boot="dc"
sdl=0
opengl=1
vnc=1
vncdisplay="10"
vnclisten="192.168.1.42"
vncpasswd='42'
stdvga=0
#nographic=1 #!!! only uncomment this if you are using win8 or are trying to get a nvidia card to work
serial='pty'
tsc_mode=0
soundhw=''
usb=1
usbdevice='mouse'
gfx_passthru=0
pci=[ '01:00.0', '01:00.1' ]
#pci=[ '01:00.0', '01:00.1' , '00:1a.0' , '00:1d.0' , '24:00.0' , '24:00.1' , '24:00.2' , '25:00.0' , '26:00.0' ]


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> I've beem trying only 01:00.0 and 01:00.1 and got that ... and yes, I DDed the dvd into iso ...
> is there any way to force xen to ignore the error and start the VM?


Ok.

Can you post the output of xm dmesg ? Try this command after you have detached the PCI devices, as well as after running xm create or xm new (see below).

Let's have another look at the win7.cfg file. If I remember correctly, the preferred way is to use VNC.

1. Check that you have vncviewer installed on your Linux system and on the remote system.
2. Use only these three lines for VNC in your win7.cfg file:
vnc=1
vncpasswd=''
stdvga=0

3. See if VNC works from your remote computer. If not, connect to your laptop using ssh -X ... You may have to reboot your laptop before running the following commands:
a) As root, run xm pci-list-assignable-devices
b) If they don't show up, run the pciback script as you did before. Check again with xm pci-list...
c) xm new /etc/xen/win7.cfg
d) virt-manager (you may have to install virt-manager and its dependancies AND edit your /etc/xen/xend-config.sxp file to uncomment (xend-unix-server yes))
e) Switch to the X window and you should see the virt-manager GUI. Double-click your win7 domU, click the Terminal button and then > (run).
f) Does it start? Wait a little as the boot process takes some time, ~ 15-30 sec.

4. If nothing of the above works, particularly if VNC doesn't work, comment out the above lines in your win7.cfg file, except stdvga=0, and change that to stdvga=1. Check for the stdvga option on the Xen.org page for more explanations. I haven't used stdvga as VNC works for me. IIRC stdvga=1 tells Xen to use the standard X window output so you will need to use ssh -X to get to the screen. However, I remember seeing comments that discourage the use of stdvga=1.

In any of the above attempts, check if you still have access to your laptop via VNC or ssh (command line access) after booting or when running the xm ... commands.

See my next post for additional thoughts!


----------



## ghormoon

I've tried most of that before, now I'm installing win7 on the partition traditionally and I'll try if it loads in xen then







(also in half an hour, my first lab starts at school and i need at least some working OS ...)
I'll experiment more in the evening


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> kernel = "hvmloader"
> builder='hvm'
> memory = 3072
> name = "win7"
> vcpus=3
> pae=1
> acpi=1
> apic=1
> #vif = [ 'type=ioemu, bridge=xenbr0' ]
> disk = [ 'phy:/dev/sda2,hda,w' , 'file:/win7recovery.iso,hdc:cdrom,r' ]
> device_model = 'qemu-dm'
> boot="dc"
> sdl=0
> opengl=1
> vnc=1
> vncdisplay="10"
> vnclisten="192.168.1.42"
> vncpasswd='42'
> stdvga=0
> #nographic=1 #!!! only uncomment this if you are using win8 or are trying to get a nvidia card to work
> serial='pty'
> tsc_mode=0
> soundhw=''
> usb=1
> usbdevice='mouse'
> gfx_passthru=0
> pci=[ '01:00.0', '01:00.1' ]
> #pci=[ '01:00.0', '01:00.1' , '00:1a.0' , '00:1d.0' , '24:00.0' , '24:00.1' , '24:00.2' , '25:00.0' , '26:00.0' ]


Looks good, except that I would comment out soundhw='' (# soundhw='') for now, and change the vnc options as mentioned in my above post. With regard to VNC, have a look at the /etc/xen/xend-config.sxp file and check if there are any VNC related settings that could overwrite your win7.cfg file settings, for example I uncommented (vncpasswd '').

Now my afterthoughts on my above comments:

You are trying to pass through your primary display adapter. (I've had success only with the secondary display adapter, though I believe my problems were with my previous graphics card that didn't support VGA passthrough.)

When you boot your display adapter will be bound to the display driver you use for your graphics card, unless you detach the PCI ID before the driver gets loaded. This may complicate things. In my setup I had to blacklist the nouveau driver for my secondary display adapter since it interfered. My PC would get stuck right after login, with error messages pointing to the nouveau driver. In my case I didn't need the driver, as I'm using my primary display adapter (AMD) for the dom0 Linux, and the secondary (Nvidia) display adapter is only used by Windows.

To pass through a primary display adapter, you might need to prevent the graphics driver to be loaded under Linux. Your access to the Linux dom0 would then be via Windows and VNC or similar, which doesn't require the graphics driver for your card.

Check the output of dmesg to see if and when your graphics driver is run. To avoid loading the graphics driver, try booting into runlevel 3 (init 3) or once booted, run init 3 to switch to that run level (i.e. command line only). I'm not sure if that disables the graphics driver (when switching from runlevel 5 to 3), but it's worth a try.

After that run the pciback script etc.


----------



## powerhouse

Quote:


> Originally Posted by *ghormoon*
> 
> I've tried most of that before, now I'm installing win7 on the partition traditionally and I'll try if it loads in xen then
> 
> 
> 
> 
> 
> 
> 
> (also in half an hour, my first lab starts at school and i need at least some working OS ...)
> I'll experiment more in the evening


You won't be able to boot Windows into Xen from a traditional installation. Even if you get something started under Xen, Windows will discover a totally new system and will probably fail to boot.

Yep, it's hard to work with a PC without functioning OS.

You mentioned that you are using a recovery disk for Windows? Is that a fully installable Windows version, or the disk that came with your pre-installed Windows laptop?

To make sure that your Windows disk is in fact installable under Xen, I would first try to install it as a normal HVM guest without passthrough at all. If it installs and runs, then you can try the passthrough part.

Good luck!


----------



## Killhouse

Quote:


> Originally Posted by *powerhouse*
> 
> 1. With a multi-threaded CPU a 4-core processor would give you 8 threads. You could assign 7 vcpus to the guest and leave 1 vcpu for the host, but that may be borderline. In a 7:1 setup you could increase the dom0 queue priority, giving it precedence over CPU time when needed.
> 2. If RAID1 doesn't work, use LVM with mirror (similar to RAID1). Well, I'm actually a little confused now. It seems you are trying to access a Linux software RAID via Windows? I doubt this works, neither will a natively installed Windows be able to read LVM volumes.
> Here is what I would consider:
> a. Run your SSD (/dev/sda) with LVM for everything except /boot (and /efi if you got it - though I discourage using UEFI unless really needed). Once everything works, boot your PC with a Linux live USB stick and backup the entire SSD to disk of the same or larger size using dd if=/dev/sda of=/dev/backup_disk or into an image file with dd if=/dev/sda of=backup.img which must reside on another disk. That will take quite some time, but with the first option you would have a bootable, fully installed disk that you can just swap for your SSD; with the second option you would need to replace the SSD for a new one of the same or larger size and install the disk image via dd command (Linux live USB stick).
> b. To use Windows with Linux software RAID or LVM, Windows must be installed as domU. In this case it runs on top of the hypervisor / Linux and they will take care of it. Xen will create some storage container (I don't know how it's called) that will reside on your LVM volume(s). I've tried to format an LVM striped volume (similar to RAID0) with NTFS and specified it using the phy:/dev... option in the domU configuration file, but this gave me some strange results. Windows would see it as a drive, but with multiple partitions, some formatted, some not. I abandoned this attempt.
> To access the Windows file system under Linux (dom0) you need to use the kpartx command.
> c. As to the data drives: You could run software RAID1 combined with LVM, or just LVM with mirror. I think hardware RAID is generally discouraged. For example, if your computer or specifically the hardware RAID controller fails on you, you would have to get the exact same computer/controller to restore your information, as each manufacturer may implement RAID in a different way. With Linux software RAID or LVM you won't have this issue, as it's either backwards compatible (if you use a newer version) or you can just keep one or two USB sticks with the Linux live system in case you need it.
> As mentioned under b., if you use these data drives from within a Windows domU, you can mount/access them under Linux via kpartx.
> Other backup possibilities with LVM are snapshots, which can be done on a live system. You need a snapshot volume that must be as big or bigger as the amount of data you expect to be written/changed during the live backup. Just for clarity: a snapshot by itself doesn't provide a backup!!! Once the snapshot is created, you need to back it up to another drive/media or remote location (rsync ?, which of course works locally as well). To do the actual backup use dd or better perhaps rsync, or a backup utility under Linux. At the end you must remove the snapshot. It's best to read up on LVM and snapshots to get a better understanding.
> To sum up the backup strategy:
> - RAID1 or LVM mirror will give you online backup of your data. The caveat is that when you accidentally delete some file(s) they are gone.
> - With LVM snapshot and backup you can create copies of your data manually or at given times (via a cron job). Some backup utilities like rsync don't need to copy the whole data each time, but will backup only the files that have changed. There are also backup utilities that keep copies of the original files so in case you deleted some files you can recover them even after you did a backup.
> I would go with a combination of both RAID1 or LVM mirror and some snapshot/backup scheme with history, with the backup preferably on external media.
> 3. Yeah, pciback takes some time to figure out. Let's start with this command:
> xm pci-list-assignable-devices
> If you don't get a list of devices that can be assigned, then they haven't been detached yet. A short how-to on this and VGA passthrough (for Ubuntu-based systems) is http://gro.solexiv.de/2012/08/pci-passthrough-howto/, a longer one with more details (again for Ubuntu-based systems) is mine here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&p=629268#p629268. In my how-to see part 3 steps 3-6.
> Be aware that if you use Fedora, you may need to make some changes or use different commands (to update your init.d). But the idea is the same.
> Here in essence:
> a) If xen-pciback isn't compiled into the kernel, you need to load it. Under Ubuntu/Debian systems you can use the /etc/modules file and add "xen-pciback passthrough=1" (without quotes) to load the pciback module during boot.
> b) You need to detach the PCI devices from their drivers to make them assignable later on. In both how-tos this is done by the pciback script. It unbinds the driver from it's PCI device, adds a new slot to the PCI backend list, and binds it to the new slot. Note the long PCI IDs with domain, e.g. 0000:01:00.0.
> c) Once the PCI device has been bound to pciback, it becomes assignable. You can use the command above (xm pci-list...) to see the assignable devices and check that all is OK.
> From there on you should be able to execute xm create /etc/xen/your_domU.cfg, or xm new /etc/xen/your_domU.cfg followed by virt-manager (in both cases pointing to your domU configuration file).
> Good luck!


1. I'm using an AMD chip with 4 cores, so I would only have 4 threads I believe. I think what I would want to do is impossible: assign all 4 threads to both dom0 and domU and have the clock cycles shared such that a heavy application could run on dom0 constantly, but allow gaming on domU temporarily by sharing CPU time with dom0.

2. The SSD is moot really, its just an LVM for the different doms, no problem there. The issue was my "data" drives which I could only manage to do a software RAID on. Its not an issue, I just unRAIDed them and I'll use them as regular drives. I'll run some sort of automatic backup software on them from dom0. I also install a backup OS on one of the mechanical drives in case of SSD failure - I don't need full usage then, just a simple OS I can log in to to trouble-shoot (it'll probably be Fedora or Windows).

I just wanted to make sure that data was safe. And I'd rather do it in a transparent copy+paste into a file system that I *know* can be opened by Windows, Linux... or any other OS that I can use to retrieve the data in case of my original installs breaking









3. I've been trying a few random things with pciback, I will try your links in the next day or so when I'm back home. I ran a script that was meant to detach the graphics driver, make a slot in pciback for it, then attach it to pciback. I don't know if it works, because my screen obviously shuts off after the first line







I have a feeling that is the script I should be forcing at init, but I'll do some more investigation!

Quote:


> Originally Posted by *powerhouse*
> 
> ...
> To pass through a primary display adapter, you might need to prevent the graphics driver to be loaded under Linux. Your access to the Linux dom0 would then be via Windows and VNC or similar, which doesn't require the graphics driver for your card...


I'm going to try very hard to get around this and end up with a system that switches (full screen) between dom0 and domU, each having total control of the graphics card and therefore the monitor









Thanks for your tremendous help again. +rep!


----------



## cladus

VM2-5 will be using remote desktop's on my families computers/laptops and I will use VM1 on the main computer.


----------



## TheDave

Has anyone tried using ZFS as the data store with xen or even just having ZFS alongside xen (used for separate storage)? I am very interested in migrating my FreeBSD server using ZFS to Fedora/Xen but do not want to ditch ZFS.


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> 1. I'm using an AMD chip with 4 cores, so I would only have 4 threads I believe. I think what I would want to do is impossible: assign all 4 threads to both dom0 and domU and have the clock cycles shared such that a heavy application could run on dom0 constantly, but allow gaming on domU temporarily by sharing CPU time with dom0.
> 2. The SSD is moot really, its just an LVM for the different doms, no problem there. The issue was my "data" drives which I could only manage to do a software RAID on. Its not an issue, I just unRAIDed them and I'll use them as regular drives. I'll run some sort of automatic backup software on them from dom0. I also install a backup OS on one of the mechanical drives in case of SSD failure - I don't need full usage then, just a simple OS I can log in to to trouble-shoot (it'll probably be Fedora or Windows).
> I just wanted to make sure that data was safe. And I'd rather do it in a transparent copy+paste into a file system that I *know* can be opened by Windows, Linux... or any other OS that I can use to retrieve the data in case of my original installs breaking
> 
> 
> 
> 
> 
> 
> 
> 
> 3. I've been trying a few random things with pciback, I will try your links in the next day or so when I'm back home. I ran a script that was meant to detach the graphics driver, make a slot in pciback for it, then attach it to pciback. I don't know if it works, because my screen obviously shuts off after the first line
> 
> 
> 
> 
> 
> 
> 
> I have a feeling that is the script I should be forcing at init, but I'll do some more investigation!
> I'm going to try very hard to get around this and end up with a system that switches (full screen) between dom0 and domU, each having total control of the graphics card and therefore the monitor
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks for your tremendous help again. +rep!


1. If you have a native Windows installation running, check in Device Manager and see how many threads you got. On Intel I got twice as many threads as cores. Could be Intel-specific, though.

Actually I read today something about oversubscribing vcpus. It makes perfect sense, since I know there are features like queue priority and CPU pinning which only make sense with some sort of floating CPU management. To be perfectly honest, I'n very new to all this and half a year ago I wouldn't have known how to spell Xen, let alone hypervisor.

So, I suggest you check out the Xen.org documentation on vcpus and all the options in the domU config file and xend-config files under /etc/xen. I clearly remember that dom0 should be given dedicated CPU resources to prevent bottlenecks. You could experiment with different settings and see how they work out.

2. If you need to work in Windows, then that is a good option to keep you going when things go wrong. But otherwise use Linux, as you have much more over your system, particularly the ext4 drives and LVM. I'd also keep a bootable live USB stick with your current Linux system handy.

3. That sounds good for a start! Before putting anything in init, I would use remote access via VNC or ssh (-X) to access your machine and see what's going on. I think I posted a link to my how-to for Ubuntu-based systems. You could check there to see what's needed. If you feel comfortable with Fedora, I'm sure you can manage to change the steps to make it work. Unfortunately Fedora and me don't get along very well.

Keep us informed.


----------



## powerhouse

Quote:


> Originally Posted by *cladus*
> 
> VM2-5 will be using remote desktop's on my families computers/laptops and I will use VM1 on the main computer.


Thanks for the explanation. A year ago I've experimented a little with different remote desktop protocols. One thing I noticed is that there are some differences in performance and CPU usage between the protocols. For example using VNC might be just about OK for normal office, web browsing, etc work, but you can forget any video (youtube or VLC) stuff unless you stream VLC from your desktop to the clients. NX is an interesting option but it put a lot of load on my remote machine (perhaps also on the desktop, I can't remember). IIRC the Spice project is also working on a solution, though I haven't tried it.

In essence, any remote desktop solution that works well with CAD-CAM or video editing should be a good starting point. If you haven't got one, I suggest you get a Gigabit router cause 100Mb/s are just not enough in most cases.

By the way I thought about a similar setup to avoid having to deal with all the PCs in the family. I use remote desktop for some time now, either from the desktop to all the PCs or wireless from my Macbook to various PCs. For desktop work it's fine. I haven't tried any game playing as this is most likely not working in my current setup. I've tried video running on the remote and it's not usable.

So, I don't want to discourage you but remote desktop is another thing you will have to look into.


----------



## spikexp

Now that suck, just got offered a 2500k+GA-Z68XP-UD3 for 200$... and now I see that none of it support VT-d...

Why do "k" don't support vt-d, is there a physical reason?


----------



## powerhouse

Quote:


> Originally Posted by *spikexp*
> 
> Now that suck, just got offered a 2500k+GA-Z68XP-UD3 for 200$... and now I see that none of it support VT-d...
> Why do "k" don't support vt-d, is there a physical reason?


It's either been turned off inside the CPU or no code included. Anyway, for all practical purposes, there is no VT-d. And yes, you need VT-d support both in the CPU and the motherboard, unless you try with older CPUs where the memory controller and VT-d is on the board chipset (again, the board must provide it).


----------



## Killhouse

Quote:


> Originally Posted by *powerhouse*
> 
> 1. If you have a native Windows installation running, check in Device Manager and see how many threads you got. On Intel I got twice as many threads as cores. Could be Intel-specific, though.
> Actually I read today something about oversubscribing vcpus. It makes perfect sense, since I know there are features like queue priority and CPU pinning which only make sense with some sort of floating CPU management. To be perfectly honest, I'n very new to all this and half a year ago I wouldn't have known how to spell Xen, let alone hypervisor.
> So, I suggest you check out the Xen.org documentation on vcpus and all the options in the domU config file and xend-config files under /etc/xen. I clearly remember that dom0 should be given dedicated CPU resources to prevent bottlenecks. You could experiment with different settings and see how they work out.
> 2. If you need to work in Windows, then that is a good option to keep you going when things go wrong. But otherwise use Linux, as you have much more over your system, particularly the ext4 drives and LVM. I'd also keep a bootable live USB stick with your current Linux system handy.
> 3. That sounds good for a start! Before putting anything in init, I would use remote access via VNC or ssh (-X) to access your machine and see what's going on. I think I posted a link to my how-to for Ubuntu-based systems. You could check there to see what's needed. If you feel comfortable with Fedora, I'm sure you can manage to change the steps to make it work. Unfortunately Fedora and me don't get along very well.
> Keep us informed.


1. I think you are talking about Intel Hyperthreading which AMD does not have. Its some sort of virtual threading at the hardware level I believe. I've been reading a little that you can oversubscribe CPUs - using this technique you can run 8 VMs on (for example) 1 core. The VMs will share the CPU time. I'm planning to give dom0 and domU the same 4 cores and let them fight for it. Should work well! Unfortunately, oversubscribing memory is not so trivial (impossible with Xen as far as I can tell) but this isn't such a big issue for me.

2. I decided to put a Linux backup on one mechanical drive and a Windows backup on the other (probably images of the two main VMs).

3. I don't actually know if pciback is installed as a module or... the other option... using the xen-pciback hide tools doesn't work, which makes things a bit more challenging. Hopefully I should have time to play with all this tonight.

Thanks again for your ideas and suggestions. I will check out your how-to and get some more info on VNC/SSH!


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> 1. I think you are talking about Intel Hyperthreading which AMD does not have. Its some sort of virtual threading at the hardware level I believe. I've been reading a little that you can oversubscribe CPUs - using this technique you can run 8 VMs on (for example) 1 core. The VMs will share the CPU time. I'm planning to give dom0 and domU the same 4 cores and let them fight for it. Should work well! Unfortunately, oversubscribing memory is not so trivial (impossible with Xen as far as I can tell) but this isn't such a big issue for me.
> 2. I decided to put a Linux backup on one mechanical drive and a Windows backup on the other (probably images of the two main VMs).
> 3. I don't actually know if pciback is installed as a module or... the other option... using the xen-pciback hide tools doesn't work, which makes things a bit more challenging. Hopefully I should have time to play with all this tonight.
> Thanks again for your ideas and suggestions. I will check out your how-to and get some more info on VNC/SSH!


1. You can also give dom0 higher priority. In my setup I give dom0 one CPU core = 2 threads, out of 6 cores = 12 threads. Linux works just fine like that. When I shutdown my Win domU I regain all cores. With the SSD it takes only seconds to shut down Windows, so it's no big deal. Booting takes longer, though, about 20-30 sec. So when I'm ripping DVDs for my media center under Linux I can just give all the CPU power to Linux.

2. I doubt the VMs will be bootable on bare metal. I would give it a try and boot them up to make sure things work. If not, copy (clone) the entire SSD with Xen hypervisor, dom0 and Win domU, check the fstab file and grub files to make sure they address the right drives.

3. You mean xen-pciback hide ... in grub? If it doesn't work it means xen-pciback is not compiled into the kernel. Enter the "xen-pciback passthrough=1" line into /etc/modules. Reboot and check with lsmod | grep pciback to see the module is loaded.

IIRC you have 2 graphics cards? If yes you can run the pciback script to detach the secondary graphics card. Check with xm pci-list-attachable-devices to see the PCI IDs you want to pass through. The next step is to run the domU with xm new /etc/xen/win7.cfg and virt-manager or just xm create /etc/xen/win7.cfg and connect to it via VNC.

In any case, it's much easier to passthrough a secondary graphics card, as long as it supports VGA passthrough. A range of AMD cards are good for that. In case of Nvidia you need to stick to "multi-OS" cards such as the Quadro 2000 (I have) or even more expensive Quadro 4000 etc. cards.


----------



## dizzy4

Hey all!

First off: A big sorry for being away for so long









I am mad at myself too, but life happens.

Anyway I am back now, and will start working on a new version of the guide with the revised 4.1.3 Hypervisor and a more lightweight dom0. Good news is that my system is still 100% stable and runs 24/7









Expect updates soon!

Thank you all for adding to the thread while I was away +rep


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Hey all!
> First off: A big sorry for being away for so long
> 
> 
> 
> 
> 
> 
> 
> 
> I am mad at myself too, but life happens.
> Anyway I am back now, and will start working on a new version of the guide with the revised 4.1.3 Hypervisor and a more lightweight dom0. Good news is that my system is still 100% stable and runs 24/7
> 
> 
> 
> 
> 
> 
> 
> 
> Expect updates soon!
> Thank you all for adding to the thread while I was away +rep


Good to see you back - I already felt hitchhiking your thread. I look forward to the new guide. And thanks for sharing your experience with running it 24/7.

Some questions: People have been asking about over-provisioning vcpus. I myself am not certain about it, but it seems Xen is providing virtual CPUs to the dom0 and domUs. From the Xen webpages I gather that it's good practice to pin some vcpus or core(s) to dom0. Another way of ensuring that dom0 gets enough resources is to increase it's priority for processor time. Any experience with this or suggestions?

Memory ballooning: This is another feature of Xen. However, it is suggested to disable ballooning and use fixed memory for dom0 and domUs since Linux supposedly isn't good at dealing with changing memory sizes. Any experience with that?

I found it difficult to find a good source on VGA passthrough supported graphics adapters. The Xen wiki pages http://wiki.xen.org/wiki/Xen_VGA_Passthrough and http://wiki.xen.org/wiki/Xen_VGA_Passthrough_Tested_Adapters seem pretty much outdated or incomplete. I would like to compile a list of graphics adapters have been successfully passed through as either primary or secondary adapters, with notes on the Xen hypervisor and kernel used. I will add a link here to the new thread. Help is greatly appreciated.

Here is the link to the VGA passthrough enabled graphics adapters list: http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters. Please post your successfully passed through adapters there.


----------



## dizzy4

Overprovisioning CPUs is really not a good idea. I tried it with a domU running win7 and then I booted my win8 preview. I gave each of them 5 CPUs and the result was horrendous. With any load on the win8 domU simple tasks like streaming video from netflix to the win7 domU were choppy. Not to my surprise dom0 was also very slow and almost unresponsive. I tried assigning the dom0 its own cpu thread not used by either domU. It helped some, but the domU systems were still somewhat glitchy. Remember these are HVM domUs so you are actually assigning physical threads from your CPU. However, with only one HVM domU I assume there could be one or many other paravirtual domUs existing.

On to memory... Do not overprovision or balloon it! Errors will occur in your domU and your dom0. Once again I think this is something to be reserved for paravirtual machines. (I have seen it used in Xen VPS node setups)

I would stick to HD radeon cards for now. I think they patched it for nvidia but I can't confirm it until later. If not I will attempt to patch it for you nvidia guys.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Overprovisioning CPUs is really not a good idea. I tried it with a domU running win7 and then I booted my win8 preview. I gave each of them 5 CPUs and the result was horrendous. With any load on the win8 domU simple tasks like streaming video from netflix to the win7 domU were choppy. Not to my surprise dom0 was also very slow and almost unresponsive. I tried assigning the dom0 its own cpu thread not used by either domU. It helped some, but the domU systems were still somewhat glitchy. Remember these are HVM domUs so you are actually assigning physical threads from your CPU. However, with only one HVM domU I assume there could be one or many other paravirtual domUs existing.
> On to memory... Do not overprovision or balloon it! Errors will occur in your domU and your dom0. Once again I think this is something to be reserved for paravirtual machines. (I have seen it used in Xen VPS node setups)
> I would stick to HD radeon cards for now. I think they patched it for nvidia but I can't confirm it until later. If not I will attempt to patch it for you nvidia guys.


Wow, thanks for the speedy reply.

CPU over-provisioning: I didn't know that HVM guests would get physical cores or threads. I've also never experimented with PV guests versus HVM guests, or a mix of it. Well, I assume my Linux dom0 is sort of a PV guest, isn't it? I should do some experimenting to see how multiple PV or HVM guests work together.
Have you tried increasing the dom0 CPU priority? I think it's the queue priority for CPU time. I've increased my dom0 priority but don't notice any difference, but then I never had any issue with starving CPU resources (I use 2 threads for dom0 when running the Windows HVM guest with 10 threads). The nice thing is that when closing Windows I get the full 12 threads for my dom0 (at least that's what my system monitor shows me - never done any benchmarking though).

Memory: Thanks for the advise. I will disable ballooning now. With 32GB memory I don't see much need for it anyway.

Graphics cards: I've opened a thread under the graphics cards hardware section (see link in previous post). Would you be able to share your experience with specific AMD cards? I know that AMD has generally better support for VGA passthrough, but I guess not all AMD cards are equal. I've posted the specs for my Nvidia Quadro 2000 card which works just perfect under the Windows HVM guest, using the latest Nvidia Windows driver for the card. See other thread for details.


----------



## shadman

I've got a 3930k c2, but a GTX680. I didn't know this existed until now, butonce this picks up a little again, I'm interested









Subbed.


----------



## powerhouse

Here is some interesting reading on how Xen manages CPU resources, with links to further information: http://www.techinferno.com/2012/07/05/xen-and-debian-an-introduction-to-virtualization/9/. Scroll down.

The above link is a tutorial for using Debian with Xen PV and HVM guests. Lot's of information and scripts (pciback script), and easy to read.


----------



## powerhouse

Quote:


> Originally Posted by *shadman*
> 
> I've got a 3930k c2, but a GTX680. I didn't know this existed until now, butonce this picks up a little again, I'm interested
> 
> 
> 
> 
> 
> 
> 
> 
> Subbed.


I guess the GTX680 won't be easy. See http://lists.xen.org/archives/html/xen-users/2012-07/msg00195.html for problems relating to the large VGA BIOS size.

David Techer in this thread has a how-to on compiling Xen hypervisor with video BIOS and patches for Nvidia cards - see http://www.davidgis.fr:8080/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through.

But compiling your own Xen hypervisor and kernel isn't for everyone - I've failed to get my previous Quadro 600 to work with a compiled Xen hypervisor., patches and video BIOS. As to Nvidia cards, as far as I know only the "multi-OS" cards are good to go without any patches etc. My Quadro 2000 worked out of the box - no compilation, no patches, just vanilla 3.2.0-30-generic kernel (Linux Mint 13 Mate 64bit / Ubuntu 12.04) and Xen hypervisor 4.1.2.

The OP and many others suggest to try with AMD HD video cards. Check this thread for more information on supported AMD cards, perhaps you want to give it a try? I've started a new thread to list VGA passthrough compatible graphics cards here: http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters, but need input from the community to make it a usable resource.

Also, check to see that your motherboard supports VT-d. Your CPU should do great (I got the same and it rocks)!


----------



## Killhouse

I've been doing a lot of thinking and realised a fatal flaw in my plan. Even if I manage to boot a domU and get it to take my active graphics card - to get my GPU hot-plugging in and out of dom0 and domU as I switch between the two VMs.... would be rather tricky! Probably impossible, in fact.


----------



## drbaltazar

microsoft azure FTWall hailMICROSOFT!!!(sorry!couldnt resist)


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> I've been doing a lot of thinking and realised a fatal flaw in my plan. Even if I manage to boot a domU and get it to take my active graphics card - to get my GPU hot-plugging in and out of dom0 and domU as I switch between the two VMs.... would be rather tricky! Probably impossible, in fact.


A second graphics card helps. I use a simple AMD card (6450 - though I would recommend a better/stronger model) as primary card for dom0 Linux and my Quadro 2000 for the domU Windows guest. Both cards are connected via DVI cable to two ports on my screen. My screen even detects a new DVI input signal and automatically switches to that port







. You will find a KVM switch helpful for switching your USB keyboard/mouse between Linux dom0 and Windows domU (passed through USB host).


----------



## powerhouse

Xen 4.2 just released (September 17, 2012) !!!

See here for more info: http://wiki.xen.org/wiki/Xen_4.2_Release_Notes

Xen 4.2 is supposed to bring some improvements for VGA passthrough, along with the new xl toolstack and better CPU management / scheduling. I hope to see Xen 4.2 in the repository soon.

Caveats: Xen 4.2 uses xl by default, but has xm (xend) support for backward compatibility. I hope the transition to xl will not be too painful. Need to get my hands onto this.


----------



## Killhouse

The issue will remain that I will always want my best hardware in the VM I currently have full-screened. I basically want to run two operating systems at once, sharing resources (CPU). One of them will always be headless and I want to switch heads between the two.

Im pretty sure what I want to achieve is impossible!

Edit:

Unless I can create a script that will hotplug out VGA from Linux, switch to VM and then hotplug in VGA to Windows - and vice versa... probably impossible.


----------



## wTheOnew

Quote:


> Originally Posted by *powerhouse*
> 
> Xen 4.2 just released (September 17, 2012) !!!
> See here for more info: http://wiki.xen.org/wiki/Xen_4.2_Release_Notes
> Xen 4.2 is supposed to bring some improvements for VGA passthrough, along with the new xl toolstack and better CPU management / scheduling. I hope to see Xen 4.2 in the repository soon.
> Caveats: Xen 4.2 uses xl by default, but has xm (xend) support for backward compatibility. I hope the transition to xl will not be too painful. Need to get my hands onto this.


Awesome, thanks.

I played around with the early builds of 4.2 and could never really get it working right. Will have to try it out again...just as I got my system running perfectly.


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> Xen 4.2 just released (September 17, 2012) !!!
> See here for more info: http://wiki.xen.org/wiki/Xen_4.2_Release_Notes
> Xen 4.2 is supposed to bring some improvements for VGA passthrough, along with the new xl toolstack and better CPU management / scheduling. I hope to see Xen 4.2 in the repository soon.
> Caveats: Xen 4.2 uses xl by default, but has xm (xend) support for backward compatibility. I hope the transition to xl will not be too painful. Need to get my hands onto this.


Great news!
Quote:


> Originally Posted by *Killhouse*
> 
> I've been doing a lot of thinking and realised a fatal flaw in my plan. Even if I manage to boot a domU and get it to take my active graphics card - to get my GPU hot-plugging in and out of dom0 and domU as I switch between the two VMs.... would be rather tricky! Probably impossible, in fact.


I am going to include some scripts in my next release, so I will try to make one to accomplish this too. It shouldn't be too hard.

In fact, I am going to run out to fry's to get a cheap SSD to do a fresh install. I don't want to mess up a good thing (my working VM setup







)


----------



## Killhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Great news!
> I am going to include some scripts in my next release, so I will try to make one to accomplish this too. It shouldn't be too hard.
> In fact, I am going to run out to fry's to get a cheap SSD to do a fresh install. I don't want to mess up a good thing (my working VM setup
> 
> 
> 
> 
> 
> 
> 
> )


I look forward to this!

Im back to W7 using XMing now, but I will watch out for this and I have a 30gb partition ready for when I want to mess around with Xen again


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> I look forward to this!
> Im back to W7 using XMing now, but I will watch out for this and I have a 30gb partition ready for when I want to mess around with Xen again


Hello Killhouse: I know you gave it a try. Can you tell me/us where you got stuck? And which graphic card you are trying to pass-through (you probably mentioned it before)?

I sincerely hope the new hypervisor 4.2 will make things a little easier.


----------



## Killhouse

I needed to try a little harder with pciback. But I probably should have not passed through the graphics card immediately anyway. I needed to bring my pc back to functional for the weekend, so I'm back on W7 but will keep watching this thread as I plan to do hypervisor VMs soon.

Currently I am using W7 and running Ubuntu on VMware with the disc image on a RAMDisk. With OpenGL passthrough and Unity (VMware unity, not Ubuntu unity), its pretty sweet...


----------



## powerhouse

Quote:


> Originally Posted by *Killhouse*
> 
> I needed to try a little harder with pciback. But I probably should have not passed through the graphics card immediately anyway. I needed to bring my pc back to functional for the weekend, so I'm back on W7 but will keep watching this thread as I plan to do hypervisor VMs soon.
> Currently I am using W7 and running Ubuntu on VMware with the disc image on a RAMDisk. With OpenGL passthrough and Unity (VMware unity, not Ubuntu unity), its pretty sweet...


Sounds like a nice setup.

As I said before, a second graphics adapter helps a lot. With regard to Linux performance on "cheaper" graphics adapters, I've been running Linux Mint with compiz and all the graphical bells and whistles on a 5 year old passively cooled Nvidia card using the proprietary nvidia driver and never had any performance issues. If you run graphics intensive games under Windows in the domU (with VGA passthrough) using your power graphics card, a less powerful card could be more than good enough for Linux. Unfortunately the proprietary Nvidia drivers don't work under Xen, at least not 4.1.2. I read somewhere that it's possible to get them working, but it would probably require kernel compilation etc. I haven't given it a real trial, though.

There should be good AMD cards too. Just don't get the cheapo AMD 6450 - it really sucks!


----------



## nyxcharon

I was wanting to try this, and I wanted some suggestions.
Should i get the
i5-2400
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115074
or the
i5-3570
http://www.newegg.com/Product/Product.aspx?Item=N82E16819115233

and for motherboard, the
GIGABYTE GA-Z77MX-D3H
http://www.newegg.com/Product/Product.aspx?Item=N82E16813128542
or
ASRock Z77 Pro3
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157297

Cpu wise i'd just like to save the money if i can, and motherboard wise not sure which is the better choice. I'll be buying a 7950 for passthru since my gtx 560 doesn't want to work with it. (already tried on my current system to see if it would work)


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> I was wanting to try this, and I wanted some suggestions.
> Should i get the
> i5-2400
> http://www.newegg.com/Product/Product.aspx?Item=N82E16819115074
> or the
> i5-3570
> http://www.newegg.com/Product/Product.aspx?Item=N82E16819115233
> and for motherboard, the
> GIGABYTE GA-Z77MX-D3H
> http://www.newegg.com/Product/Product.aspx?Item=N82E16813128542
> or
> ASRock Z77 Pro3
> http://www.newegg.com/Product/Product.aspx?Item=N82E16813157297
> Cpu wise i'd just like to save the money if i can, and motherboard wise not sure which is the better choice. I'll be buying a 7950 for passthru since my gtx 560 doesn't want to work with it. (already tried on my current system to see if it would work)


Re CPU: I think both should do fine. Check also that the HD 2000 or HD 2500 onboard GPU are supported - I believe they are. See also http://intellinuxgraphics.org/.

Remark: You won't save much if anything buying the cheaper i5 2400 versus the i5 3570. The 2400 has a max TDP of 95W, the 3570 a max TDP of 77W. If you run your system 24/7, you pay the difference in your electricity bill probably within a year or two.

According to http://wiki.xen.org/wiki/VTd_HowTo both boards are supported, though the Gigabyte is only mentioned with rev1. Search for user reports on PCI / VGA passthrough with these boards to make sure they work (also, check which BIOS release is needed).

Sorry I can't help more, as I haven't used either of them. Perhaps someone here can be more specific.


----------



## powerhouse

For those still reading this thread, I did some "benchmarking" under both Linux dom0 and Windows domU. The results are posted on this thread: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 - see post 12 at bottom.

To sum it up: When both dom0 and domU are under load, domU uses up to the maximum CPU resources (vpcu=10 for a 12 thread CPU) specified in its config file. Spare CPU time is used by dom0. In other words, when my Windows guest is idle or under partial load, dom0 Linux will use the available CPU time when needed.

Code:



Code:


sudo xm top

shows me the CPU time as % given to both dom0 and domU.

CPU usage under WIndows seems to have been limited by IO performance (reading 16-20MB files and writing back 200-300K files to the same virtual drive residing on normal HD drives/not SSD). domU (Windows) gave about the same performance under load, with or without putting CPU and IO load on the dom0. While running a CPU intensive application (ripping a DVD) under dom0, it would dynamically use whatever CPU resources were available at any given time, up to the full 12 vcpus when the Windows domU was idle.

The only bottleneck I still have is network throughput in both dom0 and domU, when copying files from a local drive to a remote PC using Samba. Also, VNC sometimes gets stuck when connecting from dom0 (or remote PC) to domU, in which case I have to reconnect. My PCs are connected via a Gigabit WiFi router that also offers 150Mb/s over wireless, so there should be no bottleneck. My Xen machine has only one Gigabit Ethernet port, and I use a virtual bridge for dom0 and domU networking.

If someone has an idea on how to improve network speed (currently 4.2MB/s for file transfer between Samba shares), I'd appreciate your advise!


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> For those still reading this thread, I did some "benchmarking" under both Linux dom0 and Windows domU. The results are posted on this thread: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 - see post 12 at bottom.
> To sum it up: When both dom0 and domU are under load, domU uses up to the maximum CPU resources (vpcu=10 for a 12 thread CPU) specified in its config file. Spare CPU time is used by dom0. In other words, when my Windows guest is idle or under partial load, dom0 Linux will use the available CPU time when needed.
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> sudo xm top
> 
> shows me the CPU time as % given to both dom0 and domU.
> CPU usage under WIndows seems to have been limited by IO performance (reading 16-20MB files and writing back 200-300K files to the same virtual drive residing on normal HD drives/not SSD). domU (Windows) gave about the same performance under load, with or without putting CPU and IO load on the dom0. While running a CPU intensive application (ripping a DVD) under dom0, it would dynamically use whatever CPU resources were available at any given time, up to the full 12 vcpus when the Windows domU was idle.
> The only bottleneck I still have is network throughput in both dom0 and domU, when copying files from a local drive to a remote PC using Samba. Also, VNC sometimes gets stuck when connecting from dom0 (or remote PC) to domU, in which case I have to reconnect. My PCs are connected via a Gigabit WiFi router that also offers 150Mb/s over wireless, so there should be no bottleneck. My Xen machine has only one Gigabit Ethernet port, and I use a virtual bridge for dom0 and domU networking.
> If someone has an idea on how to improve network speed (currently 4.2MB/s for file transfer between Samba shares), I'd appreciate your advise!


Nice results! The issues I was encountering were when 2 domUs were booted that share physical resources. Ideally with Xen you don't want dom0 to do anything except manage the domUs it is hosting. I have kept dom0 as a pretty bloated OS because it is how I got it to work. I am working on getting something like centOS stable. From there it should be possible to boot a PV or HVM domU for windows and one for Linux. Essentially the resources would be split between them all, but in your case you could assign 5 cores to each domU while letting the dom0 have 2 (or even one if you wanted).

As for the networking -- I had some issues too. Originally I had each virtual interface connected to a NAT, but that limited ports or forced a lot of translation. It was really inefficient, so I used brctl to manage my network and disabled network manager. Basically each vif gets assigned its own address on the network directly. It seemed to work well for me and essentially turned dom0 into a switch. My performance seems very fast, but I don't do a lot of samba or other network transfers. Internet speeds are not compromised in any way, and I have seen my speeds hit 3 MB/s download. (24 mbit). I would be happy to do some more testing once I get it stable again. It also gives me another issue to monitor.

Thanks as always!


----------



## Killhouse

Quote:


> Originally Posted by *powerhouse*
> 
> Sounds like a nice setup.


You could say that!


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Nice results! The issues I was encountering were when 2 domUs were booted that share physical resources. Ideally with Xen you don't want dom0 to do anything except manage the domUs it is hosting. I have kept dom0 as a pretty bloated OS because it is how I got it to work. I am working on getting something like centOS stable. From there it should be possible to boot a PV or HVM domU for windows and one for Linux. Essentially the resources would be split between them all, but in your case you could assign 5 cores to each domU while letting the dom0 have 2 (or even one if you wanted).
> As for the networking -- I had some issues too. Originally I had each virtual interface connected to a NAT, but that limited ports or forced a lot of translation. It was really inefficient, so I used brctl to manage my network and disabled network manager. Basically each vif gets assigned its own address on the network directly. It seemed to work well for me and essentially turned dom0 into a switch. My performance seems very fast, but I don't do a lot of samba or other network transfers. Internet speeds are not compromised in any way, and I have seen my speeds hit 3 MB/s download. (24 mbit). I would be happy to do some more testing once I get it stable again. It also gives me another issue to monitor.
> Thanks as always!


Hello dizzy4: Thanks for the reply and suggestions.

Originally I was thinking along the same lines as you with regard to a thin hypervisor / dom0 (without desktop), and Linux and Windows in PV respectively HVM domUs. But like you I wound up with a working system using only dom0 Linux and domU.

Given the results of my tests and the performance I get, there is little I would want to change. Essentially I need all my CPU etc. resources under Windows, or as much as I can give it without creating a bottleneck in dom0. Occasionally my Linux Mint dom0 needs extra CPU resources, like when ripping DVDs or CDs for my media center. I use handbrake for DVD ripping (conversion/encoding) and it works real nice. It can also use all the CPU resources it can get.

In my setup with dom0 Linux and domU Windows, dom0 is able to grab free CPU time while my Windows guest isn't too busy. I wonder how that would work with a slim dom0 and two domU, where both fight about CPU time? I'm not sure one can "over-provision" vcpus to both domU, like giving each one 10 vcpus and let them fight it out. Perhaps it works in conjunction with dom0 vcpu pinning, where I would reserve 2 vcpus for dom0?

With regard to network speed, I also use brctl (actually the /etc/network/interfaces file). Internet speed in both dom0 and domU are perfect - couldn't complain. Only file transfers between Samba shares are slow, as well as the occasional (or frequent) VNC/RDP issue. I haven't tested file transfers between domU and dom0 via Samba shares, but I expect to get similar results. I uninstalled network-manager as it gave me troubles, and configured a /etc/resolve.conf file with the DNS entries.

Here is my /etc/network/interfaces configuration:

Code:



Code:


auto lo
iface lo inet loopback

auto xenbr0

# I use a static IP for my NIC; you can also use DHCP:
iface xenbr0 inet static
bridge_ports eth0
address 192.168.0.120
broadcast 192.168.0.255
netmask 255.255.255.0
gateway 192.168.0.1
bridge_stp off   # disable Spanning Tree Protocol
bridge_waitport 0       # no delay before a port becomes available
bridge_fd 0      # no forwarding delay

# There are no IP configurations for eth0, it's all defined under xenbr0:
auto eth0


----------



## powerhouse

Network issues: I did some further investigations on the slow Samba performance.

Ran

Code:



Code:


ifconfig

and saw some errors on eth0.

Then captured some Ethernet traffic while running a Samba file transfer:

Code:



Code:


tcpdump -nvvi eth0

It turns out I get checksum errors.

I tried the following suggestions:
http://www1.shorewall.net/XenMyWay.html using

Code:



Code:


ethtool -K eth0 tx off

but that didn't do it







.

Then I tried these suggestions:
http://www.tuxradar.com/answers/488
to disable the offload. It didn't work either. In fact, it caused my remote desktop connection to Windows domU go real slow!

I didn't get any further with this, but at least I'm onto something.

P.S.: I solved my remote desktop issues from Macbook to Windows 7 domU by using CoRD, creating a server entry and using default values. Works great now and I can even watch VLC video running on Windows through the remote desktop. A windowed view is almost as good as streaming, just full frame is a little choppy. Don't try the Microsoft RDP utility on Mac - it's totally useless and doesn't give me any working connection.


----------



## nyxcharon

Been trying to get this setup with my current rig before i upgrade my motherboard and cpu, and been running into some issues.
sidenote: I also just got a 7850 since I saw so many people have problems with nvidia cards.

Anyways, i started in debian and was able to get everything working (setup win7, etc) up to the point of attaching the card. That just wont work for me.
So i just installed fedora and after installing xen and rebooting, i just get a black screen. I've tried a couple different things such as hiding the amd card with xen.pciback-hide or whatever the command is for grub with no luck.

Anyone have any suggestions?
Ideally i'd like to disable my amd card, and then attach it to a win7 vm. I have two monitors, and at the moment I have one plugged into the onboard graphics and the other my card. I was hoping I could hide the amd card, setup the vm on one monitor then run it on the other monitor (that's plugged into the amd card)

EDIT:
Got it almost there! I ended up installing Linux Mint 13 and got everything setup. I can attach my graphics card, and it shows up under xm list-assignable-devices. However, when i try to start the vm with it attached, i get this error:
Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err "failed to assign device: maybe the platform doesn't support VT-d, or VT-d isn't enabled properly?")

My e8400 and mobo bother support vt-d, and i double checked my bios. So what else am i missing?

Code:



Code:


[email protected] /etc/xen $ sudo xm pci-list Windows7
Device
0000:01:00.0
0000:01:00.1

Code:



Code:


[email protected] /etc/xen $ lspci
00:00.0 Host bridge: Intel Corporation 4 Series Chipset DRAM Controller (rev 03)
00:01.0 PCI bridge: Intel Corporation 4 Series Chipset PCI Express Root Port (rev 03)
00:1b.0 Audio device: Intel Corporation N10/ICH 7 Family High Definition Audio Controller (rev 01)
00:1c.0 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 1 (rev 01)
00:1c.1 PCI bridge: Intel Corporation N10/ICH 7 Family PCI Express Port 2 (rev 01)
00:1d.0 USB controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #1 (rev 01)
00:1d.1 USB controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #2 (rev 01)
00:1d.2 USB controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #3 (rev 01)
00:1d.3 USB controller: Intel Corporation N10/ICH 7 Family USB UHCI Controller #4 (rev 01)
00:1d.7 USB controller: Intel Corporation N10/ICH 7 Family USB2 EHCI Controller (rev 01)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev e1)
00:1f.0 ISA bridge: Intel Corporation 82801GB/GR (ICH7 Family) LPC Interface Bridge (rev 01)
00:1f.1 IDE interface: Intel Corporation 82801G (ICH7 Family) IDE Controller (rev 01)
00:1f.2 IDE interface: Intel Corporation N10/ICH7 Family SATA Controller [IDE mode] (rev 01)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Device 6819
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Device aab0
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
03:00.0 Multimedia video controller: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb (rev 04)
04:01.0 RAID bus controller: VIA Technologies, Inc. VT6421 IDE/SATA Controller (rev 50)

Code:



Code:


ny[email protected] /etc/xen $ cat /etc/default/grub 
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
#GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
GRUB_CMDLINE_LINUX=""

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M xen-pciback.hide=(01:00.0) (01:00.0)"


----------



## wTheOnew

^^^ Run 'xm dmesg' and post the results.


----------



## nyxcharon

Code:



Code:


[email protected] ~ $ sudo xm dmesg
[sudo] password for nyxcharon: 
(XEN) Xen version 4.1.2 (Ubuntu 4.1.2-2ubuntu2.1) ([email protected]) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) Mon Jun 18 14:13:52 UTC 2012
(XEN) Bootloader: GRUB 1.99-21ubuntu3
(XEN) Command line: placeholder iommu=1 dom0_mem=2048M xen-pciback.hide=(01:00.0) (01:00.0)
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 4 MBR signatures
(XEN)  Found 4 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009fc00 (usable)
(XEN)  000000000009fc00 - 00000000000a0000 (reserved)
(XEN)  00000000000e4000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 00000000dff80000 (usable)
(XEN)  00000000dff80000 - 00000000dff8e000 (ACPI data)
(XEN)  00000000dff8e000 - 00000000dffd0000 (ACPI NVS)
(XEN)  00000000dffd0000 - 00000000e0000000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000fff00000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 0000000120000000 (usable)
(XEN) ACPI: RSDP 000FB760, 0024 (r2 ACPIAM)
(XEN) ACPI: XSDT DFF80100, 005C (r1 A_M_I_ OEMXSDT  10001121 MSFT       97)
(XEN) ACPI: FACP DFF80290, 00F4 (r3 A_M_I_ OEMFACP  10001121 MSFT       97)
(XEN) ACPI: DSDT DFF80440, 8794 (r1  A1745 A1745000        0 INTL 20060113)
(XEN) ACPI: FACS DFF8E000, 0040
(XEN) ACPI: APIC DFF80390, 006C (r1 A_M_I_ OEMAPIC  10001121 MSFT       97)
(XEN) ACPI: MCFG DFF80400, 003C (r1 A_M_I_ OEMMCFG  10001121 MSFT       97)
(XEN) ACPI: OEMB DFF8E040, 0089 (r1 A_M_I_ AMI_OEM  10001121 MSFT       97)
(XEN) ACPI: HPET DFF88BE0, 0038 (r1 A_M_I_ OEMHPET  10001121 MSFT       97)
(XEN) ACPI: GSCI DFF8E0D0, 2024 (r1 A_M_I_ GMCHSCI  10001121 MSFT       97)
(XEN) ACPI: SSDT DFF90510, 0363 (r1 DpgPmm    CpuPm       12 INTL 20060113)
(XEN) System RAM: 4095MB (4193404kB)
(XEN) Domain heap initialised
(XEN) Processor #0 7:7 APIC version 20
(XEN) Processor #1 7:7 APIC version 20
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Table is not found!
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3435.411 MHz processor.
(XEN) Initing memory sharing.
(XEN) I/O virtualisation disabled
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using new ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN) HVM: ASIDs disabled.
(XEN) HVM: VMX enabled
(XEN) Brought up 2 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x205f000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000114000000->0000000118000000 (498221 pages to be allocated)
(XEN)  Init. ramdisk: 000000011da2d000->000000011ffff800
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff8205f000
(XEN)  Init. ramdisk: ffffffff8205f000->ffffffff84631800
(XEN)  Phys-Mach map: ffffffff84632000->ffffffff84a32000
(XEN)  Start info:    ffffffff84a32000->ffffffff84a324b4
(XEN)  Page tables:   ffffffff84a33000->ffffffff84a5c000
(XEN)  Boot stack:    ffffffff84a5c000->ffffffff84a5d000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84c00000
(XEN)  ENTRY ADDRESS: ffffffff81cfb200
(XEN) Dom0 has maximum 2 VCPUs
(XEN) Scrubbing Free RAM: ...................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 220kB init memory.
(XEN) physdev.c:155: dom0: wrong map_pirq type 3


----------



## dizzy4

Quote:


> Originally Posted by *nyxcharon*
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> [email protected] ~ $ sudo xm dmesg
> [sudo] password for nyxcharon:
> (XEN) Xen version 4.1.2 (Ubuntu 4.1.2-2ubuntu2.1) ([email protected]) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) Mon Jun 18 14:13:52 UTC 2012
> (XEN) Bootloader: GRUB 1.99-21ubuntu3
> (XEN) Command line: placeholder iommu=1 dom0_mem=2048M xen-pciback.hide=(01:00.0) (01:00.0)
> (XEN) I/O virtualisation disabled


Xen isn't enabling VT-d for some reason. There should be some lines that say Intel VT-d. Just go double and tripple check everything to make sure VT-d is on.

Code:



Code:


(XEN) Intel VT-d supported page sizes: 4kb.
...

On another note, I have been playing around with Xen4.2. I am using CentOS 6.3 for my dom0 and it is working fairly well (I guess). VT-d is enabled and hot swapping is going to be amazing. It is so easy with the new XL toolstack, but the other operations of XL feel foreign to me. I am struggling trying to get it to work properly still, but I have made progress.

Code:



Code:


xl list

shows my domain running, but no luck getting anything to display. The configurations have changed and there are some issues with the binaries I found for CentOS 6.3 too. I don't think all the new qemu code was included, so I don't think the guests are running properly.

Progress is slow, but I think the iommu portions of Xen 4.2 are already working. The issues I am working on have more to do with the new toolstack and the configuration files. I am very hopeful that this next version will be a low-resource usb-stick guide that will have automatic hot-swapping! (eventually).

I will post what I have done tomorrow and maybe someone would be willing to help me test


----------



## TheDave

Right on! Good luck with it.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Been trying to get this setup with my current rig before i upgrade my motherboard and cpu, and been running into some issues.
> sidenote: I also just got a 7850 since I saw so many people have problems with nvidia cards.
> ...
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> [email protected] /etc/xen $ cat /etc/default/grub
> # If you change this file, run 'update-grub' afterwards to update
> # /boot/grub/grub.cfg.
> # For full documentation of the options in this file, see:
> #   info -f grub -n 'Simple configuration'
> GRUB_DEFAULT=0
> #GRUB_HIDDEN_TIMEOUT=0
> GRUB_HIDDEN_TIMEOUT_QUIET=true
> GRUB_TIMEOUT=10
> GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
> GRUB_CMDLINE_LINUX_DEFAULT="quiet splash"
> GRUB_CMDLINE_LINUX=""
> # Uncomment to enable BadRAM filtering, modify to suit your needs
> # This works with Linux (no patch required) and with any kernel that obtains
> # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
> #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
> # Uncomment to disable graphical terminal (grub-pc only)
> #GRUB_TERMINAL=console
> # The resolution used on graphical terminal
> # note that you can use only modes which your graphic card supports via VBE
> # you can see them in real GRUB with the command `vbeinfo'
> #GRUB_GFXMODE=640x480
> # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
> #GRUB_DISABLE_LINUX_UUID=true
> # Uncomment to disable generation of recovery mode menu entries
> #GRUB_DISABLE_RECOVERY="true"
> # Uncomment to get a beep at grub start
> #GRUB_INIT_TUNE="480 440 1"
> GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M xen-pciback.hide=(01:00.0) (01:00.0)"


Your grub line is wrong:

Code:



Code:


GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M xen-pciback.hide=(01:00.0) (01:00.0)"

should read:

Code:



Code:


GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M xen-pciback.hide=(01:00.0) (01:00.1)"

Did you run

Code:



Code:


update-grub

after changing the above default/grub file?

Your real problem lies in the xm dmesg output. You got:

Code:



Code:


(XEN) Initing memory sharing.
(XEN) I/O virtualisation disabled

What you should get is something like this:

Code:



Code:


(XEN) Initing memory sharing.
(XEN) Intel VT-d Snoop Control enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled

If you haven't run update-grub after modifying the /etc/default/grub file, that would explain everything.

EDIT: You may find some specific Linux Mint 13 stuff in my how-to: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013


----------



## powerhouse

dizzy4: Good luck with 4.2 and xl. I look forward to reading more about your experiences with that.


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> If you haven't run update-grub after modifying the /etc/default/grub file, that would explain everything.
> EDIT: You may find some specific Linux Mint 13 stuff in my how-to: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013


Fixed my grub line and updated grub.
Still get the

Code:



Code:


(XEN) Initing memory sharing.
(XEN) I/O virtualisation disabled

And yeah, your guide is actually the one i was following to get this far. Not sure what the issue might be, though kinda suspecting my motherboard at this point.


----------



## wTheOnew

Quote:


> Originally Posted by *nyxcharon*
> 
> Fixed my grub line and updated grub.
> Still get the
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> (XEN) Initing memory sharing.
> (XEN) I/O virtualisation disabled
> 
> And yeah, your guide is actually the one i was following to get this far. Not sure what the issue might be, though kinda suspecting my motherboard at this point.


fwiw, from the xen wiki:
Quote:


> Unfortunately there are many buggy BIOSes causing Xen to disable IO virtualization because of errors in the BIOS DMAR/ACPI tables. Xen tries to workaround these bugs in the BIOS, but sometimes it's not possible. Please report all the details about your hardware and software to xen-devel mailinglist if IO virtualization gets disabled due to buggy BIOS. Also see below for troubleshooting tips.
> 
> Add "iommu=verbose" option for Xen hypervisor (xen.gz) in grub.conf and reboot. After rebooting read "xm dmesg" log (or set up a serial console). As a default Xen 4.0.0 is not verbose about IOMMU initialization and related ACPI DMAR table parsing.


What motherboard do you have?


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Fixed my grub line and updated grub.
> Still get the
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> (XEN) Initing memory sharing.
> (XEN) I/O virtualisation disabled
> 
> And yeah, your guide is actually the one i was following to get this far. Not sure what the issue might be, though kinda suspecting my motherboard at this point.


Yeah, it might be the motherboard. Can you specify the motherboard model and BIOS release?

Sometimes a newer BIOS release fixes things. Have a look here http://wiki.xen.org/wiki/VTd_HowTo to see if you find anything useful. Look at the end for grub boot options:

workaround_bios_bug
iommu_inclusive_mapping=1

I have no experience with these command line options, but it may be worth trying.

In any case, start with checking the motherboard BIOS. If there is a newer BIOS, read the manufacturer's release notes to see if VT-d is mentioned. Following that you may want to try grub boot options.

Good luck!


----------



## nyxcharon

Quote:


> Originally Posted by *wTheOnew*
> 
> What motherboard do you have?


The same one in my sig:
ASUS P5G41T-M LX PLUS


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> The same one in my sig:
> ASUS P5G41T-M LX PLUS


Hello nyxcharon:

I had a look at the user manual for your MB. It doesn't loose a word on VT-d or IOMMU.

VT-d is *not* VT-x or "Intel Virtualization Technology". If you haven't got a specific VT-d or IOMMU option in your BIOS, then your board will most likely not support VT-d.

Please check the link in my previous post. Here some excerpt:


Spoiler: Warning: Spoiler!



Notes on VT-d compatibility:

VT-d is enabled on the following chipsets:
Intel Q35 (desktop / workstation)
Intel Q45 (desktop / workstation)
Intel X58 (desktop / workstation)
Intel 55x0 (server)
Intel 3450 (workstation / server)

The following chipsets have VT-d capability in theory, but most OEMs (such as Asus and Gigabyte) do not have it enabled on boards based on these:

Intel X38 (desktop / workstation)
Intel X48 (desktop / workstation)
Intel 32x0 (server)

For Intel Desktop Boards, these have VT-d support enabled:

Intel DQ35JO
Intel DQ35MP
Intel DX38BT
Intel DX48BT2
Intel DQ45CB (BIOS 0061 required, previous versions are known to cause problems)
Intel DQ45EK
Intel DX58SO
Intel DQ67SW

For ASUS Desktop Boards, these have VT-d support enabled, but Asus does NOT support Linux, so you are on your own with any Linux or Xen issues like broken BIOSes:

ASUS P5E-VM DO (Intel Q35 chipset) requires IGD to be enabled (otherwise DMAR-table becomes corrupted)
ASUS P6T Deluxe (Intel X58 chipset) requires (currently non-public) BIOS update to correct DMAR-table issue
ASUS P6T6 WS Revolution (Intel X58 chipset) incorrect bios DMAR-table
ASUS Sabertooth X58 (Intel X58 chipset) incorrect bios DMAR-table (2nd RMRR structure is incorrect)
ASUS Z8NA-D6 (dual processor nehalem board) works

Most server boards based on the Tylersburg chipset (55x0) and few boards based on 32x0 should have working VT-d, known examples are:

Intel server board: S3210SHLX (BIOS >R0044 required.)
Supermicro server mainboard: X8DT3-F
Supermicro server 5026T-TB: [1] Only work USB 1.1.

These motherboards are known to have broken BIOS preventing IO virtualization (VT-d IOMMU) from working:

Supermicro X7SB4 (with official BIOS 1.2a) has broken ACPI DMAR table with zero length entries. BIOS version 1.3 Beta fixes the problem.
Samsung X460 laptop: BIOS doesn't provide DMAR table so VT-d cannot be used.

As far as we know, following OEM systems also have VT-d enabled. Feel free to add others as they become available.

Dell: Optiplex 755, Optiplex 980
HP Compaq: DC7800
Fujitsu-Siemens: Esprimo 5925
Lenovo: ThinkStation E20 and Thinkserver TS200v

VT-d Compatible ASRock Motherboards (Socket 1155) probably all Z68 and Z77 board's

ASRock Z68 Extreme4 Gen3
ASRock Z68 Professional Gen3
ASRock Z77 Pro3
ASRock Z77 Extreme4
ASRock Z77 Extreme6
ASRock Z77 Professional

Gigabyte Motherboards (Socket 1155) suggested by manufacturer on inquiry

GA-X79-UD5 (rev. 1.0)
GA-X79-UD7 (rev. 1.0)
GA-Z77X-D3H (rev. 1.0)

MSI 1155 Motherboards (Socket 1155) suggested by manufacturer on inquiry

Z77A-GD80
Z77A-GD65
Z77A-GD55
Z77A-G45
Z77A-G43



I didn't see your chipset (G41) listed.

Again, virtualization support or VT-x is nice to have, but does nothing to enable PCI or VGA passthrough.

Your motherboard must support VT-d (for Intel CPU based boards), more correctly named IOMMU. It's an I/O memory mapping feature.

Perhaps you can get a cheap (or free?) second hand board that does support VT-d? Refer to the link (or list) above.

Sorry for the bad news.


----------



## nyxcharon

Yeah, I was starting to think that was the issue.
I planned on upgrading to a i5 and asrock mobo anyway, I just wanted to get it working before I went through with it to make sure i got it working. It's ready to go now, minus the whole motherboard support. So I guess I'll be taking the plunge and upgrading so I can run xen.









I doubt my C2D would have enough power to run a gaming VM anyway. It was more of a proof of concept thing.









Thanks for the help


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Yeah, I was starting to think that was the issue.
> I planned on upgrading to a i5 and asrock mobo anyway, I just wanted to get it working before I went through with it to make sure i got it working. It's ready to go now, minus the whole motherboard support. So I guess I'll be taking the plunge and upgrading so I can run xen.
> 
> 
> 
> 
> 
> 
> 
> 
> I doubt my C2D would have enough power to run a gaming VM anyway. It was more of a proof of concept thing.
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks for the help


Once you get it working (with your new hardware), please share with us. If you use my how-to, leave a note there too.


----------



## dizzy4

The C2D might be ok. If you are keeping a socket 775 setup you need to look at all the Q45 chipsets and then look for vt-d in the manual. C2Q would actually be decent


----------



## powerhouse

For those who try VGA passthrough using an Asus X79 board, have a look at this: http://vip.asus.com/forum/view.aspx?id=20120806185452699&board_id=1&model=SABERTOOTH+X79&page=1&SLanguage=en-us.

The two latest BIOS releases 2002 and 2104 for the Sabertooth X79 have broken VT-d support ! According to Asus, there is no rolling back to older BIOS versions. So watch out.


----------



## powerhouse

Update on network "issues":

I posted earlier that I have issues with file transfers over the network (Samba shares). The speed from both Xen dom0 and Windows domU are slow when transferring to remote PCs over Fast Ethernet and/or Gigabit Ethernet (around 2-5MB/s).

Yesterday I copied some 4GByte of files from Windows domU to a share on the Linux dom0 - transfer speed is ~130MB/s







. So communication between domU and dom0 works nicely.


----------



## wTheOnew

Quote:


> Originally Posted by *powerhouse*
> 
> Update on network "issues":
> I posted earlier that I have issues with file transfers over the network (Samba shares). The speed from both Xen dom0 and Windows domU are slow when transferring to remote PCs over Fast Ethernet and/or Gigabit Ethernet (around 2-5MB/s).
> Yesterday I copied some 4GByte of files from Windows domU to a share on the Linux dom0 - transfer speed is ~130MB/s
> 
> 
> 
> 
> 
> 
> 
> . So communication between domU and dom0 works nicely.


Awesome.

Interestingly enough, I got my Win Server 2012 Essentials VM set up on Saturday. Didn't need any of the Xen drivers. Everything runs great right out of the box.


----------



## doublehelix

Hello dizzy4,

first of all I want to thank you for this awesome guide. Allthoug I'm a complete newbie in linux I've managed to setup fedora easyly only because of your gude! But I ran into some problems with xen. I've installed all exactly like you did all the partitioning and stuff is the same. Fedora boots and runs just fine I've also installed the rpmfusion drivers which work fine:thumb:. But when I try to boot into xen I see the loading bar and than lots of text and then it stops. No graphic interface no bothing rolleyes.gif. I've also tried to boot with nomodeset, but still doesnt work. Because of my i5 I've tried to boot with the integrated graphiccard but that also didnt work. I hope you can help me a little bit thumb.gif

PS my sys:
Mobo: Zotac Z68ITX
CPU: i5 2500k
GPU: Nvidia GTX 580

kindly regards doublehelix!

Hi again,

I've managed to boot into Xen but now I stuck in step7. If I enter

Code:



Code:


[[email protected] xenhost]# xm new /etc/xen/win7a.sxp

than allways an error appears saying

Code:



Code:


Error: [Errno 2] No such file or directory]

My config file looks like this:

Code:



Code:


kernel = "hvmloader"
builder='hvm'
memory = 4096
# set your memory however you want. 4G is enough for me, and i want the
# rest for the host and the other VM
name = "win7a"
vcpus=4
# I have 8 threads, 5 was a good number for me. 4 to game, 1 to run the rest
pae=1
acpi=1
apic=1
vif = [ 'type=ioemu, bridge=virbr0' ]
disk = [ 'phy:/dev/vg_dom0/domU,sda1, hda,w' , 'file:/home/xenhost/win7ult.iso,hdc:cdrom,r' ]
# More disks can be added later using this same method. The path to the device or a file can be added
# After the first comma is how the device will appear. "hda" is the first, "hdb" the second etc. "sda" format is also acceptable"
# had will appear as IDE, sda will appear as SCSI or SATA. After the second comma "r" means read only and "w" is for write
device_model = 'qemu-dm'
boot="dc"
sdl=0
opengl=1
vnc=1
vncpasswd=''
stdvga=0
#nographic=1 #!!! only uncomment this if you are using win8 or are trying to get a nvidia card to work
serial='pty'
tsc_mode=0
soundhw='all'
usb=1
usbdevice='mouse'
gfx_passthru=0
# Leaving this as 0 is how it works for me with Radeon HD cards
# You can change this later for iGPUs or nVdia if needed.
# Try it with 0 first
pci=[ '01:00.0', '01:00.1' , '00:1d.0'  ]
# These values are the ones you found out by doing the lspci command earlier
# I also passed through an entire USB controller for native support

*PLEASE* help me I'm so close to set up my first xen VM


----------



## wTheOnew

What's /var/log/xen/xend.log look like. It will hopefully say what file it can't find.


----------



## Fir3Chi3f

Already rep'ed for this guide, but I finally got my new board and i7 3820. I'll hopefully be able to get this going soon! Great stuff my friend


----------



## doublehelix

Hi, Big THX for the reply wTheOnew









here is my log file. I cant really read something useful out of it maybe you can help me?



Spoiler: Warning: Spoiler!



Code:



Code:


[2012-10-19 22:23:30 1236] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 22:23:30 1236] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 22:23:32 1236] DEBUG (XendNode:332) pscsi record count: 12
[2012-10-19 22:23:32 1236] DEBUG (XendCPUPool:747) recreate_active_pools
[2012-10-19 22:23:32 1236] DEBUG (XendDomainInfo:151) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 11589403014L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 7086280L, 'blocked': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'cpupool': 0, 'name': 'Domain-0'})
[2012-10-19 22:23:32 1236] INFO (XendDomainInfo:169) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-10-19 22:23:32 1236] DEBUG (XendDomainInfo:3420) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': "(linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0))", 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'name': 'Domain-0'}
[2012-10-19 22:23:32 1236] DEBUG (XendDomainInfo:1794) Storing domain details: {'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'memory/target': '7086280', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-10-19 22:23:32 1236] DEBUG (XendDomain:476) Adding Domain: 0
[2012-10-19 22:23:32 1236] DEBUG (XendDomain:410) number of vcpus to use is 0
[2012-10-19 22:23:32 1236] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2012-10-19 22:23:32 1236] INFO (SrvServer:184) unix path=/var/lib/xend/xend-socket
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: VBD.set_device not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: VBD.set_type not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: session.get_all_records not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: event.get_record not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: event.get_all not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: VIF.set_device not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: VIF.set_MAC not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: VIF.set_MTU not found
[2012-10-19 22:23:32 1236] WARNING (XendAPI:708) API call: debug.get_all not found
[2012-10-19 22:23:32 1236] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2012-10-19 22:23:32 1236] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2012-10-19 22:34:50 1247] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 22:34:50 1247] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 22:34:51 1247] DEBUG (XendNode:332) pscsi record count: 12
[2012-10-19 22:34:51 1247] DEBUG (XendCPUPool:747) recreate_active_pools
[2012-10-19 22:34:51 1247] DEBUG (XendDomainInfo:151) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 11038693989L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 1164928L, 'blocked': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'cpupool': 0, 'name': 'Domain-0'})
[2012-10-19 22:34:51 1247] INFO (XendDomainInfo:169) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-10-19 22:34:51 1247] DEBUG (XendDomainInfo:3420) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': "(linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0))", 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'name': 'Domain-0'}
[2012-10-19 22:34:52 1247] DEBUG (XendDomainInfo:1794) Storing domain details: {'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'memory/target': '1164928', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-10-19 22:34:52 1247] DEBUG (XendDomain:476) Adding Domain: 0
[2012-10-19 22:34:52 1247] DEBUG (XendDomain:410) number of vcpus to use is 0
[2012-10-19 22:34:52 1247] INFO (SrvServer:184) unix path=/var/lib/xend/xend-socket
[2012-10-19 22:34:52 1247] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: VBD.set_device not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: VBD.set_type not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: session.get_all_records not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: event.get_record not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: event.get_all not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: VIF.set_device not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: VIF.set_MAC not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: VIF.set_MTU not found
[2012-10-19 22:34:52 1247] WARNING (XendAPI:708) API call: debug.get_all not found
[2012-10-19 22:34:52 1247] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2012-10-19 22:34:52 1247] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2012-10-19 22:56:39 1233] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 22:56:39 1233] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 22:56:40 1233] DEBUG (XendNode:332) pscsi record count: 12
[2012-10-19 22:56:40 1233] DEBUG (XendCPUPool:747) recreate_active_pools
[2012-10-19 22:56:40 1233] DEBUG (XendDomainInfo:151) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 10384246434L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 1164928L, 'blocked': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'cpupool': 0, 'name': 'Domain-0'})
[2012-10-19 22:56:40 1233] INFO (XendDomainInfo:169) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-10-19 22:56:40 1233] DEBUG (XendDomainInfo:3420) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': "(linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0))", 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'name': 'Domain-0'}
[2012-10-19 22:56:40 1233] DEBUG (XendDomainInfo:1794) Storing domain details: {'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'memory/target': '1164928', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-10-19 22:56:40 1233] DEBUG (XendDomain:476) Adding Domain: 0
[2012-10-19 22:56:40 1233] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2012-10-19 22:56:40 1233] DEBUG (XendDomain:410) number of vcpus to use is 0
[2012-10-19 22:56:40 1233] INFO (SrvServer:184) unix path=/var/lib/xend/xend-socket
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: VBD.set_device not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: VBD.set_type not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: session.get_all_records not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: event.get_record not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: event.get_all not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: VIF.set_device not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: VIF.set_MAC not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: VIF.set_MTU not found
[2012-10-19 22:56:40 1233] WARNING (XendAPI:708) API call: debug.get_all not found
[2012-10-19 22:56:40 1233] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2012-10-19 22:56:40 1233] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2012-10-19 22:57:43 1233] DEBUG (SrvServer:77) SrvServer.cleanup()
[2012-10-19 22:57:43 1233] DEBUG (XMLRPCServer:251) XMLRPCServer.cleanup()
[2012-10-19 22:57:43 1233] DEBUG (XMLRPCServer:251) XMLRPCServer.cleanup()
[2012-10-19 22:57:43 1233] DEBUG (XendDomain:644) cleanup_domains
[2012-10-19 23:15:38 1223] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:15:38 1223] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:15:40 1223] DEBUG (XendNode:332) pscsi record count: 12
[2012-10-19 23:15:40 1223] DEBUG (XendCPUPool:747) recreate_active_pools
[2012-10-19 23:15:40 1223] DEBUG (XendDomainInfo:151) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 10152927738L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 1164928L, 'blocked': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'cpupool': 0, 'name': 'Domain-0'})
[2012-10-19 23:15:40 1223] INFO (XendDomainInfo:169) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-10-19 23:15:40 1223] DEBUG (XendDomainInfo:3420) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': "(linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0))", 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'name': 'Domain-0'}
[2012-10-19 23:15:40 1223] DEBUG (XendDomainInfo:1794) Storing domain details: {'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'memory/target': '1164928', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-10-19 23:15:40 1223] DEBUG (XendDomain:476) Adding Domain: 0
[2012-10-19 23:15:40 1223] DEBUG (XendDomain:410) number of vcpus to use is 0
[2012-10-19 23:15:40 1223] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2012-10-19 23:15:40 1223] INFO (SrvServer:184) unix path=/var/lib/xend/xend-socket
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: VBD.set_device not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: VBD.set_type not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: session.get_all_records not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: event.get_record not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708)Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0")

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1090, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 510, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0")
 API call: event.get_all not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: VIF.set_device not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: VIF.set_MAC not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: VIF.set_MTU not found
[2012-10-19 23:15:40 1223] WARNING (XendAPI:708) API call: debug.get_all not found
[2012-10-19 23:15:40 1223] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2012-10-19 23:15:40 1223] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2012-10-19 23:23:07 1255] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:23:07 1255] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:23:09 1255] DEBUG (XendNode:332) pscsi record count: 12
[2012-10-19 23:23:09 1255] DEBUG (XendCPUPool:747) recreate_active_pools
[2012-10-19 23:23:09 1255] DEBUG (XendDomainInfo:151) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 17419197800L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 1160832L, 'blocked': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'cpupool': 0, 'name': 'Domain-0'})
[2012-10-19 23:23:09 1255] INFO (XendDomainInfo:169) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-10-19 23:23:09 1255] DEBUG (XendDomainInfo:3420) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': "(linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0))", 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'name': 'Domain-0'}
[2012-10-19 23:23:09 1255] DEBUG (XendDomainInfo:1794) Storing domain details: {'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'memory/target': '1160832', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-10-19 23:23:09 1255] DEBUG (XendDomain:476) Adding Domain: 0
[2012-10-19 23:23:09 1255] DEBUG (XendDomain:410) number of vcpus to use is 0
[2012-10-19 23:23:09 1255] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2012-10-19 23:23:09 1255] INFO (SrvServer:184) unix path=/var/lib/xend/xend-socket
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: VBD.set_device not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: VBD.set_type not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: session.get_all_records not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: event.get_record not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: event.get_all not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: VIF.set_device not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: VIF.set_MAC not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: VIF.set_MTU not found
[2012-10-19 23:23:09 1255] WARNING (XendAPI:708) API call: debug.get_all not found
[2012-10-19 23:23:09 1255] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2012-10-19 23:23:09 1255] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2012-10-19 23:33:40 1255] DEBUG (SrvServer:77) SrvServer.cleanup()
[2012-10-19 23:33:40 1255] DEBUG (XMLRPCServer:251) XMLRPCServer.cleanup()
[2012-10-19 23:33:40 1255] DEBUG (XMLRPCServer:251) XMLRPCServer.cleanup()
[2012-10-19 23:33:40 1255] DEBUG (XendDomain:644) cleanup_domains
[2012-10-19 23:35:12 1241] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:35:12 1241] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:35:13 1241] DEBUG (XendNode:332) pscsi record count: 12
[2012-10-19 23:35:13 1241] DEBUG (XendCPUPool:747) recreate_active_pools
[2012-10-19 23:35:13 1241] DEBUG (XendDomainInfo:151) XendDomainInfo.recreate({'max_vcpu_id': 3, 'cpu_time': 10493081589L, 'ssidref': 0, 'hvm': 0, 'shutdown_reason': 255, 'dying': 0, 'online_vcpus': 4, 'domid': 0, 'paused': 0, 'crashed': 0, 'running': 1, 'maxmem_kb': 17179869180L, 'shutdown': 0, 'mem_kb': 1164928L, 'blocked': 0, 'handle': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'cpupool': 0, 'name': 'Domain-0'})
[2012-10-19 23:35:13 1241] INFO (XendDomainInfo:169) Recreating domain 0, UUID 00000000-0000-0000-0000-000000000000. at /local/domain/0
[2012-10-19 23:35:13 1241] DEBUG (XendDomainInfo:3420) Storing VM details: {'on_xend_stop': 'ignore', 'pool_name': 'Pool-0', 'shadow_memory': '0', 'uuid': '00000000-0000-0000-0000-000000000000', 'on_reboot': 'restart', 'image': "(linux (kernel '') (superpages 0) (nomigrate 0) (tsc_mode 0))", 'on_poweroff': 'destroy', 'bootloader_args': '', 'on_xend_start': 'ignore', 'on_crash': 'restart', 'xend/restart_count': '0', 'vcpus': '4', 'vcpu_avail': '15', 'bootloader': '', 'name': 'Domain-0'}
[2012-10-19 23:35:13 1241] DEBUG (XendDomainInfo:1794) Storing domain details: {'cpu/3/availability': 'online', 'description': '', 'console/limit': '1048576', 'memory/target': '1164928', 'cpu/2/availability': 'online', 'vm': '/vm/00000000-0000-0000-0000-000000000000', 'domid': '0', 'cpu/0/availability': 'online', 'cpu/1/availability': 'online', 'control/platform-feature-multiprocessor-suspend': '1', 'console/type': 'xenconsoled', 'name': 'Domain-0'}
[2012-10-19 23:35:13 1241] DEBUG (XendDomain:476) Adding Domain: 0
[2012-10-19 23:35:13 1241] DEBUG (XendDomain:410) number of vcpus to use is 0
[2012-10-19 23:35:13 1241] DEBUG (XendDomainInfo:1881) XendDomainInfo.handleShutdownWatch
[2012-10-19 23:35:13 1241] INFO (SrvServer:184) unix path=/var/lib/xend/xend-socket
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: VBD.set_device not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: VBD.set_type not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: session.get_all_records not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: event.get_record not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: event.get_all not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: VIF.set_device not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: VIF.set_MAC not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: VIF.set_MTU not found
[2012-10-19 23:35:13 1241] WARNING (XendAPI:708) API call: debug.get_all not found
[2012-10-19 23:35:13 1241] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xen-api.sock; authentication has been disabled for this server.
[2012-10-19 23:35:13 1241] INFO (XMLRPCServer:161) Opening Unix domain socket XML-RPC server on /var/run/xend/xmlrpc.sock.
[2012-10-19 23:38:59 1236] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:38:59 1236] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:38:59 1236] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0")

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1090, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 510, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0")

  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-19 23:41:45 1240] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:41:45 1240] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:41:45 1240] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-19 23:44:14 1237] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:44:14 1237] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:44:14 1237] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-19 23:46:47 1247] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:46:47 1247] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:46:47 1247] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-19 23:49:32 1229] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:49:32 1229] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:49:32 1229] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-19 23:51:09 1253] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:51:09 1253] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:51:09 1253] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-19 23:59:01 1236] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-19 23:59:01 1236] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-19 23:59:01 1236] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 00:01:56 1234] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 00:01:56 1234] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 00:01:56 1234] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 00:07:39 1240] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 00:07:39 1240] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 00:07:39 1240] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 00:08:29 1248] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 00:08:29 1248] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 00:08:29 1248] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 00:21:04 1256] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 00:21:04 1256] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 00:21:04 1256] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 01:07:26 1258] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 01:07:26 1258] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 01:07:26 1258] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 01:15:41 1233] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 01:15:41 1233] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 01:15:41 1233] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 01:35:22 1234] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 01:35:22 1234] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 01:35:22 1234] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 01:38:13 1228] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 01:38:13 1228] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 01:38:13 1228] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last)[2012-10-20 11:32:15 1248] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 11:32:15 1248] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 11:32:15 1248] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 11:50:00 1248] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 11:50:00 1248] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 11:50:00 1248] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 17:26:28 1326] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 17:26:28 1326] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 17:26:28 1326] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 17:59:18 1350] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 17:59:18 1350] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 17:59:18 1350] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 22:58:54 1312] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 22:58:54 1312] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 22:58:54 1312] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-20 23:00:09 1325] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-20 23:00:09 1325] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-20 23:00:09 1325] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348,  wTheOnewin run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0
[2012-10-21 10:03:05 1341] INFO (SrvDaemon:339) Xend Daemon started
[2012-10-21 10:03:05 1341] INFO (SrvDaemon:343) Xend changeset: unavailable.
[2012-10-21 10:03:05 1341] ERROR (SrvDaemon:356) Exception starting xend (no element found: line 1, column 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvDaemon.py", line 348, in run
    servers = SrvServer.create()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvServer.py", line 258, in create
    root.putChild('xend', SrvRoot())
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvRoot.py", line 40, in __init__
    self.get(name)
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 84, in get
    val = val.getobj()
  File "/usr/lib64/python2.7/site-packages/xen/web/SrvDir.py", line 52, in getobj
    self.obj = klassobj()
  File "/usr/lib64/python2.7/site-packages/xen/xend/server/SrvNode.py", line 30, in __init__
    self.xn = XendNode.instance()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 1181, in instance
    inst = XendNode()
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendNode.py", line 71, in __init__
    saved_host = self.state_store.load_state('host')
  File "/usr/lib64/python2.7/site-packages/xen/xend/XendStateStore.py", line 107, in load_state
    dom = minidom.parse(xml_path)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/minidom.py", line 1915, in parse
    return expatbuilder.parse(file)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 926, in parse
    result = builder.parseFile(fp)
  File "/usr/lib64/python2.7/site-packages/_xmlplus/dom/expatbuilder.py", line 211, in parseFile
    parser.Parse("", True)
ExpatError: no element found: line 1, column 0





FU** YEAH I've made it past this error. I Just had to remove all xml files from the xend directory. So far so good. Now I've added some PCI and USB devices like in the guide, but when I try to run the VM another error comes up:

Code:



Code:


Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0")

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 96, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 117, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/domain.py", line 1090, in startup
    self._backend.create()
  File "/usr/lib64/python2.7/site-packages/libvirt.py", line 510, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirtError: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0")

Can someone help me pls?


----------



## powerhouse

@doublehelix:

With regard to this error: "pci: PCI Backend and pci-stub don't own device 0000:00:1b.0", I have the following questions and suggestions.

First the suggestions:

1. Pass only those devices through that you really need !!! In most cases the graphics adapter and it's integrated sound device (if any) are all that need to be passed through to get started. Example: 01:00.0 and 01:00.1 for most graphics cards with integrated sound device residing in the first PCI slot. Check the output of lspci.
You can add additional devices for PCI passthrough later on when Windows is installed and running (with a passed through GPU).

2. The error message means essentially that the device 00:1b.0 has not been detached, that is it is not assignable. Before you start xm new, enter the following command and check the output:

Code:



Code:


xm pci-list-assignable-devices

Check to see if 00:1b.0 is listed. It probably isn't.

3. If you followed the instructions of the OP, have you defined that device in virt-manager?

Now the questions:

1. What is 00:1b.0? USB host? Sound device? Graphics adapter?
2. How far did you get? Did you manage to start xend (xm new) and install Windows? Does VGA passthrough work? From what I read you haven't yet managed to start xen new (the xend damon).
3. You listed an Nvidia graphics card as your graphics hardware, as well as the built-in i5 GPU. You will most likely not be able to get the Nvidia card passed through to Windows, at least not with standard kernel and hypervisor. Are you trying to passthrough the Intel integrated graphics adapter?
4. Just to make sure, are you using Fedora as suggested by the OP (which version)? Procedures for other Linux distros are often different.

EDIT: Following the OP instructions doesn't require pciback to be loaded at boot (it's already compiled into the Fedora 16 kernel). So there isn't any kernel boot option for pciback in the grub file.
5. Can you post the grub file (in my Ubuntu-based distro it's under /etc/default)? It should have a pciback kernel boot parameter. I believe the grub method only works with distros that have pciback compiled into the kernel. Ubuntu 12.04 and derivates for example don't have pciback compiled into the kernel, and pciback needs to be loaded as a module. This may complicate things.


----------



## blistovmhz

So, I've gotten what I believe to be most of the way there with both KVM and XEN.
I have two Nvidia cards (8800GTS and 560TI) as well as an old ATI Radeon HD card for testing.

In both KVM and XEN, the Radeon seems to just work. It only has displayport and I don't have a cable







but no errors on the hypervisor nor the guest.

As for the Nvidia cards. Same error in both Xen and KVM. Windows Error 43.

I've build Xen from unstable (git) and using xl toolkit. I create the machine to your specs (only recently found this post, but was using the same config anyhow), boot it up, connect to it with VNC, install win8, shut down, attach one of the cards (and their respective audio when applicable) but the card shows up with Error 43 again.
No errors in xend.log nor in the machine.log.

What am I missing?

Edit, I should also point out that my co-habitant has the same hardware as I've got, except that he's got an ATI card with the right goddamn cables. The hardest part about setting up passthrough is the plethora of outdated howto's out there.
Setup is pretty straight forward KVM and XEN so long as you don't rely on 2 year old instructions







(I wasted a tonne of time following old docs).

KVM (ubuntu):
apt-get install ubuntu-virt

modprobe pci-stub
virsh nodedev-list (list your devices)
virsh nodedev-dettach YOURPCIDEVICE
echo 1 > /sys/module/kvm/parameters/allow_unsafe_assigned_interrupts
virt-manager
Create your machine, use default settings, attach the PCI right now if you want to, using the virt-manager interface. Install as normal. You're done. Maybe add a USB passthrough for good measure, then once your video is working on the passthrough device, switch your primary display to the passthrough and disable the other. This works EXACTLY like this with ATI cards. Not so much with Nvidia thus far.

XEN:
I'll test a stock Xen 4.1 tomorrow to see if it's any more difficult. I've got the ATI working with Xen 4.2-unstable from GIT, but I suspect it'll work with 4.1 from the Ubuntu repos.


----------



## powerhouse

@blistovmhz:

Thanks for sharing your instructions for kvm. I wasn't even aware that kvm had VGA passthrough.

As to Xen 4.1, your ATI should work just fine. As for Nvidia, now that's another story. There are some instruction on the net, specifically David Techer's instructions and patches: http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through.

I've tried it with a Nvidia Quadro 600, but no luck. Eventually I replaced the Nvidia 600 for a (much more expensive) Nvidia Quadro 2000 and that worked right out of the box with a 4.1 hypervisor and Linux Mint 13 Mate (based on Ubuntu 12.04). The difference between the 2 cards is what Nvidia terms "multi-OS" support in the Quadro 2000 (and some other select cards, where the 2000 is the cheapest). These cards are designed for graphics workstations (CAD/CAM, Photoshop, video editing, etc.), not really for games. I don't care much for games, but graphics processing.

ATI cards - most of them - should be much easier to get working. Good luck!

P.S.: If you want to try Xen and Linux Mint 13 (with an ATI card), I can post a link to my how-to. It's very detailed (need to shorten it somehow).


----------



## blistovmhz

On the Xen side of things (Mint 13, Cinnamon, 64) running the Xen 4.1 package included in the Mint/Ubuntu repos, I was able to get the ATI card working without issue. It's just making me insane that my Nvidia card is so close in both Xen and KVM. Haven't seen much in the way of RECENT documentation for Xen 4.2 passthrough. All the old docs are so out of date that they don't seem worth following.
Just sorta pissed that I have a brand new NV card I can't get working. Seems silly to go buy an ATI just for this one issue, which I imagine I'll be able to resolve in the next few days/weeks.
But, GF is gettin' pissed at all the 4am hax









FYI, I'm pretty sure the only thing holding back the NV cards at this point (using stock Xen 4.1 and KVM) is our inability to retrieve the ROM/EEPROM. Loading with ROMBAR off doesn't make a difference, and as far as I can tell, that error 42 is all about the ROM.

Going to try digging up some NV tools for Windubs and see if I can't confirm that the ROM is unreadable. Suspect it's just a memory range thing.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> On the Xen side of things (Mint 13, Cinnamon, 64) running the Xen 4.1 package included in the Mint/Ubuntu repos, I was able to get the ATI card working without issue. It's just making me insane that my Nvidia card is so close in both Xen and KVM. Haven't seen much in the way of RECENT documentation for Xen 4.2 passthrough. All the old docs are so out of date that they don't seem worth following.
> Just sorta pissed that I have a brand new NV card I can't get working. Seems silly to go buy an ATI just for this one issue, which I imagine I'll be able to resolve in the next few days/weeks.
> But, GF is gettin' pissed at all the 4am hax
> 
> 
> 
> 
> 
> 
> 
> 
> FYI, I'm pretty sure the only thing holding back the NV cards at this point (using stock Xen 4.1 and KVM) is our inability to retrieve the ROM/EEPROM. Loading with ROMBAR off doesn't make a difference, and as far as I can tell, that error 42 is all about the ROM.
> Going to try digging up some NV tools for Windubs and see if I can't confirm that the ROM is unreadable. Suspect it's just a memory range thing.


I see you already got into video ROM retrieval. That was exactly the same point where my Quadro 600 gave me troubles. The video BIOS was taking more memory than current patches can handle (was it 64MB?).

Documentation is all over the place, and the Xen wiki is pretty much outdated concerning video card support and/or Xen 4.2, though they got a lot of documentation on xl. Unfortunately sharing information seems to be done via mailing list, and I've got no inclination signing up on a mailing list.

I've created a thread to list VGA passthrough compatible VGA adapters here http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters. Unfortunately there hasn't been much response.


----------



## blistovmhz

Well I've done a bunch more testing.

Gave up on the Nvidia card for right now. I figure there's enough unknowns with Xen/KVM versions/support/patches to deal with. ATI is far better supported.

An aside... When the hell did ATI start supporting the Linux community, and when did Nvidia stop? I remember years ago fighting with ATI about the FGLRX drivers and they told me and the rest of the community to go to hell. Meanwhile Nvidia was tossing code and firmware at us faster than we could implement.

So.

I tested KVM (bone stock) under Mint 13 (stock) with my Nvidia 8800GTS and 560TI, as well as my ATI RadeonHD 7870 (directCU ii) and 69xx. Won't bother discussing Nvidia for now.

The 69xx series just works. I'll have to double check that I was able to reboot the guest multiple times without crashing the host, but as I don't recall an issue, I assume there was none. No modifications needed at all.
The 7870 came up in both Win7 and 8 but I had to manually install the driver, and the host crashes on guest reboot with a kernel bug regarding drivers/pci/ats.c:62. I haven't figured out yet how to access the crash kernel in Ubuntu (durr) so I've no idea what's going on, but I strongly suspect it is an issue with the VGA bios on the guest's card not resetting clean (I've heard reports that newer ATI cards only REALLY POST during host boot). I'd imagine if I can strip the VGA bios / ROM that I'll be able to inject it manually during guest boot/reboot. Sound familiar? Any guidance?

I'll take a look at your other thread now.


----------



## 32oz

Couldn't you do the same thing with Hyper-V that's built into Win 8? I don't know anything about virtual machines, just asking...


----------



## lloyd mcclendon

absolutely not. For one thing that is 32 bit only (LOL). and i doubt it supports anything around pci pass through

And if you're already running windows why would you want to run an additional windows VM? makes no sense. The idea here is to use a real OS as the host and have windows around in a bastardized state just for what it is _needed for. Now that makes a ton of sense.


----------



## blistovmhz

So. Tested against stock Xen 4.1 under Mint13 as well (extensively).
Xen's infrastructure is much more robust but more work to manage. Where KVM did VGA PCI-passthrough with zero manual configuration (just click and click and click and click, Xen's toolkit is all scattered and bastard-packaged. That said however, both my 69xx and 7xxx worked perfect at bare metal performance. Benched with Heaven DX11 with tessellation on normal, all stock settings, at 1680x1050 and hit 1800 on the bench with the 7850, which is exactly the number I achieved on bare metal.
Played Portal2 for a bit, graphics were smooth but there was an occasional stutter which I believe to be caused by lack of CPU pinning.

NO GODDAMN AUDIO SUPPORT IN XEN!? ***?
Xen apparently stopped compiling qemu-dm for Alsa/Pulse support last year and instead are compiling for only OSS support?
Anyone have working audio (without a spare card) with Xen 4.1?

KVM required some configuration for audio as well, which was surprising. As your guests will run as "libvirt" in Ubuntu/Mint, they cannot access an unmodified PulseAudio server. Have to set the "system" startup in /etc/default/pulse. Seems silly they don't have something in the .deb to add a rule to pulse.


----------



## fergalom

Followed this guide and have win7 running successfully on a F17 Xen.

Installed a Radeon HD6850 card, paseed through and it was recognised by windows and installed latest drivers, reboot but greeted with a BSOD with "atikmpag.sys" being the message.
Can now only reboot in safe mode.

Anyone else come across this with their Radeon?

Re CCC - do you install this or only the drivers in a Xen DomU?

Trying to figure out if this is a xen, radeon driver or faulty card issue.

Build:
Asrock Extreme 6 MB
i7 3770 CPU
24GB Ram
120GB HDD
750W PSU
Fedora 17 Dom0
Win7 Ult DomU

Thanks


----------



## blistovmhz

I've run into this heaps of times and I've not found the cause.
I take two identical installs and one will work while the other will not, then miraculously both will work, then one will fail. I'm sure I must be changing something but I haven't yet figured out what.

I did notice however that I've had more consistent success using the xm tools and a native xen-xm config compared to using virt-manager/virsh to create the initial machine.

Here's my working setup

win7.cfg:
##########################################
kernel="/usr/lib/xen-default/boot/hvmloader"
builder='hvm'
memory = 8096
vcpus=6
name = "win7"
vif = [ 'type=ioemu, bridge=br0' ]
disk =
acpi = 1
boot="c"
sdl=0
serial='pty'
vnc=1
pci=[ '06:00.0', '06:00.1' ]
#usb=1
#usbdevice = "host:046d:c048"
###########################################

Use 'xm create win7.cfg' to bring the machine up. This will not add it to your inventory but will boot up (sorry, maybe add a CD drive as well. I have so many installs going now that I've taken to recycling).

Once the machine is up, use 'xtightvncviewer localhost" to view the machine, complete installation, reboot guest, hit it with vnc again, install ati drivers (I use most recent stable). I can't say that I see a definitive correllation between BSOD and CCC installed or not. I've stopped installing it recently (last 4 or 5 builds) and everything's been working. That said however, I've got 3 or 4 others with CCC installed (express install) which work just fine.

Once your machine is happy, you'll want to dump it into your inventory if you're using virt-manager. The official method (domxml-from-native) rarely works as your $xm.cfg has to have a tonne of stuff that is automatically generated on boot. I do it like this instead:

#Bring up the machine with
'xm create win7.cfg'
#Connect to xen with virsh
'virsh -c xen:///'
#Dump the generated XML and exit virsh
'dumpxml win7'
'exit'
#Copy/paste the output to "win7.xml"
#Shutdown the running guest.
'xm destroy win7'
#Define your new machine in Xen/Virsh
'virsh -c xen:///'
'define win7.xml' #NOTE: XML








#All done.

If all went well, you'll now see the machine in virt-manager
Also of note, when I bring up a new machine via XML config ripped directly from the XM config, I consistently get a BSOD which is why I'm doing my initial creation and driver installation with an xm.cfg.

Post your config?
Maybe also try booting to safe mode, wipe out the driver, reboot and reinstall driver (without CCC?)
I dunno. I've been up all night babysitting a goddamn VMware snapshot deletion which has left me with about 9 additional hours to my day to hack'n'slash my guests. I might be entirely delirious at this point. Please disregard everything I'm saying and have said.


----------



## powerhouse

If I remember correctly (I'm not at home), I'm booting Windows using xm new /etc/xen/win7.cfg (versus xm create ...). I put a shortcut on my desktop and it just works.

Still haven't figured out the difference between xm create and xm new.

In any case, using Linux Mint 13 (Ubuntu-based), I didn't use virt-manager but a script and /etc/modules entry to load pciback and to detach/attach the graphic card etc.

With regard to Nvidia, they still support Linux in proprietary drivers and they are usually damn good. BUT, they are useless for Xen







. Moreover, very few Nvidia cards support PCI passthrough out of the box. The Nvidia supported cards are specified "Multi-OS".

Next thing - are you trying to pass through your primary (your only) graphics card, or do you have two cards?

I gave up on primary graphics card passthrough. For once, I'm using an Nvidia card (Quadro 2000). That card works perfect under Windows 7 when passing through, but supposedly works under Linux dom0 only with the open source driver. I tried getting a Quadro 600 to work under a Linux dom0 with a proprietary driver but failed. The same card worked perfectly under native Linux Mint 13, just not with the Xen hypervisor.

It seems that ATI cards are generally recommended for use with Xen (plus the Mutli-OS Nvidia cards as secondary cards for pass-through).


----------



## blistovmhz

Quote:


> Originally Posted by *powerhouse*
> 
> If I remember correctly (I'm not at home), I'm booting Windows using xm new /etc/xen/win7.cfg (versus xm create ...). I put a shortcut on my desktop and it just works.
> Still haven't figured out the difference between xm create and xm new.
> In any case, using Linux Mint 13 (Ubuntu-based), I didn't use virt-manager but a script and /etc/modules entry to load pciback and to detach/attach the graphic card etc.
> With regard to Nvidia, they still support Linux in proprietary drivers and they are usually damn good. BUT, they are useless for Xen
> 
> 
> 
> 
> 
> 
> 
> . Moreover, very few Nvidia cards support PCI passthrough out of the box. The Nvidia supported cards are specified "Multi-OS".
> Next thing - are you trying to pass through your primary (your only) graphics card, or do you have two cards?
> I gave up on primary graphics card passthrough. For once, I'm using an Nvidia card (Quadro 2000). That card works perfect under Windows 7 when passing through, but supposedly works under Linux dom0 only with the open source driver. I tried getting a Quadro 600 to work under a Linux dom0 with a proprietary driver but failed. The same card worked perfectly under native Linux Mint 13, just not with the Xen hypervisor.
> It seems that ATI cards are generally recommended for use with Xen (plus the Mutli-OS Nvidia cards as secondary cards for pass-through).


Difference between 'xm new' and 'xm create':
xm new = create a new machine and add it to your dom0 inventory
xm create = create and run the specified config, but do not add to inventory.

I haven't tried passthrough on the primary card.

I use options in modprobe to dettach my pci dev like so (saves work, makes easier day to day):
/etc/modprobe.d/xen-pciback.conf
options xen-pciback passthrough=1 hide=(0000:06:00.0)(0000:06:00.1)

Also testing in Mint13.
I haven't got the Nvidia proprietary drivers working with the dom0 yet either, but I was under the impression that was because I borked my module pathing real bad. Will try again today.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> Difference between 'xm new' and 'xm create':
> xm new = create a new machine and add it to your dom0 inventory
> xm create = create and run the specified config, but do not add to inventory.
> I haven't tried passthrough on the primary card.
> I use options in modprobe to dettach my pci dev like so (saves work, makes easier day to day):
> /etc/modprobe.d/xen-pciback.conf
> options xen-pciback passthrough=1 hide=(0000:06:00.0)(0000:06:00.1)
> Also testing in Mint13.
> I haven't got the Nvidia proprietary drivers working with the dom0 yet either, but I was under the impression that was because I borked my module pathing real bad. Will try again today.


Thanks for the explanations. I hope one day I'll remember that.

With regard to adjusting paths in Linux Mint 13, have a look at my how-to (I know, it's lengthy, but you know what you are looking for): http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. Look at Part 3, section 2.

I believe the proprietary Nvidia driver with Xen issue stems from the need to compile the driver into the kernel (at least some compilation is done). Somehow the Xen hypervisor doesn't like the modified kernel, at least that is my impression. Haven't bothered to search for answers, though, just bought a cheap ATI card for dom0.


----------



## fergalom

blistovmhz - Thank you!!! Sorted with using your config and xm create command.

It also shows up in virt-manager, no need for the extra commands you specified.
Dom0 card is the onboard iGPU so not trying to pass through a primary card.

Re USB passthrough - usbdevice = "host:046d:c048" where do I find this type of address for the usb devices?

Next: Sort out the data transfer as per posts previously.


----------



## blistovmhz

Quote:


> Originally Posted by *fergalom*
> 
> blistovmhz - Thank you!!! Sorted with using your config and xm create command.
> It also shows up in virt-manager, no need for the extra commands you specified.
> Dom0 card is the onboard iGPU so not trying to pass through a primary card.
> Re USB passthrough - usbdevice = "host:046d:c048" where do I find this type of address for the usb devices?
> Next: Sort out the data transfer as per posts previously.


[email protected] ~ $ lsusb
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 006 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 008 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 009 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 010 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 011 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 04cc:1520 ST-Ericsson USB 2.0 Hub (Avocent KVM)
Bus 012 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 013 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 04cc:1520 ST-Ericsson USB 2.0 Hub (Avocent KVM)
Bus 007 Device 002: ID 046d:c517 Logitech, Inc. LX710 Cordless Desktop Laser
Bus 007 Device 003: ID 046d:c048 Logitech, Inc. G9 Laser Mouse
Bus 001 Device 003: ID 10d5:5552 Uni Class Technology Co., Ltd KVM Human Interface Composite Device (Keyboard/Mouse ports)
Bus 002 Device 003: ID 10d5:5552 Uni Class Technology Co., Ltd KVM Human Interface Composite Device (Keyboard/Mouse ports)
Bus 002 Device 004: ID 045e:0745 Microsoft Corp. Nano Transceiver v1.0 for Bluetooth

That's the output you're looking for. It's a pain in the ass the virt-manager doesn't yet support adding usb to Xen guests. If you're only running one keyboard/mouse, you're pretty much screwed.
I've found that the VNC display in virt-manager works fine for keyboard and mouse for anything other than gaming. Too much lag for mousing action. I keep a dom0 and domU mouse at all times, but use the vnc window to lock my keyboard to the domU.
You can also simply passthrough the PCI device for a given USB host, in which case all USB ports associated with that PCI device will be available in your domU. I haven't (embarrassingly) been able to figure out how to determine which PCI slot is associated with which USB bus. But when I do....!

Also, if you're running an older ATI card (<7xxx series), I'd give KVM serious consideration. Do some benchmarks between Xen and KVM with your older card. KVM is not as robust but it's tookchain is much more ... organized and I've had no problems with KVM and <7xxx cards. Plus, KVM is built in to pretty near every kernel in the Universe and Mr. Linux is fully behind it.
At this point, I use Xen only when I have performance or support problems with KVM. Were I using an older card (had I known about this, I'd have saved some money and got an older card) I'd be using KVM.


----------



## fergalom

Quote:


> I haven't (embarrassingly) been able to figure out how to determine which PCI slot is associated with which USB bus. But when I do....!


this I think I can help with

http://www.gossamer-threads.com/lists/xen/users/259868


----------



## nyxcharon

Well, just got a Intel i5 3570. Have a asrock Z77 pro3 mobo on the way. Should be working on this again soon, I'll post my results when I get it up and running. (or if I run into a bunch of issues







)


----------



## blistovmhz

I guess while we're on this, has ANYONE got audio working with Xen, without having to passthrough and entire APU? ie: Anyone got Xen qemu-dm working with Alsa/Pulse?
Any other brilliant workarounds in the mean time? I was thinking PA client on the guest, but that of course is no longer an option as the last update for the windows PA client was in the early 80's or something afaik.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> I haven't (embarrassingly) been able to figure out how to determine which PCI slot is associated with which USB bus. But when I do....!


In Linux Mint open Administration --> System Information. Then open Devices and click "Input Devices". Select your mouse or keyboard and you get the PCI device it's connected to.

Don't know how to do that under Fedora.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> It's a pain in the ass the virt-manager doesn't yet support adding usb to Xen guests. If you're only running one keyboard/mouse, you're pretty much screwed.


My suggestion: Get a KVM switch (keyboard-video-mouse switch) with USB ports. You need two USB input ports for mouse and keyboard, and two USB cables to connect to two different USB ports on the PC. I don't use the video port on the switch (got no use for VGA D-sub connectors).

The tricky part is that you need to identify your USB ports on the PC. To do so I used a USB pendrive.

In my case I had to install a 2-port USB panel into one free PC slot and connect its cables to the appropriate USB header on the motherboard. Be aware that USB3 ports are no good for mouse/keyboard.

The result is I can switch between Linux and Windows at the press of a button.


----------



## powerhouse

Quote:


> Originally Posted by *fergalom*
> 
> ...Next: Sort out the data transfer as per posts previously.


With regard to the "low" disk index, try the GPLPV drivers under Windows. Here is the link to the "signed" drivers:
http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers


----------



## fergalom

Quote:


> Originally Posted by *blistovmhz*
> 
> I guess while we're on this, has ANYONE got audio working with Xen, without having to passthrough and entire APU? ie: Anyone got Xen qemu-dm working with Alsa/Pulse?
> Any other brilliant workarounds in the mean time? I was thinking PA client on the guest, but that of course is no longer an option as the last update for the windows PA client was in the early 80's or something afaik.


Audio is working with my win7 domU. I used the config you specified earlier for me and passing through a Radeon HD6850

Also I have found if I click the mouse inside the domU graphical console/viewer on dom0, it will pass through the mouse to the win7 domU without any need to passthrough any USB controllers in the cfg file.
(Hope this makes sense)

@Powerhouse - I've tried the GPLGV drivers but they have no effect in my system, haven't had a chance to have a closer look / un-install / re-install.

F


----------



## blistovmhz

Quote:


> Originally Posted by *powerhouse*
> 
> My suggestion: Get a KVM switch (keyboard-video-mouse switch) with USB ports. You need two USB input ports for mouse and keyboard, and two USB cables to connect to two different USB ports on the PC. I don't use the video port on the switch (got no use for VGA D-sub connectors).
> The tricky part is that you need to identify your USB ports on the PC. To do so I used a USB pendrive.
> In my case I had to install a 2-port USB panel into one free PC slot and connect its cables to the appropriate USB header on the motherboard. Be aware that USB3 ports are no good for mouse/keyboard.
> The result is I can switch between Linux and Windows at the press of a button.


Tried this with my Starview 2 port dual DVI USB KVM switch. No dice unfortunately and I couldn't for the life of me figure out why. When I attach the KVM to DomU, DomU will occasionally see the device after 3-4 minutes







Suspect buggy KVM as I've had other unrelated issues with this box in the past. For now I'm doing two things:

Using Synergy to share my keyboard/mouse/clipboard. For those of you not gaming (why are you using passthrough if you're not gaming?) Synergy is ****in' stellar. Essentially just move your mouse to the edge of Dom0 and it'll show up in DomU. Synergy however, does not play well with DirectX/DirectInput.








Writing a script to trap SOME EVENT in DomU which will indicate that I want to re-attach my USB devices back to my Dom0 (like a soft KVM).

Quote:


> Originally Posted by *fergalom*
> 
> Audio is working with my win7 domU. I used the config you specified earlier for me and passing through a Radeon HD6850
> Also I have found if I click the mouse inside the domU graphical console/viewer on dom0, it will pass through the mouse to the win7 domU without any need to passthrough any USB controllers in the cfg file.
> (Hope this makes sense)


Makes sense. You're passing through the audio from your Radeon HD card though aren't you? In which case you're only getting an HDMI output? Same here. Works fine except that my speakers are already attached to my Dom0. I plugged my DomU monitor's analog output into my Dom0's line in which worked just fine and will probably go back to this, but this is a hack that doesn't work for everyone (ie: anyone without HDMI). I want a boxed solution we can share with non-haxor types.
Also, as for using the VNC input. Works .... okay for doin' taxes and junk but the latency is far too high for FPS.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> Tried this with my Starview 2 port dual DVI USB KVM switch. No dice unfortunately and I couldn't for the life of me figure out why. When I attach the KVM to DomU, DomU will occasionally see the device after 3-4 minutes
> 
> 
> 
> 
> 
> 
> 
> Suspect buggy KVM as I've had other unrelated issues with this box in the past. For now I'm doing two things:
> 
> Using Synergy to share my keyboard/mouse/clipboard. For those of you not gaming (why are you using passthrough if you're not gaming?) Synergy is ****in' stellar. Essentially just move your mouse to the edge of Dom0 and it'll show up in DomU. Synergy however, does not play well with DirectX/DirectInput.
> 
> 
> 
> 
> 
> 
> 
> 
> Writing a script to trap SOME EVENT in DomU which will indicate that I want to re-attach my USB devices back to my Dom0 (like a soft KVM).
> Makes sense. You're passing through the audio from your Radeon HD card though aren't you? In which case you're only getting an HDMI output? Same here. Works fine except that my speakers are already attached to my Dom0. I plugged my DomU monitor's analog output into my Dom0's line in which worked just fine and will probably go back to this, but this is a hack that doesn't work for everyone (ie: anyone without HDMI). I want a boxed solution we can share with non-haxor types.
> Also, as for using the VNC input. Works .... okay for doin' taxes and junk but the latency is far too high for FPS.


With regard to your KVM box: Are you sure you connected the USB cables of the KVM box to 2 different USB *host adapters*? One host adapter must be used by the dom0, the other one must be passed through to Windows. Another thing to watch out for is that you use USB 2 hosts for your keyboard/mouse. Don't try USB 3 host adapters - they may cause troubles in both Linux and Windows for mouse/keyboard. I had to install a 2 port USB panel and connect the cables to an appropriate USB header on my mother board. Why? Because the USB connectors on my motherboard (back and front panels) were all either USB 2 connected to one host, or USB 3 connected to another host. To get a second USB 2 host I had to add an USB panel on the back and connect it to an USB header on the motherboard (not any header, but the first one in my case). Check with an USB pendrive. You absolutely must have different host adapters for both dom0 and domU.

"For those of you not gaming (why are you using passthrough if you're not gaming?) Synergy is ****in' stellar."

1. I need to check out synergy.
2. Gaming is peanuts compared to photo editing. OK, I grant that gaming may be more demanding on the graphics card. BUT, if you want to edit photos and are serious about it, you need to calibrate your screen and - again, you are serious - your screen supports internal LUTs. In simple terms, you need to have Windows upload your screen calibration to the screen via DVI. That is the first requirement. The next one is to have Adobe Lightroom and Adobe Photoshop use the graphics card to speed up graphics processing. Believe me, RAW photo processing is VERY CPU and graphics card intensive. So, I'm not a gamer but I really need full access to the graphics card from within Windows (as well as access to the USB port to connect my colorimeter). Xen VGA passthrough works perfect for this, if you get it working.


----------



## powerhouse

Quote:


> Originally Posted by *fergalom*
> 
> Audio is working with my win7 domU. I used the config you specified earlier for me and passing through a Radeon HD6850
> Also I have found if I click the mouse inside the domU graphical console/viewer on dom0, it will pass through the mouse to the win7 domU without any need to passthrough any USB controllers in the cfg file.
> (Hope this makes sense)
> @Powerhouse - I've tried the GPLGV drivers but they have no effect in my system, haven't had a chance to have a closer look / un-install / re-install.
> F


Hm, strange. In my case they made a HUGE difference. I get a Windows index of 7.8 (up from 5.8) when installing the GPL drivers. You are talking about the GPLPV drivers, aren't you? Check my link if uncertain.


----------



## blistovmhz

Quote:


> Originally Posted by *powerhouse*
> 
> Hm, strange. In my case they made a HUGE difference. I get a Windows index of 7.8 (up from 5.8) when installing the GPL drivers. You are talking about the GPLPV drivers, aren't you? Check my link if uncertain.


Aren't those only for PV guests?


----------



## Killer7

I used to run this a long time ago, the cool part is that you can hook up two video cards and split the other resources in half and turn your PC into a two-player gaming system, separate monitors, keyboards, everything!;p


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> Aren't those only for PV guests?


Don't think so. Windows isn't available as PV guest, if I remember correctly.

In any case, they really work.


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> Don't think so. Windows isn't available as PV guest, if I remember correctly.


I believe a lot of server versions can be PV guests. Look into hyper-v. I am not entirely sure about that technology.

It is unfortunate that Redhat has dropped support for Xen, but I am happy to report that there are people building packages and still using it extensively. I am running a minimal CentOS 6.3 install with Xen 4.2.0 right now thanks to: https://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/

His work has been great and makes things very easy to manage. No need to compile Xen (or a dom0 kernel) or install all of the development tools. Something to note is that the most recent kernel (3.6.3-1) gave me persistent disk I/O errors and I had to compile my own kernel. Chances are that it is just an irregularity in my hardware or storage controller, but be warned. Hopefully I will be able to compile everything into a live image with scripts to aid in configuration.

One advantage to switching to Xen 4.2.0 is the maturity of the xl toolstack. I find it to be far superior to xm. Since there is no need to use my old libvirt hot-swap, the network configuration seems easier because there are not competing bridges. That is to say that network configuration is less complicated (at least for me). I wouldn't be the right person to develop a GUI management interface, but I assure you that using xl would make hot swapping pci devices just a "click" away.

best,
dizzy4

P.S. Sorry I have not checked in as much as I should. I have been busy learning SolidWorks and trying to further my career. On the bright side, I can port my virtual CAD machine to any system running xen


----------



## blistovmhz

Quote:


> Originally Posted by *powerhouse*
> 
> Don't think so. Windows isn't available as PV guest, if I remember correctly.
> In any case, they really work.


Well, there ARE drivers for windows, which I've installed and they seem to ... not crash anything, but I haven't seen any change in performance.
My CPU/GPU are both ranking 7.9 on Windows perf bench, but the disk is only getting a 5.9. Guest disc is an LV on a Raid0 so should be able to push 170MBps. Only seeing about 60 I think?

I've set the "xen_platform_pci=1" in my hvm .cfg but no idea how to confirm it's actually enabled?
Does the fact that I'm using disk type "xen" and "netfront" drivers successfully suggest that the PV drivers are working?


----------



## dizzy4

Quote:


> Originally Posted by *blistovmhz*
> 
> Well, there ARE drivers for windows, which I've installed and they seem to ... not crash anything, but I haven't seen any change in performance.
> My CPU/GPU are both ranking 7.9 on Windows perf bench, but the disk is only getting a 5.9. Guest disc is an LV on a Raid0 so should be able to push 170MBps. Only seeing about 60 I think?
> I've set the "xen_platform_pci=1" in my hvm .cfg but no idea how to confirm it's actually enabled?
> Does the fact that I'm using disk type "xen" and "netfront" drivers successfully suggest that the PV drivers are working?


Don't trust those ratings too much. I am only getting a 6.8.



I don't usually install any PV drivers, but I believe the xen platform device looks like this (before driver install):


----------



## blistovmhz

Quote:


> Originally Posted by *dizzy4*
> 
> Don't trust those ratings too much. I am only getting a 6.8.
> 
> I don't usually install any PV drivers, but I believe the xen platform device looks like this (before driver install):


Thanks. Looks like the device is there, but drivers not attached?

Meh...

Also, I couldn't get around the USB issue. I can attach a specific USB device but not an entire bus because all my USB2.0 ports have the same deviceID. When I plug a USB2.0 device into a USB3.0 port, that port gets shunted to the USB2.0 bus with the same goddamn deviceID. All USB are on the same PCI slot, so no way I can configure one port of my KVM to be dedicated for the guest.

That said however, I did some more reading on Synergy. Synergy has an open bug for DXinput where the relative mouse movement gets scaled all wrong because DX assumes the cursor is in the center of the screen at all time (and locks it there). Synergy however takes precedence and DX never gets a lock. The bug has been open for over a year and according to the info in the bug report, the devs had no intention of fixing this.
Turns out it was already fixed which is why they said.. they ... wouldn't fix it ? They coulda mentioned that in the bug report I guess.

If you go into your Synergy server and configure a hotkey for your guest with the action "lockCursorToScreen=toggle", when you press the designated key in your guest, your DX will work. In my case, my Synergy works as normal. When I move my cursor over to the left edge of the host, the cursor is passed into the Win7 guest. When I wanna play a DX game, I just hit "numlock" to toggle the "lockCursorToScreen" and everything works epic.

So, FYI, Synergy is probably easier than dedicating a device to your guest (unless your guest is intended to be a completely separate workstation).

Also, seriously consider donating to Synergy. I'd love to see it developed further.

Also, as of right now, my Win7 Xen HVM is benching 1600 on Heaven, 7.8/7.9/7.9/5.9 on WIn7 performance bench, playing Portal 2 right now with no issues, running on an Radeon HD 7870.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> I believe a lot of server versions can be PV guests. Look into hyper-v. I am not entirely sure about that technology.


According to the following Xen wiki, Windows is an unmodified OS and thus it is not available as PV guest. See section on Windows domU and PV drivers here: http://wiki.xen.org/wiki/DomU_Support_for_Xen.
Quote:


> Originally Posted by *dizzy4*
> 
> It is unfortunate that Redhat has dropped support for Xen, but I am happy to report that there are people building packages and still using it extensively. I am running a minimal CentOS 6.3 install with Xen 4.2.0 right now thanks to: https://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/


That's bad news and I hope other distros aren't following. I sometimes get the feeling that Xen gets pushed aside in favor of KVM, which I can't understand really. Well, actually I understand Redhat's move since they bought Qumranet, the developer of KVM, back in 2008. They probably want to push their own technology.
I can't say if KVM is any good/better/worse than Xen, I felt the KVM documentation was/is terrible - too developer focused, not really helping users to accomplish their tasks. As to PCI passthrough, blistovmhz' post above was the first one I saw on using KVM for VGA passthrough. Some time ago a KVM developer actually got VGA passthrough working without need for VT-d, but his code was dumped in favor of SPICE, if I'm not wrong. At first I wanted to use KVM, but with all this I decided to go with Xen. It wasn't easy, but Xen seems to have all the tools and functions in place - and still developing new ones - to sustain / maintain the virtualization framework.
Quote:


> Originally Posted by *dizzy4*
> 
> One advantage to switching to Xen 4.2.0 is the maturity of the xl toolstack. I find it to be far superior to xm. Since there is no need to use my old libvirt hot-swap, the network configuration seems easier because there are not competing bridges. That is to say that network configuration is less complicated (at least for me). I wouldn't be the right person to develop a GUI management interface, but I assure you that using xl would make hot swapping pci devices just a "click" away.
> best,
> dizzy4
> P.S. Sorry I have not checked in as much as I should. I have been busy learning SolidWorks and trying to further my career. On the bright side, I can port my virtual CAD machine to any system running xen


Thanks for sharing your Xen 4.2 experience. I'm waiting patiently for getting it in my Linux Mint repos. I just don't want to break my installation going with packages that aren't yet in the stable repos, though I'm dying to give xl a try.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> Well, there ARE drivers for windows, which I've installed and they seem to ... not crash anything, but I haven't seen any change in performance.
> My CPU/GPU are both ranking 7.9 on Windows perf bench, but the disk is only getting a 5.9. Guest disc is an LV on a Raid0 so should be able to push 170MBps. Only seeing about 60 I think?
> I've set the "xen_platform_pci=1" in my hvm .cfg but no idea how to confirm it's actually enabled?
> Does the fact that I'm using disk type "xen" and "netfront" drivers successfully suggest that the PV drivers are working?


60MB seems low. EDIT: Just did an unscientific speed test, copying ~4.4GB files from SSD to LVM stripped disks (like RAID0) at around 250MB/s. At another time I measured ~130MB/s transfer speed between Windows 7 domU and a Samba share on dom0 (a single disk, not RAID0). So yes, in your case there is room for improvement.

I don't have "xen_platform_pci=1" in my win7.cfg file! Still, after I installed the GPLPV drivers in Windows, the performance index jumped from 5.8ish to 7.8 (I have an SSD as primary disk). I don't know about "xen" disk type and "netfront" drivers. Here is my Device Manager screen:



Notice the "Xen Block Device Driver" under "Storage controllers" and the "Xen Net Device Driver" under "Network adapters". There is also a "Xen PCI Device Driver" under "System devices", not shown in the picture.

EDIT: I got a similar issue with the "Other devices" "Xen PCI device" - I don't have a driver present and Windows doesn't seem to care about it, except for the yellow triangle in Device Manager. On the other hand, I don't have a working audio under Windows (same as blistovmhz described - I don't use HDMI and my speakers are used by dom0). Currently #sound=all is disabled in the /etc/xen/win7.cfg file, so I don't expect to get any sound under Windows. This is my next (last?) thing to solve.

Hope it helps.


----------



## blistovmhz

Quote:


> Originally Posted by *powerhouse*
> 
> According to the following Xen wiki, Windows is an unmodified OS and thus it is not available as PV guest. See section on Windows domU and PV drivers here: http://wiki.xen.org/wiki/DomU_Support_for_Xen.
> That's bad news and I hope other distros aren't following. I sometimes get the feeling that Xen gets pushed aside in favor of KVM, which I can't understand really. Well, actually I understand Redhat's move since they bought Qumranet, the developer of KVM, back in 2008. They probably want to push their own technology.
> I can't say if KVM is any good/better/worse than Xen, I felt the KVM documentation was/is terrible - too developer focused, not really helping users to accomplish their tasks. As to PCI passthrough, blistovmhz' post above was the first one I saw on using KVM for VGA passthrough. Some time ago a KVM developer actually got VGA passthrough working without need for VT-d, but his code was dumped in favor of SPICE, if I'm not wrong. At first I wanted to use KVM, but with all this I decided to go with Xen. It wasn't easy, but Xen seems to have all the tools and functions in place - and still developing new ones - to sustain / maintain the virtualization framework.
> Thanks for sharing your Xen 4.2 experience. I'm waiting patiently for getting it in my Linux Mint repos. I just don't want to break my installation going with packages that aren't yet in the stable repos, though I'm dying to give xl a try.


As of Xen 4.2, the source does come with Debian packaging tools. I've done a test compile/build/package already. Works fine. Annoying compile though as all the dependency lists are out of date and there are a few unresolvable deps that you have to hack around. I couldn't for the life of me figure out what Pulse libs were needed to build qemu-dm with Pulse support, and there was a dependency conflict between libjpegXYZ and libjpeg-turbo-XYZ. I think I got around it by manually installing one or the other and the build didn't seem to complain.
The end result is you'll have a git clone of Xen, with .deb packages for everything.

Quote:


> Originally Posted by *powerhouse*
> 
> 60MB seems low. EDIT: Just did an unscientific speed test, copying ~4.4GB files from SSD to LVM stripped disks (like RAID0) at around 250MB/s. At another time I measured ~130MB/s transfer speed between Windows 7 domU and a Samba share on dom0 (a single disk, not RAID0). So yes, in your case there is room for improvement.
> I don't have "xen_platform_pci=1" in my win7.cfg file! Still, after I installed the GPLPV drivers in Windows, the performance index jumped from 5.8ish to 7.8 (I have an SSD as primary disk). I don't know about "xen" disk type and "netfront" drivers. Here is my Device Manager screen:
> 
> Notice the "Xen Block Device Driver" under "Storage controllers" and the "Xen Net Device Driver" under "Network adapters". There is also a "Xen PCI Device Driver" under "System devices", not shown in the picture.
> EDIT: I got a similar issue with the "Other devices" "Xen PCI device" - I don't have a driver present and Windows doesn't seem to care about it, except for the yellow triangle in Device Manager. On the other hand, I don't have a working audio under Windows (same as blistovmhz described - I don't use HDMI and my speakers are used by dom0). Currently #sound=all is disabled in the /etc/xen/win7.cfg file, so I don't expect to get any sound under Windows. This is my next (last?) thing to solve.
> Hope it helps.


Again on audio. There are several solutions but none of them pretty. Qemu-dm from git MAY support Pulse but I haven't been able to confirm as I can't resolve the PA libs dependencies.
The easiest option currently is to pass through a secondary audio card.
The next easiest is to use HDMI from your DomU to your display, and use your displays speakers or plug them back into your Dom0's line in (and manually crank up the Line feed).

If you're not terribly concerned with A/V sync, you can always install a dummy sound driver (actually any sound driver will work so long as the DomU starts) and pipe the output over network to your Dom0's Pulse server.

I did the following (won't go into huge detail unless someone requests it, as this is not at all an ideal solution):

Download Netcat for windows. http://joncraton.org/media/files/nc111nt.zip
Download Linco (LineOnCode) for Windows. http://liveincode.romanrm.ru/download/liveincode/0.94/LiveInCode-0.94.exe
Configure your Dom0's Pulse server to accept RAW TCP streams on port 5000 by adding the following line to the end of your /etc/pulse/default.pa
module-simple-protocol-tcp port=5000
Save your default.pa and restart your PA server (manually kill both processes or just reboot).
Copy your NetCat binary to the same directory Linco is installed in.
Try the following (from DomU)- `linco.exe -B 16 -C 2 -R 44100 | nc.exe IPofTHEdom0MACHINE 5000`
You should now be streaming all audio. If your WIndows sound driver doesn't exist or isn't working (make sure you have the right output selected and verify that you can see some DB levels when playing some sound) install the dummy audio driver from Barix
http://www.barix.com/downloads/file/Virtual_Audio_Card_VACard_2b1_ZIP/9481/221 - This is the dummy driver I was using before switching to HDMI.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> Again on audio. There are several solutions but none of them pretty. Qemu-dm from git MAY support Pulse but I haven't been able to confirm as I can't resolve the PA libs dependencies.
> The easiest option currently is to pass through a secondary audio card.
> The next easiest is to use HDMI from your DomU to your display, and use your displays speakers or plug them back into your Dom0's line in (and manually crank up the Line feed).
> If you're not terribly concerned with A/V sync, you can always install a dummy sound driver (actually any sound driver will work so long as the DomU starts) and pipe the output over network to your Dom0's Pulse server.
> I did the following (won't go into huge detail unless someone requests it, as this is not at all an ideal solution):
> 
> Download Netcat for windows. http://joncraton.org/media/files/nc111nt.zip
> Download Linco (LineOnCode) for Windows. http://liveincode.romanrm.ru/download/liveincode/0.94/LiveInCode-0.94.exe
> Configure your Dom0's Pulse server to accept RAW TCP streams on port 5000 by adding the following line to the end of your /etc/pulse/default.pa
> module-simple-protocol-tcp port=5000
> Save your default.pa and restart your PA server (manually kill both processes or just reboot).
> Copy your NetCat binary to the same directory Linco is installed in.
> Try the following (from DomU)- `linco.exe -B 16 -C 2 -R 44100 | nc.exe IPofTHEdom0MACHINE 5000`
> You should now be streaming all audio. If your WIndows sound driver doesn't exist or isn't working (make sure you have the right output selected and verify that you can see some DB levels when playing some sound) install the dummy audio driver from Barix
> http://www.barix.com/downloads/file/Virtual_Audio_Card_VACard_2b1_ZIP/9481/221 - This is the dummy driver I was using before switching to HDMI.


I found the following http://gleamynode.net/articles/2228/ and was trying it. Actually I modified it by using plink from PuTTY. Here is my Windows command line:

Code:



Code:


linco.exe -B 16 -C 2 -R 44100 | plink woody "cat - | pacat --playback"

I also installed the dummy audio driver you suggested, which linco is using (it says so on the command prompt). The point is, my Windows reports "No audio devices installed" and the volume icon on the bottom right has a red X. I can't configure anything sound related under Windows.

I don't know if following your instructions to the letter will do much, if I don't have an audio device installed in the first place. How do you get them working?

By the way, Device manager lists the Quadro 2000 HDMI audio drivers, but they can't be used since I have DVI to the screen.


----------



## blistovmhz

Quote:


> Originally Posted by *powerhouse*
> 
> I found the following http://gleamynode.net/articles/2228/ and was trying it. Actually I modified it by using plink from PuTTY. Here is my Windows command line:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> linco.exe -B 16 -C 2 -R 44100 | plink woody "cat - | pacat --playback"
> 
> I also installed the dummy audio driver you suggested, which linco is using (it says so on the command prompt). The point is, my Windows reports "No audio devices installed" and the volume icon on the bottom right has a red X. I can't configure anything sound related under Windows.
> I don't know if following your instructions to the letter will do much, if I don't have an audio device installed in the first place. How do you get them working?
> By the way, Device manager lists the Quadro 2000 HDMI audio drivers, but they can't be used since I have DVI to the screen.



Make sure the dummy audio driver is installed and set it as default. You should see the VU moving around when you play something.
Follow my instructions







The instructions you linked are incomplete and more complicated.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> 
> Make sure the dummy audio driver is installed and set it as default. You should see the VU moving around when you play something.
> Follow my instructions
> 
> 
> 
> 
> 
> 
> 
> The instructions you linked are incomplete and more complicated.


Thanks for your help. I happened to be near a computer store today and bought a $8 USB sound adapter (well, the extra cable was another $4).

What can I say - it isn't a nice solution either (I hate these cables everywhere), but it works.

I will try your instructions at a later time, just wanted to get things working quickly. Again, many thanks for your help!


----------



## nyxcharon

So, i got my new cpu and mobo, was very easily able to pass thru my AMD 7850.
However, after I install my drivers, it blues screens on reboot. Anyone got any advice on how to fix this?
Also, if i remove the graphics card device from the VM and restart it starts up fine.


----------



## blistovmhz

Quote:


> Originally Posted by *nyxcharon*
> 
> So, i got my new cpu and mobo, was very easily able to pass thru my AMD 7850.
> However, after I install my drivers, it blues screens on reboot. Anyone got any advice on how to fix this?
> Also, if i remove the graphics card device from the VM and restart it starts up fine.


I keep finding that machines created with virt-manager seem to hang on reboot after installing drivers.

I'd manually create your .cfg file and us 'xm create yourfile.cfg' , boot into safe mode, uninstall the drivers, reboot normally, install drivers and reboot.
Seems to work consistently for me. Never had a virt-manager created machine work beyond one reboot with Radeon drivers.

Some people have suggested that installing without Catayst control may help? Didn't work for me.

FYI, here's my current config:

kernel="/usr/lib/xen-default/boot/hvmloader"
builder='hvm'
memory = 8096
vcpus=6
name = "win7"
vif = [ 'type=ioemu, bridge=virbr0' ]
disk = [ 'phy:/dev/blarg/win7infinium,xvda,w' ]
acpi = 1
boot="c"
sdl=0
serial='pty'
vnc=1
pci=[ '06:00.0', '06:00.1' ]

I save that as "win7.cfg", then 'xm create win7.cfg'. Boot into safe, uninstall drivers, reboot normal, install drivers, all should be good now.
Once you have a stable machine, bring up the machine then dump the xml file like so:

virsh -c xen:///
dumpxml

Copy/paste the resultant XML file then use 'virsh -c xen:///' again to "define" your new machine into virt-manager.
define win7.xml

Machine will now be in your xen inventory (in virt-manager) but should reboot fine.


----------



## dizzy4

Quote:


> Originally Posted by *nyxcharon*
> 
> So, i got my new cpu and mobo, was very easily able to pass thru my AMD 7850.
> However, after I install my drivers, it blues screens on reboot. Anyone got any advice on how to fix this?
> Also, if i remove the graphics card device from the VM and restart it starts up fine.


You might also need to disable the basic video driver. That is what qemu gives to the VM so you can access it through a console of some sort (vnc or otherwise). I had different issues with windows hanging with it enabled.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> So, i got my new cpu and mobo, was very easily able to pass thru my AMD 7850.
> However, after I install my drivers, it blues screens on reboot. Anyone got any advice on how to fix this?
> Also, if i remove the graphics card device from the VM and restart it starts up fine.


In case the other suggestions didn't work, you may want to check the following:

1. xm pci-list-assignable-devices - does the graphics card show up BEFORE running xm create ... (or xm new ...)
2. Make sure to modify the domU.cfg file (win7.cfg or whatever) after you install the driver and before rebooting Windows. You don't want your Windows ISO to show up on the disk option, and also need to change the boot option to c (I hope this is not too banal).

Good luck!

P.S.: Do you use Fedora?


----------



## blistovmhz

Quote:


> Originally Posted by *nyxcharon*
> 
> So, i got my new cpu and mobo, was very easily able to pass thru my AMD 7850.
> However, after I install my drivers, it blues screens on reboot. Anyone got any advice on how to fix this?
> Also, if i remove the graphics card device from the VM and restart it starts up fine.


EEEnteresting. Will test this.


----------



## powerhouse

I was playing with my Windows 7 domU to get sound working, and in the process "cleaned" some configurations (yellow triangles in the Windows Device Manager). Specifically I disabled the secondary VGA (the virtual Cirrus VGA I used to install Windows, before installing the Nvidia driver).

The strange thing now is that Windows, after some inactive time, seems to go into hibernation or suspend mode, and isn't able to resume. I've got 24GB RAM for Windows and not enough disk space (on the Windows partition) to store that amount. So, it doesn't recover and I have to restart and use virt-manager with it's VNC screen to tell Windows to do a normal boot and don't try to resume.

I can't think of any reason why Windows would want to hibernate - I haven't changed anything except disabling the VGA driver. Is Microsoft Windows 7 really that screwed up to change such settings on its own? Or is that another problem altogether, and I'm unjustly flaming MS?

One thing I do have to flame MS for is my media server. Windows 7 detects it fine but forces me to use Windows Media Player to connect and stream media. The last thing I want is Windows Media Player! I've digitized my entire CD, VHS, and DVD media library (we're talking 10s of thousands of $$$$ investment), and I clearly remember that Windows Media Player would even refuse to play an original high-fidelity audio CD (Mile Davis Milestones gold edition) because of "copyright infringements?". So, if anyone knows how to work around using Windows Media Player as default connecting to media servers, it would be nice. On the other hand, I can always press a button to get back to Linux where VLC and media streamer access works just fine. Sorry for venting my frustration.


----------



## powerhouse

Quote:


> Originally Posted by *blistovmhz*
> 
> EEEnteresting. Will test this.


What exactly do you want to test? I've lost you.

EDIT: Removing the graphics card from the VM? I can't see how removing the graphics card from VM (i.e. removing the PCI ID from the config file) could get you into a working (VGA passthroughed) graphics environment. It must be emulated, or not?


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> You might also need to disable the basic video driver. That is what qemu gives to the VM so you can access it through a console of some sort (vnc or otherwise). I had different issues with windows hanging with it enabled.


Hi dizzy4,

I did that (disabling the basic video driver) within Windows 7 on my PC. Somehow it causes Windows to hibernate after idle time, which causes troubles in my case (see one of my previous posts above).

I believe I'll solve that somehow.

Actually I wanted to tell you that you've got one hell of a thread going on here! Thanks for posting the how-to and for still being with us. How is your Xen VGA passthrough project going?


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> Hi dizzy4,
> I did that (disabling the basic video driver) within Windows 7 on my PC. Somehow it causes Windows to hibernate after idle time, which causes troubles in my case (see one of my previous posts above).
> I believe I'll solve that somehow.


Yeah I had to disable sleep in the power management section of the control panel. I can actually re-enable my Cirrus VGA now, but I had to disable it to get the drivers to install. The process that worked for me was: Disable VGA driver and DO NOT restart until AMD/NVidia drivers are installed.
Quote:


> Originally Posted by *powerhouse*
> 
> Actually I wanted to tell you that you've got one hell of a thread going on here! Thanks for posting the how-to and for still being with us. How is your Xen VGA passthrough project going?


It's going well. The kernel I compiled for RHEL (centOS / RedHat / Fedora) seems to be working flawlessly. I was able to modify a persistent live-cd on a USB stick to boot minimal centOS 6.3 with xen, but there are some issues. Booting an existing OS on a physical partition as created by windows install does not work. I did some reading and there seems to be a process by which you can copy the drive using dd and fdisk to change boot sectors so that qemu can manage the drives. The problem is that it isn't user friendly and does not keep the original partitions intact. I think part of the process was installing gplpv drivers before porting the partitions for qemu, so there might be a way to force qemu to work from windows, but I am skeptical.

Thank you for all the work you are doing resolving sound issues. I read somewhere that it is possible to send pulse audio over a network connection. This should be low-latency since it is all going through a bridge on the same physical system. The major problems I saw were building qemu with support for pulse, and getting the network configuration correct.

Just a FYI: qemu can emulate a simple 16-bit Sound Blaster card if you were looking for which driver to install.


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> In case the other suggestions didn't work, you may want to check the following:
> 1. xm pci-list-assignable-devices - does the graphics card show up BEFORE running xm create ... (or xm new ...)
> 2. Make sure to modify the domU.cfg file (win7.cfg or whatever) after you install the driver and before rebooting Windows. You don't want your Windows ISO to show up on the disk option, and also need to change the boot option to c (I hope this is not too banal).
> Good luck!
> P.S.: Do you use Fedora?


1. They show up fine.
Quote:


> [email protected] ~ $ sudo xm pci-list-assignable-devices
> 0000:01:00.0
> 0000:01:00.1


2.I was aware I needed to remove the iso, but what is this " change boot option to c" you mention?

3.Using Linux Mint 13. It was the first distro I got working completely on my system. Fedora and Debian both gave me way too many issues.
Quote:


> Originally Posted by *dizzy4*
> 
> You might also need to disable the basic video driver. That is what qemu gives to the VM so you can access it through a console of some sort (vnc or otherwise). I had different issues with windows hanging with it enabled.


Not sure what you mean by this/how it works.

At this point I'm pretty fed up with this to be honest. I have even tried passing through an old 8800 gs. I can install the drivers, but when I reboot, and try to access the control panel, it just tells me that no active monitors or something along that are activated. Basically it suddenly forgets the card is there, even though it shows up in the hardware screen.


----------



## dizzy4

Quote:


> Originally Posted by *nyxcharon*
> 
> 1. They show up fine.
> 2.I was aware I needed to remove the iso, but what is this " change boot option to c" you mention?
> 3.Using Linux Mint 13. It was the first distro I got working completely on my system. Fedora and Debian both gave me way too many issues.
> Not sure what you mean by this/how it works.
> At this point I'm pretty fed up with this to be honest. I have even tried passing through an old 8800 gs. I can install the drivers, but when I reboot, and try to access the control panel, it just tells me that no active monitors or something along that are activated. Basically it suddenly forgets the card is there, even though it shows up in the hardware screen.


In your device manager: right click the Microsoft Basic Display Driver (or whatever it is called for you) and disable it. DO NOT reboot like it asks. Then install your drivers and reboot when that is done.



You shouldn't need to change your boot options after you remove the .iso from the config. You should leave ' boot = "dc" ' in the config file because if there is no CD it will default to the qemu disk. All it is, is boot order so 'dc' means CD first then a drive.

What specific problems is fedora giving you?


----------



## nyxcharon

^Thanks for that. Working on that now.
Also going to try a XP machine with the nvidia card, since people have success using it with the 275.6 drivers.

Hopefully between the two I can get something working soon.









As far as fedora, I had issues just setting up xen. Willing to give it another shot though, if I can't get this worked out soon. Any specific version I should use?

Edit: Still bluescreening on startup.








Guess I'll give it one more try from a fresh install of windows.

Edit2:Fresh install of win7 -> blue screen still. Getting a atikmpag.sys error message.

Guess it's time to try a new OS for this... suggestions?
Thinking fedora. Should i use the newest release or the one in the guide?


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> 1. They show up fine.
> 2.I was aware I needed to remove the iso, but what is this " change boot option to c" you mention?
> 3.Using Linux Mint 13. It was the first distro I got working completely on my system. Fedora and Debian both gave me way too many issues.
> Not sure what you mean by this/how it works.
> At this point I'm pretty fed up with this to be honest. I have even tried passing through an old 8800 gs. I can install the drivers, but when I reboot, and try to access the control panel, it just tells me that no active monitors or something along that are activated. Basically it suddenly forgets the card is there, even though it shows up in the hardware screen.


Using Linux Mint will differ somewhat from this tutorial using Fedora. You may have seen my tutorial for Linux Mint here http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. The tutorial I've given uses two graphics cards - one for dom0/Linux and one for domU/Windows.

At this point, can you share your Windows domU config file?

What does sudo xm dmesg give you?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> Using Linux Mint will differ somewhat from this tutorial using Fedora. You may have seen my tutorial for Linux Mint here http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. The tutorial I've given uses two graphics cards - one for dom0/Linux and one for domU/Windows.
> At this point, can you share your Windows domU config file?
> What does sudo xm dmesg give you?


Your guide is actually the one i've been using. I've messed with everything so much at this point that I plan on doing a fresh install of mint tomorrow. I might have missed something through all of this, so i feel a fresh install is the way to go. I'm sure i'll be back soon to report my problems.









I'll be using two graphics cards as well, a nvidia 8800 on linux and a amd 7850 for windows. Anything I should be aware of, or might have missed before I redo all of this?


----------



## fergalom

[SOLVED] Sorry found out I need to use different vif line and pass a mac address.
vif = [ 'mac=00:16:5e:02:07:45, bridge=xenbr0, model=e1000' ]

I know this thread is mainly about Xen with a win7 DomU (which I have running perfectly for me thanks to everyone here) but I'm curious if anyone has issues with Fedora DomUs?

Used a standard config with the DomU created using xm and a fedora 17 x86-64 live iso to install on a clean 80GB logical volume

Everything installs ok but once I reboot and create the user at the initial welcome screen, it tries to start the desktop and "blackscreens" to a computer logo with "ooops, something went wrong, please inform your administrator" Also if I boot it in recovery, there is no ethernet connection. All that is listed from ifconfig is the loopback device.
This also holds true using F16 though the display does not crash but again no network adapter (from xenbr0)

Here is my cfg file

Code:



Code:


kernel = "hvmloader"
builder='hvm'
memory = 8096
name = "F17-RASP"
vcpus=6
vif = [ 'type=ioemu, bridge=xenbr0' ]
#disk = [ 'phy:/dev/vg_dom0/domUa,hda,w' , 'file:/home/xenhost/Downloads/Fedora-17.iso,hdc:cdrom,r' ]
disk = [ 'phy:/dev/vg_dom0/domUa,hda,w' ]
device_model = 'qemu-dm'
boot="cd"
sdl=0
vnc=1
vncpasswd=''
stdvga=3
serial='pty'
#pci=[ '01:00.0', '01:00.1']



Dom0 ifconfig

Code:



Code:


[[email protected] xen]# ifconfig
lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 16436
        inet 127.0.0.1  netmask 255.0.0.0
        loop  txqueuelen 0  (Local Loopback)
        RX packets 6595  bytes 742765 (725.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 6595  bytes 742765 (725.3 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

p5p1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::be5f:f4ff:fe35:1717  prefixlen 64  scopeid 0x20
        ether bc:5f:f4:35:17:17  txqueuelen 1000  (Ethernet)
        RX packets 3998  bytes 417209 (407.4 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3604  bytes 676594 (660.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device interrupt 17

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 52:54:00:11:3b:4b  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

xenbr0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.5  netmask 255.255.255.0  broadcast 192.168.1.255
        inet6 fe80::be5f:f4ff:fe35:1717  prefixlen 64  scopeid 0x20
        ether bc:5f:f4:35:17:17  txqueuelen 0  (Ethernet)
        RX packets 3845  bytes 282961 (276.3 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3599  bytes 655046 (639.6 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

[[email protected] xen]#


----------



## powerhouse

@Fergalom:

Wouldn't it be easier to use a PV version of Fedora?


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> @Fergalom:
> Wouldn't it be easier to use a PV version of Fedora?


That would be the best way to go, and on top of that PCI passthrough will work for PV guests too. I am not sure of the effect it will have on graphics cards, but I know that there are settings that allow it. The wiki says that only normal pci passthrough would and that enabling gfx_passthru=1 will not work with PV guests.


----------



## nyxcharon

@Dizzy
Is there any reson to use fedora 16 over 17 for this? I figured I should use the newest version, but if there are problems with it Xen wise I'll use 16 like in the guide. Thanks


----------



## dizzy4

Quote:


> Originally Posted by *nyxcharon*
> 
> @Dizzy
> Is there any reson to use fedora 16 over 17 for this? I figured I should use the newest version, but if there are problems with it Xen wise I'll use 16 like in the guide. Thanks


There is no reason you couldn't use fedora 17. The xen package is also 4.1.3 now, but both are compatible with this guide. I am working on getting a live-usb image going that will have everything you need already packaged (including xen 4.2).


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Your guide is actually the one i've been using. I've messed with everything so much at this point that I plan on doing a fresh install of mint tomorrow. I might have missed something through all of this, so i feel a fresh install is the way to go. I'm sure i'll be back soon to report my problems.
> 
> 
> 
> 
> 
> 
> 
> 
> I'll be using two graphics cards as well, a nvidia 8800 on linux and a amd 7850 for windows. Anything I should be aware of, or might have missed before I redo all of this?


Somehow I get the feeling you got either a graphics card issue, or one of the steps during the Windows installation are causing the problems. Here my suggestions:

1. Choose the Linux distribution you feel comfortable with. If you're used to Fedora, go the Fedora way. If you are partial to Linux Mint (like I am - sorry dizzy4), there will be a way to get it working! Let's put it this way: even I managed to get it working on Linux Mint, though I haven't found any tutorial etc. specific to LM13 Maya.
2. I don't know which success stories you found with Nvidia 8800, but check them under a magnifying glas. Most (all?) of the Nvidia VGA passthrough stories I read require a patched Xen hypervisor and extracting the BIOS from the graphics card. This means compiling Xen and most likely also the kernel (to get pciback compiled into the kernel). In other words, you will have anything but a standard Linux distribution.
EXCEPTION: if you have a "Multi-OS" Nvidia graphics adapter, you should be good to go! The cheapest one I know of is the Quadro 2000.
In short: use AMD (ATI) if you can, unless you have reports that confirm that the Nvidia 8800 you have works out-of-the-box. I'm sorry but I'm not an ATI expert, and for my needs Nvidia is better.
3. I will make an effort to make another Windows 7 installation by Friday to check and confirm the Windows installation steps. I'll post the details here.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Your guide is actually the one i've been using. I've messed with everything so much at this point that I plan on doing a fresh install of mint tomorrow. I might have missed something through all of this, so i feel a fresh install is the way to go. I'm sure i'll be back soon to report my problems.
> 
> 
> 
> 
> 
> 
> 
> 
> I'll be using two graphics cards as well, a nvidia 8800 on linux and a amd 7850 for windows. Anything I should be aware of, or might have missed before I redo all of this?


I checked the instructions again. Unfortunately my Internet is down for awhile now (writing from the iPhone), else I would be more detailed.

I created another LV for a second Windows 7 installation. Using virt-manager as VNC console, installed Windows, 1. reboot, some more stuff, 2. reboot. Now via Device Manager installed the Nvidia driver (in your case the AMD driver). Had to copy my driver disk to the PC since I had. O Internet.

After driver installation, another reboot, this time into the full screen using the new driver.

My screen switches automatically to the port with the new signal.

One more thing, if/when using two GPU cards, make sure both are properly connected to the screen. I also had a black screen recently - it turned out the DVI cable was a little loose. I almost got a heart attack when I saw the black screen and it did take me some minutes to figure it out.

Hope you find the problem and solve it.


----------



## powerhouse

@nyxcharon: Internet is finally back.

As I wrote above, the Windows installation was rather unexciting. I used virt-manager for the initial steps of installation as it provides a built-in VNC viewer.

When using virt-manager, make sure to adjust the /etc/xen/xend-config.sxp file to include "(xend-unix-server yes)".

Perhaps it will be easier with virt-manager to get it installed?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> @nyxcharon: Internet is finally back.
> As I wrote above, the Windows installation was rather unexciting. I used virt-manager for the initial steps of installation as it provides a built-in VNC viewer.
> When using virt-manager, make sure to adjust the /etc/xen/xend-config.sxp file to include "(xend-unix-server yes)".
> Perhaps it will be easier with virt-manager to get it installed?


I actually used virt-manager when I made it the first time.
So everything is clear, this is the hardware I'm using that's relevant
Intel i5 3570
Asus 7850 2gb
Asrock z77 pro3 mobo
2 monitors
I only tried the nvidia card because some people found success with rather old drivers.

The problem I'm having is as follows:
I have one monitor plugged into my onboard graphics, and another plugged into the AMD card.
Now, when I have the AMD card set as the primary graphics card, I dettach it and the screen goes black.
It does not automatically switch over to the other graphics (integrated). Sometimes I'll get what looks like Zen console output.
As such, I've been booting my computer with my onboard as the main, and working it that way, but as stated the drivers are giving me blue screens.
Not sure if this information is relevant or not. If i'm reading the guide correctly, it should switch over to onboard right?

I just made 2 new USB's with Fedora and Linux Mint.
Going to try fedora first. If I get immediate success, i'll just use that. Otherwise I'll install mint and use that until I can find the problem (Since i prefer mint).
I'll be posting in this thread later tonight with my initial success/failures for fedora. Thanks for the help so far guys.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> I actually used virt-manager when I made it the first time.
> So everything is clear, this is the hardware I'm using that's relevant
> Intel i5 3570
> Asus 7850 2gb
> Asrock z77 pro3 mobo
> 2 monitors
> I only tried the nvidia card because some people found success with rather old drivers.
> The problem I'm having is as follows:
> I have one monitor plugged into my onboard graphics, and another plugged into the AMD card.
> Now, when I have the AMD card set as the primary graphics card, I dettach it and the screen goes black.
> It does not automatically switch over to the other graphics (integrated). Sometimes I'll get what looks like Zen console output.
> As such, I've been booting my computer with my onboard as the main, and working it that way, but as stated the drivers are giving me blue screens.
> Not sure if this information is relevant or not. If i'm reading the guide correctly, it should switch over to onboard right?
> I just made 2 new USB's with Fedora and Linux Mint.
> Going to try fedora first. If I get immediate success, i'll just use that. Otherwise I'll install mint and use that until I can find the problem (Since i prefer mint).
> I'll be posting in this thread later tonight with my initial success/failures for fedora. Thanks for the help so far guys.


OK, my setup is slightly different, using only one screen. However, what you describe seems perfectly normal, i.e. your screen going blank when you detach it.

I am not aware of any mechanism that can make Linux switch to another GPU after you detach it from the initial graphics adapter.

You already pointed to the right direction:

1. Use your integrated GPU (in the 3570) as your primary graphics adapter for Xen / Linux.
2. Blacklist the driver for the secondary graphics adapter - the one you are going to use for Windows. This may not be necessary, but in my case the driver got in the way with detaching / assigning it to the Windows domU.
3. Try the AMD card first, as you got more chances for success.
4. Install Windows domU
5. After 2 Windows domU reboots, your Windows installation should be ready to install the graphics driver for your AMD graphics adapter. At this point you are still working with virt-manager and a VNC screen.
5a. Open Device manager and go to the graphics adapter section. You should have two cards listed, one with a yellow triangle. Right-click the one with the triangle and install the driver.
When doing my installation with the Nvidia card, I went to the nvidia website and let it discover my card and download the correct driver. Perhaps ATI has a similar feature on their website. Try the latest ATI driver for Windows 7 (or whatever Windows you install). Windows will probably need another reboot.
5b. After the reboot you should be seeing the initial boot sequence in the virt-manager VNC screen, until the driver loads and switches to the actual graphics card (your ATI or Nvidia card). Make sure your second screen is connected and working. The Windows login screen should appear on your second (physical) screen.
5c. In case Windows BSODs here, there is a driver issue or an issue with the graphics card - it may not support VGA passthrough. One last thing to check: Make sure VT-d is enabled (in your BIOS) and working.
Please post the output of

Code:



Code:


xm dmesg


----------



## nyxcharon

SUCCESSSSSSSSSSS









Things with fedora went much more smoothly then the last time I tried. I already had internet post-install (normally had to make a bridge in mint).
Card passed through fine, and installed drivers as per Dizzy's instructions. Only thing that isn't working is audio, but I can work that out later.

7.6/7.9 cpu and 7.9/7.9 graphics scores in windows.
Thanks so much for all your help guys!









..Time to format that old windows drive I have









Oh, and when I boot the vm it goes straight to my second monitor. So now I have one monitor for linux and one for windows. Pretty cool.


----------



## dizzy4

Quote:


> Originally Posted by *nyxcharon*
> 
> SUCCESSSSSSSSSSS
> 
> 
> 
> 
> 
> 
> 
> 
> Things with fedora went much more smoothly then the last time I tried. I already had internet post-install (normally had to make a bridge in mint).
> Card passed through fine, and installed drivers as per Dizzy's instructions. Only thing that isn't working is audio, but I can work that out later.
> 7.6/7.9 cpu and 7.9/7.9 graphics scores in windows.
> Thanks so much for all your help guys!
> 
> 
> 
> 
> 
> 
> 
> 
> ..Time to format that old windows drive I have
> 
> 
> 
> 
> 
> 
> 
> 
> Oh, and when I boot the vm it goes straight to my second monitor. So now I have one monitor for linux and one for windows. Pretty cool.


Awesome! I am really glad you were able to get this working. As of now, the audio solution I am using is Radeon HDMI audio to my monitor which has a headphone jack that I can plug my speakers into. I think it's time to look up oss stuff and qemu sound emulation.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> SUCCESSSSSSSSSSS
> 
> 
> 
> 
> 
> 
> 
> 
> Things with fedora went much more smoothly then the last time I tried. I already had internet post-install (normally had to make a bridge in mint).
> Card passed through fine, and installed drivers as per Dizzy's instructions. Only thing that isn't working is audio, but I can work that out later.
> 7.6/7.9 cpu and 7.9/7.9 graphics scores in windows.
> Thanks so much for all your help guys!
> 
> 
> 
> 
> 
> 
> 
> 
> ..Time to format that old windows drive I have
> 
> 
> 
> 
> 
> 
> 
> 
> Oh, and when I boot the vm it goes straight to my second monitor. So now I have one monitor for linux and one for windows. Pretty cool.


Congratulations and welcome to the club!

7.9 graphics is awesome.

By the way, if your disk performance index is in the 5.x-ish, try install the GPLPV drivers in Windows. It made a big difference in my case.

If you want a quick solution for sound, get a cheap (~$8) USB sound adapter and a mini-RCA cable and plug the USB adapter into one of the pass throughed USB ports. Connect the headphone output to the line-in port on your on-board sound card. It may not be the nicest solution, but it's painless and works.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> ^Thanks for that. Working on that now.
> Also going to try a XP machine with the nvidia card, since people have success using it with the 275.6 drivers.
> Hopefully between the two I can get something working soon.
> 
> 
> 
> 
> 
> 
> 
> 
> As far as fedora, I had issues just setting up xen. Willing to give it another shot though, if I can't get this worked out soon. Any specific version I should use?
> Edit: Still bluescreening on startup.
> 
> 
> 
> 
> 
> 
> 
> 
> Guess I'll give it one more try from a fresh install of windows.
> Edit2:Fresh install of win7 -> blue screen still. Getting a atikmpag.sys error message.
> Guess it's time to try a new OS for this... suggestions?
> Thinking fedora. Should i use the newest release or the one in the guide?


1. With regard to your initial BSOD problem, I read a little about the atikmpag.sys error. It seams an ATI graphics driver related issue - see for example here: http://www.overclock.net/t/1237672/display-driver-stopped-responding-atikmpag-sys-0x00000116-bsod-new-his-7950.

I think blistovmhz in post 264 was onto something. The BSOD may have been caused by the CCC (Catalyst Control Center), or something went wrong during the ATI driver installation.

I understand you did follow dizzy4's instruction to install the graphics driver in Windows (post 273) and this failed even when you did a new Windows installation?

The thing is, I can't figure out why you were able to install Windows and the driver under Fedora, but not under LM13. In both LM13 and Fedora you got to the point where you detached the graphics adapter, installed Windows, and only when installing the ATI driver - Fedora worked, the other didn't. It also seems that Windows recognized the ATI 7850 graphics card and installed the driver? In other words - VGA passthrough was working in both cases.

Can you tell if you did anything different this time, I mean when installing Windows and the graphics driver?

2. I assume you use the ATI 7850 graphics adapter for Windows 7, and the Intel CPU internal GPU for Linux?

If yes, would you mind listing your ATI 7850 as VGA passthrough compatible card under the following thread: http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters, with some details on your installation as shown in the first post (Xen hypervisor version, kernel version, setup, etc.)? It may help other folks.


----------



## nyxcharon

Hm. Looks like I spoke a little too soon.
Rebooted, and tried to dettach my card again so I could give it to the Vm.
When i dettach 1:00.0 it's fine, but on 1:00.1 it hangs.

Code:



Code:


[[email protected] ~]$ sudo xm pci-list-assignable-devices 
[sudo] password for nyxcharon: 
[[email protected] ~]$ sudo virsh nodedev-dettach pci_0000_01_00_0
Device pci_0000_01_00_0 detached

[[email protected] ~]$ sudo virsh nodedev-dettach pci_0000_01_00_1

Code:



Code:


[[email protected] ~]$ sudo xm dmesg
[sudo] password for nyxcharon: 
 __  __            _  _    _   _____   ____    __      _ _____ 
 \ \/ /___ _ __   | || |  / | |___ /  | ___|  / _| ___/ |___  |
  \  // _ \ '_ \  | || |_ | |   |_ \ _|___ \ | |_ / __| |  / / 
  /  \  __/ | | | |__   _|| |_ ___) |__|__) ||  _| (__| | / /  
 /_/\_\___|_| |_|    |_|(_)_(_)____/  |____(_)_|  \___|_|/_/

(XEN) Xen version 4.1.3 ([email protected][unknown]) (gcc version 4.7.2 20120921 (Red Hat 4.7.2-2) (GCC) ) Sun Oct 28 23:08:22 UTC 2012
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GRUB 2.00~beta4
(XEN) Command line: placeholder iommu=1 dom0_mem=1138M xen-pciback.hide=(01:00.0) (01:00.1)
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 3 MBR signatures
(XEN)  Found 3 EDD information structures
(XEN) Xen-e820 RAM map:
(XEN)  0000000000000000 - 000000000009d800 (usable)
(XEN)  000000000009d800 - 00000000000a0000 (reserved)
(XEN)  00000000000e0000 - 0000000000100000 (reserved)
(XEN)  0000000000100000 - 0000000020000000 (usable)
(XEN)  0000000020000000 - 0000000020200000 (reserved)
(XEN)  0000000020200000 - 0000000040004000 (usable)
(XEN)  0000000040004000 - 0000000040005000 (reserved)
(XEN)  0000000040005000 - 00000000ad8c9000 (usable)
(XEN)  00000000ad8c9000 - 00000000ade41000 (reserved)
(XEN)  00000000ade41000 - 00000000ae0c1000 (ACPI NVS)
(XEN)  00000000ae0c1000 - 00000000ae0c6000 (ACPI data)
(XEN)  00000000ae0c6000 - 00000000ae109000 (ACPI NVS)
(XEN)  00000000ae109000 - 00000000aeb99000 (usable)
(XEN)  00000000aeb99000 - 00000000aeff2000 (reserved)
(XEN)  00000000aeff2000 - 00000000af000000 (usable)
(XEN)  00000000af800000 - 00000000bfa00000 (reserved)
(XEN)  00000000f8000000 - 00000000fc000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000023f600000 (usable)
(XEN) ACPI: RSDP 000F0450, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT AE0A8080, 0084 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP AE0B1A50, 00F4 (r4 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT AE0A81A0, 98AA (r2 ALASKA    A M I       14 INTL 20051117)
(XEN) ACPI: FACS AE0BFF80, 0040
(XEN) ACPI: APIC AE0B1B48, 0072 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: ASF! AE0B1BC0, 00A5 (r32 INTEL       HCG        1 TFSM    F4240)
(XEN) ACPI: MCFG AE0B1C68, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: SSDT AE0B1CA8, 04A6 (r1 Intel_ AoacTabl     1000 INTL 20091112)
(XEN) ACPI: AAFT AE0B2150, 00C2 (r1 ALASKA OEMAAFT   1072009 MSFT       97)
(XEN) ACPI: HPET AE0B2218, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT AE0B2250, 036D (r1 SataRe SataTabl     1000 INTL 20091112)
(XEN) ACPI: SSDT AE0B25C0, 09AA (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT AE0B2F70, 0A92 (r1  PmRef    CpuPm     3000 INTL 20051117)
(XEN) ACPI: DMAR AE0B3A08, 00B8 (r1 INTEL      SNB         1 INTL        1)
(XEN) ACPI: BGRT AE0B3AC0, 003C (r0 ALASKA    A M I  1072009 AMI     10013)
(XEN) System RAM: 7895MB (8084492kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - ae0bff80/0000000000000000, using 32
(XEN) Processor #0 7:10 APIC version 21
(XEN) Processor #2 7:10 APIC version 21
(XEN) Processor #4 7:10 APIC version 21
(XEN) Processor #6 7:10 APIC version 21
(XEN) IOAPIC[0]: apic_id 2, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Table is not found!
(XEN) Switched to APIC driver x2apic_cluster.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3400.094 MHz processor.
(XEN) Initing memory sharing.
(XEN) Intel VT-d supported page sizes: 4kB.
(XEN) Intel VT-d supported page sizes: 4kB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Platform timer appears to have unexpectedly wrapped 1 times.
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB
(XEN) Brought up 4 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x21ac000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000230000000->0000000234000000 (263580 pages to be allocated)
(XEN)  Init. ramdisk: 000000023c99c000->000000023f5ff800
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff821ac000
(XEN)  Init. ramdisk: ffffffff821ac000->ffffffff84e0f800
(XEN)  Phys-Mach map: ffffffff84e10000->ffffffff85049000
(XEN)  Start info:    ffffffff85049000->ffffffff850494b4
(XEN)  Page tables:   ffffffff8504a000->ffffffff85077000
(XEN)  Boot stack:    ffffffff85077000->ffffffff85078000
(XEN)  TOTAL:         ffffffff80000000->ffffffff85400000
(XEN)  ENTRY ADDRESS: ffffffff81aed200
(XEN) Dom0 has maximum 4 VCPUs
(XEN) Scrubbing Free RAM: ..................................................................done.
(XEN) Xen trace buffers: disabled
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 220kB init memory.
(XEN) physdev.c:171: dom0: wrong map_pirq type 3
[[email protected] ~]$

Tried adding to the grub entry to no avail.:

Code:



Code:


[[email protected] ~]$ cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Fedora"
GRUB_DEFAULT=saved
GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 rd.lvm.lv=vg_dom0/dom0  KEYTABLE=us rd.luks=0 SYSFONT=True LANG=en_US.UTF-8 rhgb quiet"
#GRUB_THEME="/boot/grub2/themes/system/theme.txt"
GRUB_CMDLINE_XEN="iommu=1 dom0_mem=1138M xen-pciback.hide=(01:00.0) (01:00.1)"

Suggestions? Not sure why the audio portion of my card is refusing to work with me now...
The only thing that changed is I switched the DE to xfce, but I can't see how that matters...


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Hm. Looks like I spoke a little too soon.
> Rebooted, and tried to dettach my card again so I could give it to the Vm.
> When i dettach 1:00.0 it's fine, but on 1:00.1 it hangs.
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> [[email protected] ~]$ sudo xm pci-list-assignable-devices
> [sudo] password for nyxcharon:
> [[email protected] ~]$ sudo virsh nodedev-dettach pci_0000_01_00_0
> Device pci_0000_01_00_0 detached
> [[email protected] ~]$ sudo virsh nodedev-dettach pci_0000_01_00_1
> 
> Tried adding to the grub entry to no avail.:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> [[email protected] ~]$ cat /etc/default/grub
> GRUB_TIMEOUT=5
> GRUB_DISTRIBUTOR="Fedora"
> GRUB_DEFAULT=saved
> GRUB_CMDLINE_LINUX="rd.md=0 rd.dm=0 rd.lvm.lv=vg_dom0/dom0  KEYTABLE=us rd.luks=0 SYSFONT=True LANG=en_US.UTF-8 rhgb quiet"
> #GRUB_THEME="/boot/grub2/themes/system/theme.txt"
> GRUB_CMDLINE_XEN="iommu=1 dom0_mem=1138M xen-pciback.hide=(01:00.0) (01:00.1)"
> 
> Suggestions? Not sure why the audio portion of my card is refusing to work with me now...
> The only thing that changed is I switched the DE to xfce, but I can't see how that matters...


For the changes in /etc/default/grub to become effective, you need to run update-grub in Ubuntu/Mint, dizzy4 uses this for Fedora:

Code:



Code:


grub2-mkconfig > /boot/grub2/grub.cfg

EDIT: Actually, you SHOULD be able to detach the pci devices 01.00.0 and 01.00.1 within grub, since Fedora should have pciback compiled into the kernel. See here http://wiki.xen.org/wiki/Xen_PCI_Passthrough#Static_assignment_for_built-in_xen-pciback for instructions.

If you can't detach the graphics card (audio) with virsh or via grub, you may want to try the pciback script here:

Code:



Code:


#!/bin/sh
    #
    ### BEGIN INIT INFO
    # Provides: pciback
    # Required-Start: $local_fs
    # Required-Stop: $local_fs
    # Should-Start:
    # Should-Stop:
    # X-Start-Before: xend
    # X-Stop-After:
    # Default-Start: 2 3 4 5
    # Default-Stop:
    ### END INIT INFO

    . /lib/lsb/init-functions
    scriptname=/etc/init.d/pciback
    configfile=/etc/xen/pciback.conf

    unbind() {
    # Unbind a PCI function from its driver as necessary
    [ ! -e /sys/bus/pci/devices/$1/driver/unbind ] || \
    echo -n $1 > /sys/bus/pci/devices/$1/driver/unbind
    }

    bind() {
    # Add a new slot to the PCI Backend's list
    echo -n $1 > /sys/bus/pci/drivers/pciback/new_slot
    # Now that the backend is watching for the slot, bind to it
    echo -n $1 > /sys/bus/pci/drivers/pciback/bind
    }

    case $1 in

    start)
    cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
    unbind $line
    bind $line
    done
    ;;

    stop|status|restart|force-reload)
    # As we don't know which driver was bound before, there is not much we can do here
    ;;

    *)
    echo "Usage: $scriptname start" >&2
    exit 3
    ;;

    esac

save it in /etc/init.d and make it executable. Run

Code:



Code:


update-rc.d pciback defaults 21 19

for Ubuntu/Mint or "chkconfig" (check for exact syntax) for Fedora to edit the Sys V runlevel.

Then create the config file for the script:

Code:



Code:


# This file contains the list of pci(e) devices the pciback driver should bind
    # to at system startup.
    # Please provide the devices in long BDF notation.
    # Example:
    # 0000:08:00.0
    # NO EMPTY LINES!
    #
    0000:01:00.0
    0000:01:00.1

and save it as *pciback.conf* under /etc/xen/.

Finally edit the /etc/modules file and add on a separate line:

Code:



Code:


xen-pciback passthrough=1

After you reboot the PC the graphics card should be detached and available for the Windows domU. Check with xm pci-list-assignable-devices.

The script and /etc/modules entry make it permanent, that is each time you boot your graphics card 01.00.0 / 01.00.1 will be available for Windows (or another domU).

I don't know if and how the procedure described by dizzy4 using virt-manager detaches the PCI devices after the next reboot?

Good luck! I'm sure you will eventually iron out these things and get a rock-solid Windows virtual gaming (or whatever) machine, with all the goodies of Linux and Xen.


----------



## powerhouse

Actually, you should be able to detach the pci devices 01.00.0 and 01.00.1 within grub, since Fedora should have pciback compiled into the kernel.

See here http://wiki.xen.org/wiki/Xen_PCI_Passthrough#Static_assignment_for_built-in_xen-pciback for instructions.

Please try this first!


----------



## nyxcharon

So i tried both to no avail.

I attempted to modprobe pciback, can't find it.

Code:



Code:


[[email protected] ~]$ sudo modprobe pciback
FATAL: Module pciback not found.

After trying that script, I noticed in the startup it failed, told me to check this:

Code:



Code:


[[email protected] ~]$ systemctl status pciback.service
pciback.service
          Loaded: loaded (/etc/rc.d/init.d/pciback)
          Active: failed (Result: exit-code) since Tue, 13 Nov 2012 07:17:58 -0500
         Process: 801 ExecStart=/etc/rc.d/init.d/pciback start (code=exited, status=1/FAILURE)
          CGroup: name=systemd:/system/pciback.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.

Running it myself, I get this:

Code:



Code:


[[email protected] ~]$ sudo /etc/rc.d/init.d/pciback start
[sudo] password for nyxcharon: 
/etc/rc.d/init.d/pciback: line 15: /lib/lsb/init-functions: No such file or directory

Being curious, I just commented it out to see what happened.
I get this:

Code:



Code:


[[email protected] ~]$ sudo /etc/rc.d/init.d/pciback start
/etc/rc.d/init.d/pciback: line 27: /sys/bus/pci/drivers/pciback/new_slot: No such file or directory
/etc/rc.d/init.d/pciback: line 29: /sys/bus/pci/drivers/pciback/bind: No such file or directory

At which point it just hangs. So, I'm having a couple issues now.








I still can't figure out why It just stopped working for no reason.

Not sure what is the best approach to take at this point.

Edit: Also noticed this:

Code:



Code:


[[email protected] ~]$ systemctl status systemd-modules-load.service 
systemd-modules-load.service - Load Kernel Modules
          Loaded: loaded (/usr/lib/systemd/system/systemd-modules-load.service; static)
          Active: failed (Result: exit-code) since Tue, 13 Nov 2012 07:33:16 -0500
            Docs: man:systemd-modules-load.service(8)
                  man:modules-load.d(5)
         Process: 404 ExecStart=/usr/lib/systemd/systemd-modules-load (code=exited, status=1/FAILURE)
          CGroup: name=systemd:/system/systemd-modules-load.service

Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
[[email protected] ~]$ sudo /usr/lib/systemd/systemd-modules-load 
[sudo] password for nyxcharon: 
Failed to find module 'xen-pciback passthrough=1'

I have a file named pciback.conf in /etc/modules-load.d/

One final thing I found weird.

Code:



Code:


[[email protected] ~]$ sudo xm pci-attach win7a 01:00.0
Error: pci: improper device assignment specified: pci: 0000:01:00.1 must be co-assigned to the same guest with 0000:01:00.0, but it is not owned by pciback or pci-stub.
Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func> [virtual slot]

Insert a new pass-through pci device.
[[email protected] ~]$ sudo xm pci-attach win7a 01:00.1
Error: pci: PCI Backend and pci-stub don't own device 0000:01:00.1
Usage: xm pci-attach [-o|--options=<opt>] <Domain> <domain:bus:slot.func> [virtual slot]

Insert a new pass-through pci device.

Am I missing something here?


----------



## tuffy12345

Woah...this thread is cool.


----------



## powerhouse

_About "sudo modprobe pciback" - module not found:
_
I edited my reply above (and posted another one). If I'm not mistaken, Fedora has pciback in the kernel, so you don't need to load the module. Just in case, the command to load the pciback module is "sudo modprobe xen-pciback". To check if it's loaded as a module, run "lsmod | grep pciback" , the output should be "xen_pciback".

_About: $ systemctl status pciback.service
pciback.service
Loaded: loaded (/etc/rc.d/init.d/pciback)
Active: failed (Result: exit-code) since Tue, 13 Nov 2012 07:17:58 -0500
Process: 801 ExecStart=/etc/rc.d/init.d/pciback start (code=exited, status=1/FAILURE)
CGroup: name=systemd:/system/pciback.service

and

sudo /etc/rc.d/init.d/pciback start
[sudo] password for nyxcharon:
/etc/rc.d/init.d/pciback: line 15: /lib/lsb/init-functions: No such file or directory_

My mistake. I totally forgot that Fedora might use different paths etc., so the script must be adapted to Fedora. Unfortunately I'm not familiar with Fedora, sorry.

In addition, if Fedora uses pciback compiled into the kernel there is no need for editing the /etc/modules file to add xen-pciback... I believe this error:

_Failed to find module 'xen-pciback passthrough=1'_

points to it.

_Error: pci: improper device assignment specified: pci: 0000:01:00.1 must be co-assigned to the same guest with 0000:01:00.0, but it is not owned by pciback or pci-stub._

This means it (0000:01:00.1 - the audio part) hasn't been detached. The Xen wiki explains it much better than I can: http://wiki.xen.org/wiki/Xen_PCI_Passthrough#Preparing_a_device_for_passthrough

Yes, it is confusing sometimes, and I apologize for having caused more confusion. Here are some links that might be useful:

https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#PCI_device_assignmen
http://tavi-tech.blogspot.co.il/2012/03/vga-passthrough-with-kvm-on-fedora-16.html - see 7.1. for some hints, though this one is for KVM.
http://xen.1045712.n5.nabble.com/PCI-Passthrough-td5711612.html - talks about problems with newest Fedora release / kernel and your graphics card, but not sure it's very helpful.
http://wiki.xen.org/wiki/Comprehensive_Xen_Debian_Wheezy_PCI_Passthrough_Tutorial - yeah, it's Debian, but very detailed.

By the way, did you blacklist the ATI driver? It might be the ATI graphics card driver under Linux that gets in the way with detaching your card.


----------



## powerhouse

What does

Code:



Code:


lspci -k

give you? Any driver assigned to the graphics card you try to pass through?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> What does
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> lspci -k
> 
> give you? Any driver assigned to the graphics card you try to pass through?


Quote:


> [[email protected] ~]$ sudo lspci -k
> 00:00.0 Host bridge: Intel Corporation Ivy Bridge DRAM Controller (rev 09)
> Subsystem: ASRock Incorporation Device 0150
> Kernel driver in use: agpgart-intel
> 00:01.0 PCI bridge: Intel Corporation Ivy Bridge PCI Express Root Port (rev 09)
> Kernel driver in use: pcieport
> 00:02.0 VGA compatible controller: Intel Corporation Ivy Bridge Graphics Controller (rev 09)
> Subsystem: ASRock Incorporation Device 0152
> Kernel driver in use: i915
> 00:14.0 USB Controller: Intel Corporation Panther Point USB xHCI Host Controller (rev 04)
> Subsystem: ASRock Incorporation Device 1e31
> Kernel driver in use: xhci_hcd
> 00:16.0 Communication controller: Intel Corporation Panther Point MEI Controller #1 (rev 04)
> Subsystem: ASRock Incorporation Device 1e3a
> 00:1a.0 USB Controller: Intel Corporation Panther Point USB Enhanced Host Controller #2 (rev 04)
> Subsystem: ASRock Incorporation Device 1e2d
> Kernel driver in use: ehci_hcd
> 00:1b.0 Audio device: Intel Corporation Panther Point High Definition Audio Controller (rev 04)
> Subsystem: ASRock Incorporation Device 7892
> Kernel driver in use: snd_hda_intel
> 00:1c.0 PCI bridge: Intel Corporation Panther Point PCI Express Root Port 1 (rev c4)
> Kernel driver in use: pcieport
> 00:1c.4 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
> 00:1c.5 PCI bridge: Intel Corporation Panther Point PCI Express Root Port 6 (rev c4)
> Kernel driver in use: pcieport
> 00:1c.6 PCI bridge: Intel Corporation Panther Point PCI Express Root Port 7 (rev c4)
> Kernel driver in use: pcieport
> 00:1d.0 USB Controller: Intel Corporation Panther Point USB Enhanced Host Controller #1 (rev 04)
> Subsystem: ASRock Incorporation Device 1e26
> Kernel driver in use: ehci_hcd
> 00:1f.0 ISA bridge: Intel Corporation Panther Point LPC Controller (rev 04)
> Subsystem: ASRock Incorporation Device 1e44
> 00:1f.2 SATA controller: Intel Corporation Panther Point 6 port SATA AHCI Controller (rev 04)
> Subsystem: ASRock Incorporation Device 1e02
> Kernel driver in use: ahci
> 00:1f.3 SMBus: Intel Corporation Panther Point SMBus Controller (rev 04)
> Subsystem: ASRock Incorporation Device 1e22
> *01:00.0 VGA compatible controller: ATI Technologies Inc Device 6819
> Subsystem: ASUSTeK Computer Inc. Device 042c
> Kernel driver in use: pci-stub
> 01:00.1 Audio device: ATI Technologies Inc Device aab0
> Subsystem: ASUSTeK Computer Inc. Device aab0*
> 02:00.0 VGA compatible controller: nVidia Corporation G86 [GeForce 8400 GS] (rev a1)
> Subsystem: Jaton Corp Device 0422
> Kernel driver in use: nouveau
> 03:00.0 PCI bridge: ASMedia Technology Inc. Device 1080 (rev 03)
> 04:01.0 RAID bus controller: VIA Technologies, Inc. VT6421 IDE RAID Controller (rev 50)
> Subsystem: VIA Technologies, Inc. VT6421 IDE RAID Controller
> Kernel driver in use: sata_via
> 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168B PCI Express Gigabit Ethernet controller (rev 06)
> Subsystem: ASRock Incorporation Motherboard (one of many)
> Kernel driver in use: r8169
> 06:00.0 Multimedia video controller: Conexant Systems, Inc. CX23887/8 PCIe Broadcast Audio and Video Decoder with 3D Comb (rev 04)
> Subsystem: Hauppauge computer works Inc. Device 2259
> Kernel driver in use: cx23885


What's the next step boss?








Thanks for all the help so far.









EDIT:
So, found a "workaround"
Boot into gnome3, dettach the card (both 01:00.0 and 01:00.1)
Log out, log into xfce. Dettach the card again. Re-assgn to the VM.
Boot VM

No idea of why this works. I thought it might have something to do with gnome services, so I checked that startup box for them in Xfce, but that didn't help at all. But hey, if it works it works









Sidenote: Is it possible to passthrough my motherboard audio? If not, i'll just go the cheap soundcard route as mentioned before.
Nevermind. Passed through my audio from my motherboard with no problems, just had to do it from gnome3.









For those curious, it's
00:1B:0 Panther Point High Definition Audio Controller
Looks like I made a good choice of motherboard.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> What's the next step boss?
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks for all the help so far.
> 
> 
> 
> 
> 
> 
> 
> 
> EDIT:
> So, found a "workaround"
> Boot into gnome3, dettach the card (both 01:00.0 and 01:00.1)
> Log out, log into xfce. Dettach the card again. Re-assgn to the VM.
> Boot VM
> No idea of why this works. I thought it might have something to do with gnome services, so I checked that startup box for them in Xfce, but that didn't help at all. But hey, if it works it works
> 
> 
> 
> 
> 
> 
> 
> 
> Sidenote: Is it possible to passthrough my motherboard audio? If not, i'll just go the cheap soundcard route as mentioned before.
> Nevermind. Passed through my audio from my motherboard with no problems, just had to do it from gnome3.
> 
> 
> 
> 
> 
> 
> 
> 
> For those curious, it's
> 00:1B:0 Panther Point High Definition Audio Controller
> Looks like I made a good choice of motherboard.


Wow, I love the creativity you are showing here! Seriously, I'd never tried booting into gnome3, detach graphic card, logout/login to xfce etc.

Your lspci -k shows that somethings not working with assigning 01.00.1 (audio) to pci-stub. Can you check that there is no mistake in the command that detaches 01:00.1 ?

For reference, here is my lspci -k output:

Code:



Code:


02:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
        Subsystem: NVIDIA Corporation Device 084a
        Kernel driver in use: pciback
        Kernel modules: nouveau, nvidiafb
02:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation Device 084a
        Kernel driver in use: pciback
        Kernel modules: snd-hda-intel

My graphics card for passthrough to Windows is 02:00.0/1.
As you can see, I use pciback whereas you use pci-stub.

Have you blacklisted your ATI driver under Linux dom0?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> Wow, I love the creativity you are showing here! Seriously, I'd never tried booting into gnome3, detach graphic card, logout/login to xfce etc.
> Your lspci -k shows that somethings not working with assigning 01.00.1 (audio) to pci-stub. Can you check that there is no mistake in the command that detaches 01:00.1 ?
> For reference, here is my lspci -k output:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> 02:00.0 VGA compatible controller: NVIDIA Corporation GF106GL [Quadro 2000] (rev a1)
> Subsystem: NVIDIA Corporation Device 084a
> Kernel driver in use: pciback
> Kernel modules: nouveau, nvidiafb
> 02:00.1 Audio device: NVIDIA Corporation GF106 High Definition Audio Controller (rev a1)
> Subsystem: NVIDIA Corporation Device 084a
> Kernel driver in use: pciback
> Kernel modules: snd-hda-intel
> 
> My graphics card for passthrough to Windows is 02:00.0/1.
> As you can see, I use pciback whereas you use pci-stub.
> Have you blacklisted your ATI driver under Linux dom0?


I'm using the same command to dettach both, minus one having .0 and the other .1.
I'll try blacklisting the generic ati driver for fedora and see where that gets me.

Sidenote: So i can dettach my motherboard audio and pass it through to the vm. When I'm done with the vm, how can I re-attach it to my host again so I have sound?
I tried sudo xm pci-attach Domain-0 00:1b.0
The command attached it with no complaints, but i still have no sound

Tip: If you have some very bad mouse lag in your VM, I suggest chaging usb-device from mouse to tablet in your xen config. Worked wonders for me.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> I'm using the same command to dettach both, minus one having .0 and the other .1.
> I'll try blacklisting the generic ati driver for fedora and see where that gets me.
> Sidenote: So i can dettach my motherboard audio and pass it through to the vm. When I'm done with the vm, how can I re-attach it to my host again so I have sound?
> I tried sudo xm pci-attach Domain-0 00:1b.0
> The command attached it with no complaints, but i still have no sound
> Tip: If you have some very bad mouse lag in your VM, I suggest chaging usb-device from mouse to tablet in your xen config. Worked wonders for me.


Regarding gnome3 versus xfce, maybe dmesg (or other /var/log files) will give some clues as to what happens when xfce is loaded, versus gnome3. The only time I had issues with detaching a PCI device was when the nouveau (Nvidia) driver got in the way - blacklisting it solved the problem once and for all. I read a little about blacklisting under Fedora - it looks complicated. In Debian/Ubuntu/Linux Mint it's an entry in a file. But then I'm very unfamiliar with Fedora.

With regard to your motherboard audio: It's not enough to reattach it to dom0, you probably need to load some driver to make it work. lspci -k should come in handy for this to determine which driver (module) is needed and which one is assigned to the PCI device. Like here:

Code:



Code:


Kernel driver in use: pciback
Kernel modules: snd-hda-intel

I can't remember having tried to "reuse" a PCI device, but the Xen wiki link I posted in one of the above posts talks about this.

Thanks for the tip re mouse! It's funny, I've used "tablet" in my win7.cfg file, but actually never used this feature. I have discrete USB hosts/ports for dom0 (Linux) and domU (Windows) and switch between them using a KVM switch (KVM means keyboard video mouse for those who confuse it with KVM virtualization - we're living in the era of acronyms - anyone who guesses what EB2B means gets a free beer in the city I live).

nyxcharon: Have you solved the Windows 7 domU BSOD issue now? I mean, is the (creative) solution you found OK with you? If yes you may want to look at the posts on my how-to to find ways of backing up your dom0 and/or domU, that is, if you use LVM.


----------



## nyxcharon

The solution? It's simple enough. I just made a simple bash script that runs on startup for both gnome and xfce, so i log in to gnome 3 and then immediately back out., then into xfce and then I'm good to go. One day I'm sure i'll figure out the root of all the problems.
With regards to the BSOD: I've never had one on fedora. Not sure exactly what was causing them in mint.

As far as backing up, I'll be working on that soon. On the topic of lvm(Which I am indeed using), is there anyway I can pass files through to this windows VM? I've read some people have tried to mount the lvm partiton when not in use, but I've been unsuccessful with that. ssh, ftp, scp, etc have all gotten me no where. Starting to think I might need to make a dedicated interface on the network for this, but not sure.

Lastly, with the audio: I think I'm just going to buy a cheap pci soundcard and pass that through, then just grab a Y splitter and feed both my linux and windows audio into a set of speakers. Seems liked the ideal solution for this.

Thanks for all the help!


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> The solution? It's simple enough. I just made a simple bash script that runs on startup for both gnome and xfce, so i log in to gnome 3 and then immediately back out., then into xfce and then I'm good to go. One day I'm sure i'll figure out the root of all the problems.
> With regards to the BSOD: I've never had one on fedora. Not sure exactly what was causing them in mint.
> As far as backing up, I'll be working on that soon. On the topic of lvm(Which I am indeed using), is there anyway I can pass files through to this windows VM? I've read some people have tried to mount the lvm partiton when not in use, but I've been unsuccessful with that. ssh, ftp, scp, etc have all gotten me no where. Starting to think I might need to make a dedicated interface on the network for this, but not sure.
> Lastly, with the audio: I think I'm just going to buy a cheap pci soundcard and pass that through, then just grab a Y splitter and feed both my linux and windows audio into a set of speakers. Seems liked the ideal solution for this.
> Thanks for all the help!


Re script: You are crazy








. Loading gnome3 to get xfce working with Windows domU passthrough? Com'on, isn't there any better way?

To pass files or have access to Xen domU files (Windows) you can use *kpartx*. Mount the Xen Windows partition in read only mode!

Here is some more on kpartx: http://forums.linuxmint.com/viewtopic.php?f=42&t=111783

You can even mount image (backup) files with kpartx. See above link, second post.

Audio: If you're not too discerning about audio quality from your Windows domU, you can use a cheap $8 USB audio controller. Of course, a PCI card should be better. For me the USB method is good enough, but I don't intend to play music using Windows. (I still need to find an adequate sound card or USB adapter that compares with my CD (audio) player. Let's don't get into audio stuff, it's very controversial.)

Please report on the sound card you chose - particularly on it's sound quality.

P.S.: I know this is a computer/Linux forum, but I must mention the fantastic Ion and Deva series amplifiers I have and am using. I also met the Welsh-born (?) developer of this line of audio equipment , he's fantastic!!!!


----------



## nyxcharon

Thanks for the instructions on using kpartx, worked great.

As far as audio, I'll be sure to post back on my experience once I get something working. I'll more then likely grab a $15-20 one off newegg and see where that gets me.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Thanks for the instructions on using kpartx, worked great.
> As far as audio, I'll be sure to post back on my experience once I get something working. I'll more then likely grab a $15-20 one off newegg and see where that gets me.


I'm glad kpartx worked for you!

Audio: It depends on what you need. The $x cards/USB adapters are good enough for movies. Wanna hear classic/jazz/whatever music - pay for a good sound card.


----------



## nyxcharon

Yeah. I just want it so I have audio for my games, so I wont be needing anything special.


----------



## huzzug

sorry but, i did not go thru all the posts, but just the guide, and wanted to know if i can make my machine a VM ??

if so how ??


----------



## powerhouse

Quote:


> Originally Posted by *huzzug*
> 
> sorry but, i did not go thru all the posts, but just the guide, and wanted to know if i can make my machine a VM ??
> if so how ??


It's relatively easy to run Windows in a VM, for example using Virtualbox on a Linux computer. BUT, if you want native or near-native graphics performance under Windows, for example for gaming, a normal VM isn't good enough. dizzy4's how-to talks about installing Windows on a Xen hypervisor / Linux machine with VGA passthrough, which means that Windows will have direct access to your graphics card and thus full/native/bare-metal performance.

If that is what you want, you can post the exact hardware specs so we may be able to tell more about the chance that you make it. Specifically, the following hardware components are needed:

1. CPU - the EXACT specification
2. Motherboard
3. Graphics adapter(s)

If the hardware supports VT-d (or IOMMU) and you got a graphics card that supports pass-through then chances are good. If not, either get suitable hardware or forget about it.

P.S.: Many AMD and some Nvidia cards support pass-through, though most of the Nvidia cards require more than this how-to describes (including Xen hypervisor patches, kernel compilations, and extracting the video BIOS from the graphics card). If you got the right hardware, it's easy. Otherwise, it ranges from demanding to really demanding or impossible.

P.S.S.: If the CPU/motherboard don't support VT-d / IOMMU, it won't work.


----------



## huzzug

sig updated


----------



## dizzy4

Quote:


> Originally Posted by *huzzug*
> 
> sig updated


As was stated in the first post, you will need a non-k chip (excluding the c2 stepping of 3930k and 3960x).

You will also need a motherboard that supports vt-d. Please consult the manual to find out if it can be enabled. Since it says v-pro, there will most likely be an option. Vt-d is required to be v-pro rated. You can do this guide with a different processor. www.ark.intel.com


----------



## dizzy4

well, either my video card died or the motherboard pci-e slot did... I got minimal and live centos live-usb images to work with xen which was cool. From there allmusers would have to do is install it on the hard disk. I am still working on some scripts, but I have a working kernel compiled that is very stable. I had it running for weeks without errors. I am sure it would go longer, but my video card died and I had to shutdown to assess the situation. I am looking into a completely new setup and salvaging this system for family or to part out. I was really hoping to wait for haswell to upgrade, but I may try something sooner.


----------



## Gen. Gustav Zenlav

Anyone looking at my sig system know if if this is possible for my rig?


----------



## dizzy4

Quote:


> Originally Posted by *Gen. Gustav Zenlav*
> 
> Anyone looking at my sig system know if if this is possible for my rig?


Pretty sure it is. Look into your bios. If you see a vt-d option it should be since it is fully dependent on the motherboard northbridge on your setup. I seem to remember the big bang boards having a looooot of features too so there is a good chance.


----------



## Gen. Gustav Zenlav

Thanks now i have some tinkering to do


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> well, either my video card died or the motherboard pci-e slot did... I got minimal and live centos live-usb images to work with xen which was cool. From there allmusers would have to do is install it on the hard disk. I am still working on some scripts, but I have a working kernel compiled that is very stable. I had it running for weeks without errors. I am sure it would go longer, but my video card died and I had to shutdown to assess the situation. I am looking into a completely new setup and salvaging this system for family or to part out. I was really hoping to wait for haswell to upgrade, but I may try something sooner.


Sorry to hear the video card died, or the PCI slot. Isn't there a way to check and replace the faulty component and continue? It sounds like you are going to start with a new system.

Anyway, hope things work out.


----------



## powerhouse

Quote:


> Originally Posted by *Gen. Gustav Zenlav*
> 
> Thanks now i have some tinkering to do


I gather from your signature that you have a i7 920 and a Nvidia graphics card.

Regarding CPU, check out http://ark.intel.com and look for *VT-d*. If it's available (and can be enabled in your BIOS), it's a good starting point.

Regarding your Nvidia graphics card, it may work, though it could be demanding. Most AMD cards are easier to handle, except Nvidia "multi-OS" cards like the Quadro 2000 and some more expensive models.

@dizzy4: Have you got any new information on VGA passthrough for Nvidia cards? Has there been any progress with the hypervisor (i.e. 4.2)?


----------



## N0BOX

Does anyone know if it would be possible to install Linux/Xen to a separate drive (I have a 120GB SSD that I could wipe and install to), then create a DomU that has control of the Intel motherboard RAID controller so that I can convert my physical Windows 7 install into a VM? It would be extemely excellent if that was actually possible, but I'm not sure if you can allow a VM to have the physical RAID array without having a true hardware RAID controller.

I'll experiment with Xen whether I can work out a way to convert the physical system into a VM anyway, but it would certainly make it easier to convert to Xen if there was a way to give the RAID array to one of the VMs. If I like the way Xen works, then I'll grab an AMD GPU or two.


----------



## B4dn4z0

I everyone here
I just found this really interesting thread, i didn't read everything but i'm sure it's full of usefull info









I'm not new to virtualisation, i've done alot of it over VMWare Softs, under windowson my own and under thier unix system for servers for work, but i'm new to xen and the concept of pci passthrough.
I prefer Debian rather than Fedora, so I managed to created my own config, it's VT-x Enabled, but unfortunatly i brought a i5 2500K a year ago, when i've built my computer a year ago.

This is my actual Configuration :
CPU
Intel i5 2500 (K) @ 3.3 Ghz

Motherboard
GigabyteGA-Z68MA-D2H-B3 (BIOS Version : F9 OFW) (non VT-d Enabled)

RAM
Crucial Ballistix DDR3 2 x 2 Go PC12800

GPU
ATI - Radeon HD5770
Intel - HD3000

Storage
SSD - OCZ Vertex 4 128GB
HDD - WD Velociraptor 150GB
HDD - WD Scorpio Black 250Go (Almost Dead)

In the next days, I'm going to get a 2500 non-K from a friend that just brought one and didn't get the K version so i'll be able to test the VT-d. I'll have to get a modified version from my BIOS but i heard on the internet that a guy on the Gigabyte forums have the possibility to create custom bios.

I Also recently brought 2 screens recently, curently one is on my Intel CPU, and the 2 others are on the HD5770, I'm gonna upgrade for a HD7770, but since ATI GPU seems to be fine in xen, that's not a problem. So my main problem now is about my triple-screen. As far as you know it guys, what will I be capable of doing with 3 screen ?

In the end, I would like to have a Dual-Screen Linux to dev some softs in Ruby, a Single screen for for my Media (One of my screen is actually a TV) probably under Linux, and another VM under windows for Gaming that will use the Velociraptor for games storage and that could be either on the TV for multiplayer games with controlers or on my main screen.


----------



## N0BOX

Quote:


> Originally Posted by *B4dn4z0*
> 
> In the end, I would like to have a Dual-Screen Linux to dev some softs in Ruby, a Single screen for for my Media (One of my screen is actually a TV) probably under Linux, and another VM under windows for Gaming that will use the Velociraptor for games storage and that could be either on the TV for multiplayer games with controlers or on my main screen.


You can give one graphics card to one VM. That means that you could use the iGPU in the i5-2500 for one screen (or multiple screens if your motherboard happens to have multiple DVI or HDMI outputs) and you could use the 5770 (or the 7770 if you replace the 5770) for two or three more screens. If your motherboard has room for more than one GPU, you could use both the 5770 and the 7770, which would allow you to have screens set up for three different VMs*.

The bottom line is that one GPU is owned by one VM. If one GPU can support multiple monitors, then you can hook those monitors up to the VM that controls the GPU. You can't give one screen on a GPU to one VM and another screen on the same GPU to a different VM. I haven't heard anything about whether one VM can have control of multiple GPUs, though that is theoretically possible. As I understand it, no one has reported on whether SLI or CrossFireX works.

* I'm not sure if Xen is capable of giving away multiple GPUs to multiple VMs. I assume it can, but I've read conflicting stories, so I can't be sure. It may be that I was reading that you can't give multiple GPUs to a single VM, or I may have been reading specifically about an Nvidia problem that doesn't exist with AMD GPUs. I have not actually tried to set up a Xen machine, so I can only discuss "theory".


----------



## B4dn4z0

Quote:


> Originally Posted by *N0BOX*
> 
> You can give one graphics card to one VM. That means that you could use the iGPU in the i5-2500 for one screen (or multiple screens if your motherboard happens to have multiple DVI or HDMI outputs) and you could use the 5770 (or the 7770 if you replace the 5770) for two or three more screens. If your motherboard has room for more than one GPU, you could use both the 5770 and the 7770, which would allow you to have screens set up for three different VMs*.


Okay, For the moment, i have a 2 digital output on the embedded and 3 on the 5770, and maybe 4 more if i remove my audio card to keep the old graphics and the new one (Mini-ATX board).

Once a VM is linked to a graphic card, what really happend ? Does it go fullscreen on the screens that are connected to the card ? Am i still in within th window of my vnc client, like another VM ? or should i directly go start writing a tool to attach/deattach GPU to VMs and launching them direclty fullscreen in xserver on my Dom0 Host ?


----------



## dizzy4

Quote:


> Originally Posted by *B4dn4z0*
> 
> Okay, For the moment, i have a 2 digital output on the embedded and 3 on the 5770, and maybe 4 more if i remove my audio card to keep the old graphics and the new one (Mini-ATX board).
> Once a VM is linked to a graphic card, what really happend ? Does it go fullscreen on the screens that are connected to the card ? Am i still in within th window of my vnc client, like another VM ? or should i directly go start writing a tool to attach/deattach GPU to VMs and launching them direclty fullscreen in xserver on my Dom0 Host ?


for this guide the virtual machine actually takes ownership of the card. The interrupts are all rerouted to the virtual machine so it would essentially own the video card. This means that the dom0 will not have access to the card until it has been released and rebound to pciback (a linux module). The video card could then be reassigned to a virtual machine (including the host). Xen is a type 1 or baremetal hypervisor which means that the hypervisor runs on the hardware and the "host" (dom0) is actually just a virtual machine with permission to access the hypervisor and the hardware.

So to answer your question, you would not be in a VNC window. It will display full screen. A VNC window can be used to attach to guests, but the performance will be poor because the VNC will only attach to emulated video hardware. However, if you were to set up a VNC server within windows and access it from the network it would perform and display as would any VNC over the network on a physical machine.


----------



## N0BOX

Yeah, like dizzy4 said, the VMs can be given full control of the GPU, which would mean that the OS in the VM has the whole/full screen. You should give control of the GPUs to the VMs that need (near-)native GPU performance, like those that are used for gaming (or [email protected]/etc, for that matter).

I'd set up a GPU for a Windows guest for the purpose of gaming and media playback (since a lot of codecs can take advantage of the GPU for playback), and any VMs that are set up for the purpose of running servers could be accessed via VNC or SSH (which would be a lot easier than detaching and reattaching GPUs among VMs).


----------



## B4dn4z0

Maybe there is anther solution around connecting my TV to multiple graphic adapter. I've 1 DVI and 2 HDMI on this screen. it may be some more cable, but hey, that's already a mess behind my desk ^^

I'll certainly hook up my 3 screens to my 7770 to be able to use Eyefinity, and also hook my TV on the 5770. In this way, i'll be able to start a 3-screen optimized windows gaming VM, a linux VM to work on 2 Screens (without passthrough ?), a VM for multimedia (on the HD5770), and a single screen gaming VM (witch i should be able to choose on which graphic adapter i want it)

Correct ?

And how is managed the focus of the mouse/keybord when pci-passthrough is enabled ?


----------



## N0BOX

Mouse/Keyboard swapping can be done in a couple of different ways. It is possible to swap USB devices between VMs relatively easily... it works similarly to unplugging a USB device and plugging it into another PC as far as the VMs are concerned.

Another option is to use a program called Synergy (free and open source), which allows you to share one keyboard and mouse with any number of computers (VMs) over the network. Since all the computers are on a "network" that exists completely within your computer you don't have to worry about excessive latency, though it would make sense to use the VM on which you do the most intense gaming as the Synergy "server" (that is, that VM has the devices connected to it, and other VMs connect to that VM to receive their commands when they have focus).


----------



## powerhouse

I tried Synergy. On one side it's the ultimate solution, on the other side it can cause havoc. Here is what I found out with Synergy:

1. It works nicely as long as you don't use YouTube or try to do adminstrative stuff under Windows. If you try either of it it will freeze. I tried "elevated" but it didn't work. Will try some more.
2. Given the caveats above, copy/paste between say dom0 and domU works perfect. Only for that I will try find a solution to above problems.
3. I installed the server on the dom0 Linux Mint VM. This is what I boot into and the domU Windows is only booted when needed.
4. I luckily have a KVM switch and passed through some USB ports to always have "physical" access to the hardware, both from Linux and Windows.

Some earlier post says "(near) native" access to the graphics adapter that has been passed through to the Windows guest (domU). Well, all that I've seen is that the passed through video card performs just like native when running Windows in the domU. I'm using the Windows domU to upload the LUT adjustments to the LCD screen via the DVI cable. In short, this means that my Windows domU (VM) has direct access to the graphics hardware. My graphics card (Nvidia Quadro 2000) is also able to display a wide color gamut (more than 8 bit per color or 24 bit for RGB) such as 12 bit and it shows.

I have no experience with gaming under a Windows domU but for graphics and photo/video editing it's perfect. It's 100% native performance!


----------



## evermooingcow

Synergy (client) should work in UAC prompts, login screens, etc. Try opening the config as administrator and registering the client to start as a service if you have not tried that. Newer versions may handle it differently. installing as a service enables some extended functions.


----------



## nyxcharon

So, back with my experiences after using this for a month or so.

1.Got my audio sorted. I originally got this pci card:
http://www.newegg.com/Product/Product.aspx?Item=N82E16829180004
It didn't work on linux or windows, so I gave up on that. Then settled for this:
http://www.newegg.com/Product/Product.aspx?Item=N82E16829128004
which works great. I then have a Y splitter running both my motherboard audio on linux and the usb adapter from windows into my speakers, so I have audio on both at the same time.
Seems to work out really well so far, though my overall audio output level has gotten cut because of it (somehow).

2.I have noticed the longer my windows VM runs, the worse my mouse input gets. I normally leave it on since it has it's own monitor with my setup, and about at the 10-12 hour mark it stops inputting correctly and becomes unusable. I just ended up passing through my USB ports on the front of my case and have my wireless mouse plugged into it. Works nicely.

3.Still have some issues with storage. I created a second disk for storage for my VM using virt-manager. Windows sees it and it shows up in device manager, but not in explorer. Not sure why that is. Haven't had time to work out a solution for it yet.

Overall, this is a great solution for me, and I'm pretty happy with it so far. Performance is amazing, and other then the occasionally odd bug(like the mouse) I haven't had any issues. I've been getting used to fedora over debian(after 6 years of debian) and it's been alright








Thanks for the guide, and all the help I got.!









At some point, I'd like to configure a second network adapter for both linux and windows with a static ip, and then run a samba server on linux and connect to it with the windows VM. It's really nice for transferring files, i used to do it in virtualbox with good results.


----------



## dizzy4

Glad to see it's working so well for you. I don't have as many glitches, but it's a different situation and setup. I am happy to say I have my system working again, but I am having some storage issues as well. I suspect a hardware, not software issue though. Another odd issue I have run into is lack of internet connection to dom0. DomU has functional ethernet and internet connections, but something is disallowing dom0 from access. I need to check my network scripts again, but networking with Xen has always been a problem for me.

Just more things to fix for a new release.


----------



## nyxcharon

Networking problems? Strange.
I had some problems networking with xen in debian, but in fedora 17 "it just works". It was pre-bridged for me and I haven't had to touch it since.


----------



## Eggs and bacon

i read through the guide, but I am not sure why VT-d is needed?
I am getting a 3570k so I assume this wont work for me, right? or would it just have reduced performance?

also why don't the K processors have it?

thanks


----------



## nyxcharon

Vt-d is needed because it allows for VGA pass through, which is the whole point of setting up a xen system. (in this context)
Getting a k series will guarantee you wont be able to do this.
As far as why? That's an intel decision.


----------



## Eggs and bacon

Quote:


> Originally Posted by *nyxcharon*
> 
> Vt-d is needed because it allows for VGA pass through, which is the whole point of setting up a xen system. (in this context)
> Getting a k series will guarantee you wont be able to do this.
> As far as why? That's an intel decision.


thanks, Looks like I will need to try another solution to do my gaming.


----------



## nyxcharon

Do you plan on overclocking? If not, the 3570 nonk is a great choice for this kind of setup, it's what I use.


----------



## Eggs and bacon

Quote:


> Originally Posted by *nyxcharon*
> 
> Do you plan on overclocking? If not, the 3570 nonk is a great choice for this kind of setup, it's what I use.


Yeah I did plan on over clocking, I might use vmware workstation or just dual boot.


----------



## powerhouse

Quote:


> Originally Posted by *Eggs and bacon*
> 
> Yeah I did plan on over clocking, I might use vmware workstation or just dual boot.


I'm not familiar with VMware workstation, but does it compare with Xen? I mean, does it even offer VGA passthrough?

As to the Intel CPUs with VT-d support, here is a list: http://ark.intel.com/search/advanced?VTD=true

But, strangely my 3930K is missing in this list, and it DOES have VT-d (the C2 stepping version).

I would go with one of the VT-d CPUs, even if you don't plan to use it now. You never know. I can't explain how happy I am that I don't have to dual-boot anymore.

Good luck!


----------



## powerhouse

Quote:


> Originally Posted by *evermooingcow*
> 
> Synergy (client) should work in UAC prompts, login screens, etc. Try opening the config as administrator and registering the client to start as a service if you have not tried that. Newer versions may handle it differently. installing as a service enables some extended functions.


Thanks! I need to look into it. Haven't had the time now.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> So, back with my experiences after using this for a month or so.
> 1.Got my audio sorted. I originally got this pci card:
> http://www.newegg.com/Product/Product.aspx?Item=N82E16829180004
> It didn't work on linux or windows, so I gave up on that. Then settled for this:
> http://www.newegg.com/Product/Product.aspx?Item=N82E16829128004
> which works great. I then have a Y splitter running both my motherboard audio on linux and the usb adapter from windows into my speakers, so I have audio on both at the same time.
> Seems to work out really well so far, though my overall audio output level has gotten cut because of it (somehow).
> 2.I have noticed the longer my windows VM runs, the worse my mouse input gets. I normally leave it on since it has it's own monitor with my setup, and about at the 10-12 hour mark it stops inputting correctly and becomes unusable. I just ended up passing through my USB ports on the front of my case and have my wireless mouse plugged into it. Works nicely.
> 3.Still have some issues with storage. I created a second disk for storage for my VM using virt-manager. Windows sees it and it shows up in device manager, but not in explorer. Not sure why that is. Haven't had time to work out a solution for it yet.
> Overall, this is a great solution for me, and I'm pretty happy with it so far. Performance is amazing, and other then the occasionally odd bug(like the mouse) I haven't had any issues. I've been getting used to fedora over debian(after 6 years of debian) and it's been alright
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks for the guide, and all the help I got.!
> 
> 
> 
> 
> 
> 
> 
> 
> At some point, I'd like to configure a second network adapter for both linux and windows with a static ip, and then run a samba server on linux and connect to it with the windows VM. It's really nice for transferring files, i used to do it in virtualbox with good results.


1. Glad you sorted out audio. I used the same method, though a different USB adapter.

2. I'm using a KVM and a dedicated passed through USB host for the Windows domU. So I switch between a non-passed through USB host for Linux and the other for Windows. It works great, but adds to the cable clutter.
I started to experiment with Synergy and this could be the ultimate solution, if only there weren't these issues with Youtube video and UAC admin screen in Windows. Once I find a little time I hope to make it work.

3. I've never used virt-manager to create a new storage device. I use LVM, create a PV, assign it to a VG and create a LV, but don't format it. Then just edit my win7.cfg file and add the new volume as:

Code:



Code:


disk = [ 'phy:/dev/mapper/lm13-win7,hda,w' , 'phy:/dev/mapper/photos-photo1_stripe,hdb,w' , 'phy:/dev/mapper/photos-photos_raw,hdc,w' ]

As you can see, I have 3 disks: lm13-win7 (the Windows system drive on a SSD); photos-photo1_stripe which is a striped LV using 2 physical disks (for RAID-like performance); and photos-photos_raw, a regular LV used for backup copies of my RAW photos. It works great.

I also have a Samba server under Linux and created a link under Windows to a shared directory. It also works nice, though I think the transfer speed could be better.


----------



## N0BOX

Quote:


> Originally Posted by *powerhouse*
> 
> I'm not familiar with VMware workstation, but does it compare with Xen? I mean, does it even offer VGA passthrough?
> As to the Intel CPUs with VT-d support, here is a list: http://ark.intel.com/search/advanced?VTD=true
> But, strangely my 3930K is missing in this list, and it DOES have VT-d (the C2 stepping version).
> I would go with one of the VT-d CPUs, even if you don't plan to use it now. You never know. I can't explain how happy I am that I don't have to dual-boot anymore.
> Good luck!


VMWare Workstation doesn't support VT-d, but you could have a Windows 7 host (on which you install your graphics-intensive applications/games) and boot your other Linux/Windows guests in Workstation. This is how my machine is currently configured. I have a Windows 7 Ultimate x64 host with 32GB of memory, and I always boot a Linux Mint 13 x64 guest when I start the machine up. The Mint guest runs MythTV for me (thank the electronic gods for the HDHomeRun tuners!) among other fun linuxy applications. I also have a Win 7 Home Premium VM and a few Linux VMs of various distributions that I spin up for short periods of time depending on what I need to do.

I would prefer to have a Windows 7 Ultimate x64 and Mint Linux 14 x64 VM running on Xen than having to depend on the stability of Windows and VMWare Workstation to get my TV shows recorded by MythTV on my Mint VM. Until I get the chance to shuffle around several terabytes worth of data, that's just the way things are going to have to stay.


----------



## N0BOX

Quote:


> Originally Posted by *nyxcharon*
> 
> 3.Still have some issues with storage. I created a second disk for storage for my VM using virt-manager. Windows sees it and it shows up in device manager, but not in explorer. Not sure why that is. Haven't had time to work out a solution for it yet.


The reason your new disk doesn't show up in Explorer is because it hasn't yet been partitioned and formatted. What you need to do is go to the Control Panel, open Computer Management, and then select the Disk Management option in the panel on the left. This will bring up a graphical representation of all the disks you have attached to your Windows machine. You can select the new disk, partition it, format it to NTFS, and optionally set a drive letter and volume name of your choosing for it. Once it is formatted, it will show up in Explorer.


----------



## dizzy4

Quote:


> Originally Posted by *N0BOX*
> 
> The reason your new disk doesn't show up in Explorer is because it hasn't yet been partitioned and formatted. What you need to do is go to the Control Panel, open Computer Management, and then select the Disk Management option in the panel on the left. This will bring up a graphical representation of all the disks you have attached to your Windows machine. You can select the new disk, partition it, format it to NTFS, and optionally set a drive letter and volume name of your choosing for it. Once it is formatted, it will show up in Explorer.


I am still looking into getting disks ported to windows, but it is a long term goal. The way Qemu handles the disks is different than the way windows natively handles it, so it is a little daunting. I assume the best way to port over storage disks is to set up a samba share from the dom0. That way you don't have to reformat and lose data. Unfortunately I don't think this will work for system disks. There is supposedly a way to get system disks to work by installing the pv-ops drivers before trying to port it to a virtual machine then changing some boot sectors.


----------



## powerhouse

Quote:


> Originally Posted by *N0BOX*
> 
> VMWare Workstation doesn't support VT-d, but you could have a Windows 7 host (on which you install your graphics-intensive applications/games) and boot your other Linux/Windows guests in Workstation. This is how my machine is currently configured. I have a Windows 7 Ultimate x64 host with 32GB of memory, and I always boot a Linux Mint 13 x64 guest when I start the machine up. The Mint guest runs MythTV for me (thank the electronic gods for the HDHomeRun tuners!) among other fun linuxy applications. I also have a Win 7 Home Premium VM and a few Linux VMs of various distributions that I spin up for short periods of time depending on what I need to do.
> I would prefer to have a Windows 7 Ultimate x64 and Mint Linux 14 x64 VM running on Xen than having to depend on the stability of Windows and VMWare Workstation to get my TV shows recorded by MythTV on my Mint VM. Until I get the chance to shuffle around several terabytes worth of data, that's just the way things are going to have to stay.


Looks like a nice setup, though I've lost my trust in Windows and would not want to run Windows as primary OS. You can use Samba to share disks between dom0 and domU, perhaps it solves the issue of shuffling around lots of data.

On the other hand, I transfered my entire photo collection (~2 TB) from several native NTFS drives to a striped LVM volume consisting of 2 2TB drives accessed from within the Windows domU. It wasn't much of an effort, but requires spare HDs.


----------



## powerhouse

Quote:


> Originally Posted by *N0BOX*
> 
> The reason your new disk doesn't show up in Explorer is because it hasn't yet been partitioned and formatted. What you need to do is go to the Control Panel, open Computer Management, and then select the Disk Management option in the panel on the left. This will bring up a graphical representation of all the disks you have attached to your Windows machine. You can select the new disk, partition it, format it to NTFS, and optionally set a drive letter and volume name of your choosing for it. Once it is formatted, it will show up in Explorer.


You hit the nail on the top! I totally forgot that part in my reply.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> I am still looking into getting disks ported to windows, but it is a long term goal. The way Qemu handles the disks is different than the way windows natively handles it, so it is a little daunting. I assume the best way to port over storage disks is to set up a samba share from the dom0. That way you don't have to reformat and lose data. Unfortunately I don't think this will work for system disks. There is supposedly a way to get system disks to work by installing the pv-ops drivers before trying to port it to a virtual machine then changing some boot sectors.


Samba is one option, but you may want to try kpartx under Linux. Here is a short howto I wrote: http://forums.linuxmint.com/viewtopic.php?f=42&t=111783. There is really nothing much to it.

See also the second post - a short script to backup your Windows domU.


----------



## powerhouse

Double posted.


----------



## Ghost26

hi all !

I saw that thread and decided to post my experience in virtualization.

When I bought my PC a year ago, I've also searched on VT-d ready hardware. But that's quite hard to get.

First of all, you have to get wether a Xeon or a non-K CPU. That removes the fun you get when overclocking and running an overclocked PC. Sure, it guarantees you the most stable and the highest reliability ever, but you have to think it twice. Getting 1 GHz overclock on Sandy/Ivy Bridge CPU is common and the performance gain is not negligeable.

Second, you have to get the motherboard that is actually VT-d ready. This is another story. In fact, motherboard supporting this feature are rare. When I was looking for that, only the Intel Q Express chipset supported the VT-d since it is business oriented chipset. OR, I could get the Asus C206/C216 P8B/P8C WS motherboard. once again, no overclocking, standard workstation class motherboard, no cool feature. Only a motherboard thats works and performs normally, anything extraordinary here.

So, with all this, I ended up with a P8P67 WS Revolution and Core i7 2700K. No VT-d, but 1 GHz overclocking, magnificient power efficiency and general system stability, a huge board with which you can have fun, overclock, and build a powerhouse machine on it.

I mainly use VMWare Workstation for virtualization purpose. I haven't tried Xen yet, but I think it is an hypervisor in the same style of VMWare ESXi right ? Anyway, I'm used with VMWare Workstation and barely know it by hearth.

So that was my two cents


----------



## N0BOX

Quote:


> Originally Posted by *Ghost26*
> 
> hi all !
> I saw that thread and decided to post my experience in virtualization.
> When I bought my PC a year ago, I've also searched on VT-d ready hardware. But that's quite hard to get.
> First of all, you have to get wether a Xeon or a non-K CPU. That removes the fun you get when overclocking and running an overclocked PC.


There is always the option of the C2-stepping Sandy Bridge-E chips (i7-3820, i7-3930K, i7-3960X). They all support VT-d, and they all support overclocking (the 3820 isn't "unlocked", but it's multiplier is "partially unlocked"... it allows you to go to a certain point, but not near as far as the "K" and "X" chips).

I have the i7-3930K, and my Asus P9X79 Deluxe motherboard supports VT-d. The way I understand it, many of the X79 chipset boards do support VT-d. These are the perfect boards and chips for consumer virtualization, especially if you go for the full 64GB or 96GB of RAM that some of the boards support and use the 6-core (12-thread / 12 vCPU) chips.


----------



## nyxcharon

Quote:


> Originally Posted by *N0BOX*
> 
> The reason your new disk doesn't show up in Explorer is because it hasn't yet been partitioned and formatted. What you need to do is go to the Control Panel, open Computer Management, and then select the Disk Management option in the panel on the left. This will bring up a graphical representation of all the disks you have attached to your Windows machine. You can select the new disk, partition it, format it to NTFS, and optionally set a drive letter and volume name of your choosing for it. Once it is formatted, it will show up in Explorer.


I've tried that. I get a "The request could not be performed because of an I/O device error." Searching online yields two answers: get a new power adapter or replace the disk. Neither one makes sense in this context


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> I've tried that. I get a "The request could not be performed because of an I/O device error." Searching online yields two answers: get a new power adapter or replace the disk. Neither one makes sense in this context


Can you share the config file line? Example:

Code:



Code:


disk = [ 'phy:/dev/mapper/lm13-win7,hda,w' , 'phy:/dev/mapper/photos-photo1_stripe,hdb,w' , 'phy:/dev/mapper/photos-photos_raw,hdc,w' ]

If I recall correctly, the disk shouldn't be formatted (NO file system) once you're done with the lvcreate command (I assume you use LVM).

EDIT: You mentioned you used virt-manager. I never used it for creating/editing a VM. I find it easier to edit the /etc/xen/win7.cfg or whatever file directly (see code above). Make a backup copy of the config file, to be save.

EDIT2: What is your hardware?


----------



## powerhouse

Quote:


> Originally Posted by *N0BOX*
> 
> There is always the option of the C2-stepping Sandy Bridge-E chips (i7-3820, i7-3930K, i7-3960X). They all support VT-d, and they all support overclocking (the 3820 isn't "unlocked", but it's multiplier is "partially unlocked"... it allows you to go to a certain point, but not near as far as the "K" and "X" chips).
> I have the i7-3930K, and my Asus P9X79 Deluxe motherboard supports VT-d. The way I understand it, many of the X79 chipset boards do support VT-d. These are the perfect boards and chips for consumer virtualization, especially if you go for the full 64GB or 96GB of RAM that some of the boards support and use the 6-core (12-thread / 12 vCPU) chips.


+1 for the C2 stepping Sandy Bridge-E chips. My i7-3930K works nicely with VT-d.

With regard to the Asus motherboard:

1. The Marvell SATA controller has a bug. I posted a bug report re Marvell and here the response from a kernel dev:
Quote:


> This bug affects vanilla kernels up to and including 3.6.3.
> 
> The problem seems to be a design issue with the Marvell controller. With
> VT-d enabled, each device gets its own "view" of memory it can get
> access to. The Marvell chip only registers one device per SATA port, but
> actually uses more than one device. It's this second phantom device that
> is not allowed memory access when VT-d is enabled.
> 
> It may eventually be worked around in the kernel, but it's not an easy
> fix. AFAIK the current thinking is to register the phantom device
> automatically when the first one is found.
> 
> Also see https://bugzilla.kernel.org/show_bug.cgi?id=42679
> 
> ** Bug watch added: Linux Kernel Bug Tracker #42679
> http://bugzilla.kernel.org/show_bug.cgi?id=42679


So make sure you're not using the Marvell controller. In my case, at some stage of installing the system the Xen hypervisor wouldn't even boot with the Marvell SATA enabled, however, a regular kernel (w/o Xen) would boot OK.

2. Which BIOS release do you run on the Asus motherboard? Someone on the Asus forum reported a broken VT-d for BIOS release 2104 of the Asus X79 Sabertooth board. I wrote to Asus techsup and they denied the broken VT-d, but in the same response they also stated that they don't support Linux !!!
I can't confirm the VT-d bug, as I haven't tried to upgrade the BIOS. The upgrade is supposedly irreversible.

I would greatly appreciate if you could let us know the BIOS release you are using with your Asus P9X79 Deluxe motherboard. By the way, did you manage to run Xen / Linux dom0 with a Windows domU on your hardware?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> Can you share the config file line? Example:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> disk = [ 'phy:/dev/mapper/lm13-win7,hda,w' , 'phy:/dev/mapper/photos-photo1_stripe,hdb,w' , 'phy:/dev/mapper/photos-photos_raw,hdc,w' ]
> 
> If I recall correctly, the disk shouldn't be formatted (NO file system) once you're done with the lvcreate command (I assume you use LVM).
> EDIT: You mentioned you used virt-manager. I never used it for creating/editing a VM. I find it easier to edit the /etc/xen/win7.cfg or whatever file directly (see code above). Make a backup copy of the config file, to be save.
> EDIT2: What is your hardware?


So i got rid of the disk I made with virt-manager and just made another volume using lvcreate. Now when I try to initialize it, I get it's write protected. Here's the line from my config:

disk = [ 'phy:/dev/vg_dom0/domUa,hda,w' 'phy:/dev/vg_dom0/lvol0,hdb,w' ]

Got it, this link solved it:
http://support.microsoft.com/kb/971436
Note: My drive wasn't actually listed as read only in disk part, but clearing the attribute again fixed it. Oh windows, i'll never understand you


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> So i got rid of the disk I made with virt-manager and just made another volume using lvcreate. Now when I try to initialize it, I get it's write protected. Here's the line from my config:
> disk = [ 'phy:/dev/vg_dom0/domUa,hda,w' 'phy:/dev/vg_dom0/lvol0,hdb,w' ]
> Got it, this link solved it:
> http://support.microsoft.com/kb/971436
> Note: My drive wasn't actually listed as read only in disk part, but clearing the attribute again fixed it. Oh windows, i'll never understand you


Oh yeah, it sometimes surprises me that Windows is still around







.

Your config line is missing a comma:

Code:



Code:


disk = [ 'phy:/dev/vg_dom0/domUa,hda,w' , 'phy:/dev/vg_dom0/lvol0,hdb,w' ]

So you are using the MS virtual disk format. How does it perform? Any advantages to it over NTFS?

By the way, is there any Linux support for this format? The reason I'm asking is because I use kpartx to access my Windows domU partitions under Linux dom0, and mount them as NTFS. I need to read data from the Windows domU to my Linux dom0 and kpartx provides a real nice solution for that, without a need for a running Windows domU and Windows share (or Samba on the Linux side).


----------



## dizzy4

Quote:


> Originally Posted by *nyxcharon*
> 
> Networking problems? Strange.
> I had some problems networking with xen in debian, but in fedora 17 "it just works". It was pre-bridged for me and I haven't had to touch it since.


When you say this, do you mean it is using eth0 as the bridge itself? I followed standard procedure to get my xen bridge, which was to create a xenbr0 bridge in network-scripts. I actually have time today to work on the guide, so I am hoping to get this resolved so I can get a release out tonight.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> When you say this, do you mean it is using eth0 as the bridge itself? I followed standard procedure to get my xen bridge, which was to create a xenbr0 bridge in network-scripts. I actually have time today to work on the guide, so I am hoping to get this resolved so I can get a release out tonight.


I don't use network-scripts. You can configure the bridge in the /etc/network/interfaces file (first make a backup of the file!):

Code:



Code:


auto lo
iface lo inet loopback

auto xenbr0

iface xenbr0 inet static
bridge_ports eth0
address 192.168.1.160
broadcast 192.168.1.255
netmask 255.255.255.0
gateway 192.168.1.1
bridge_stp off          # disable Spanning Tree Protocol
bridge_waitport 0       # no delay before a port becomes available
bridge_fd 0             # no forwarding delay

auto eth0

Above is the config for static IP. Here is for DHCP:

Code:



Code:


auto lo
iface lo inet loopback

auto xenbr0
iface xenbr0 inet dhcp
bridge_ports eth0

auto eth0

After editing the interfaces file, run:

Code:



Code:


/etc/init.d/networking restart

The above is based on Debian/Ubuntu, you may need to tweak it to work with Fedora.

I also uncommented the (vif-script vif-bridge) entry in /etc/xen/xend-config.sxp, though this should not be necessary. In fact, defining the network bridge in the interfaces file should make the network-script do nothing.

I once had a problem with the network, after playing with the taskbar network setup utility, and had to remove the network-manager altogether. Since then no problems at all. So if you have the network-manager package installed, it's best to remove it.

Good luck!


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> I don't use network-scripts. You can configure the bridge in the /etc/network/interfaces file (first make a backup of the file!):
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> auto lo
> iface lo inet loopback
> auto xenbr0
> iface xenbr0 inet static
> bridge_ports eth0
> address 192.168.1.160
> broadcast 192.168.1.255
> netmask 255.255.255.0
> gateway 192.168.1.1
> bridge_stp off          # disable Spanning Tree Protocol
> bridge_waitport 0       # no delay before a port becomes available
> bridge_fd 0             # no forwarding delay
> auto eth0
> 
> Above is the config for static IP. Here is for DHCP:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> auto lo
> iface lo inet loopback
> auto xenbr0
> iface xenbr0 inet dhcp
> bridge_ports eth0
> auto eth0
> 
> After editing the interfaces file, run:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> /etc/init.d/networking restart
> 
> The above is based on Debian/Ubuntu, you may need to tweak it to work with Fedora.
> I also uncommented the (vif-script vif-bridge) entry in /etc/xen/xend-config.sxp, though this should not be necessary. In fact, defining the network bridge in the interfaces file should make the network-script do nothing.
> I once had a problem with the network, after playing with the taskbar network setup utility, and had to remove the network-manager altogether. Since then no problems at all. So if you have the network-manager package installed, it's best to remove it.
> Good luck!


Yeah thanks. I may need to just do it this way. I have been really unhappy in general with the way redhat based systems handle networking. I think I will just need to disable network manager and just manually add everything. Even with the flag 'NM_CONTROLLED = no' there are issues. I also might try returning my scripts to defaults and see if it knows how to sort itself out.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Yeah thanks. I may need to just do it this way. I have been really unhappy in general with the way redhat based systems handle networking. I think I will just need to disable network manager and just manually add everything. Even with the flag 'NM_CONTROLLED = no' there are issues. I also might try returning my scripts to defaults and see if it knows how to sort itself out.


It's not only Redhat. I couldn't get the bridge working on the Ubuntu/Debian system no matter what I tried, until I removed the network-manager package. Strangely enough everything had worked at the beginning, only when I changed something in the configuration, there was no way going back.


----------



## Chamezz

Could this be used for gaming on say, a terminal remote-accessing the VM?

Say I have a mini-itx computer in room A, could I use it to play on the VM in room B via remote access, or would there be too much delay over LAN?


----------



## dizzy4

Quote:


> Originally Posted by *Chamezz*
> 
> Could this be used for gaming on say, a terminal remote-accessing the VM?
> Say I have a mini-itx computer in room A, could I use it to play on the VM in room B via remote access, or would there be too much delay over LAN?


I think there is a way for it to work. You would need a lot of bandwidth and some compression, but citrix uses the xen hypervisor to do remote VM stuff. I am starting to look into XCP (Xen Cloud Platform) which has a lot of cool features and would make what you are suggesting easier. There are even special workstation cards aimed at client virtualization that can give hardware acceleration to several virtual machines simultaneously. The system uses a server and what they call a thin client. The thin client would only have to have a gigabit lan connection, keyboard, mouse, speakers, a monitor and a very small USB or hard drive for a small linux install.

Of course I have absolutely no idea how this gets set up... yet.


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> It's not only Redhat. I couldn't get the bridge working on the Ubuntu/Debian system no matter what I tried, until I removed the network-manager package. Strangely enough everything had worked at the beginning, only when I changed something in the configuration, there was no way going back.


I am a big dork.... I had STP disabled on my bridge







Everything is working now. I am still bypassing network manager though. Now all I need to do is create a kickstart for my live-xen distribution and do some small mods to the image to get xen to boot first.


----------



## Chamezz

Quote:


> Originally Posted by *dizzy4*
> 
> I think there is a way for it to work. You would need a lot of bandwidth and some compression, but citrix uses the xen hypervisor to do remote VM stuff. I am starting to look into XCP (Xen Cloud Platform) which has a lot of cool features and would make what you are suggesting easier. There are even special workstation cards aimed at client virtualization that can give hardware acceleration to several virtual machines simultaneously. The system uses a server and what they call a thin client. The thin client would only have to have a gigabit lan connection, keyboard, mouse, speakers, a monitor and a very small USB or hard drive for a small linux install.
> Of course I have absolutely no idea how this gets set up... yet.


Would be pretty sweet if it was possible. Virtualize a Windows gaming PC and a OSX workstation.


----------



## dizzy4

Due to a lack of proper documentation I have not been able to get my custom live cd to be created yet. I think I have worked out all the syntax, but for some reason am getting a mounting error during the process. Currently I am trying to add to an existing .iso using livecd-crator --base-on rather than creating a new one from scratch.

Code:



Code:


mount: block device /dev/loop1 is write-protected, mounting read-only
mount: wrong fs type, bad option, bad superblock on /dev/loop0,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

/usr/lib/python2.6/site-packages/imgcreate/errors.py:40: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  return str(self.message)
/usr/lib/python2.6/site-packages/imgcreate/errors.py:45: DeprecationWarning: BaseException.message has been deprecated as of Python 2.6
  return unicode(self.message)
Error creating Live CD : Failed to loopback mount '/releases/tmp/imgcreate-8F1TSW/tmp-mlXtVc/ext3fs.img' : Failed to mount '/dev/loop0' to '/releases/tmp/imgcreate-8F1TSW/install_root'

and dmesg output

Code:



Code:


EXT3-fs (loop0): error: couldn't mount because of unsupported optional features (240)

It looks like the program is not using a proper mount type for the existing live-cd I am trying to add things to. I suppose I could do it manually and then recreate the .iso manually as well. This means setting up a proper environment into which yum can install packages and where I can add a script or two to handle booting properly. I could always try making my own live-cd from scratch (which I have done for fedora 16 before, but not centos) or instruct users on how to modify their own live-usb, but I really want to streamline this process to make further releases easy for me and the other users.

Any ideas?


----------



## N0BOX

Is it possible that you're trying to mount an ext4 fs as ext3?

I'm just taking a stab in the dark with that question. I have never dealt with fedora's liveCDs. Ubuntu builds the whole filesystem inside of a squashfs container. It's possible that your filesystem cannot be mounted by whatever bootloader is used by the liveCD, or that the kernel packaged with that liveCD doesn't have the filesystem built into the kernel statically (not as a module). I'm not sure if it's really possible to extract a liveCD, add stuff to it, and have it end up unable to load the filesystem unless you are trying to create a liveCD that creates a Xen dom0 + hypervisor from the CD... The Xen hypervisor's "kernel" may not be able to load the filesystem that the liveCD uses.


----------



## dizzy4

Quote:


> Originally Posted by *N0BOX*
> 
> Is it possible that you're trying to mount an ext4 fs as ext3?
> I'm just taking a stab in the dark with that question. I have never dealt with fedora's liveCDs. Ubuntu builds the whole filesystem inside of a squashfs container. It's possible that your filesystem cannot be mounted by whatever bootloader is used by the liveCD, or that the kernel packaged with that liveCD doesn't have the filesystem built into the kernel statically (not as a module). I'm not sure if it's really possible to extract a liveCD, add stuff to it, and have it end up unable to load the filesystem unless you are trying to create a liveCD that creates a Xen dom0 + hypervisor from the CD... The Xen hypervisor's "kernel" may not be able to load the filesystem that the liveCD uses.


The way livecd-creator works is that it uses a squashfs container to make the installation. My issue has nothing to do with booting the livecd either. The program is supposed to be able to open the livecd and add packages and features with a kickstart file. I am not having any issues mounting the .iso manually, but you actually might have hit the nail on the head. The loops might not be working because the partition I am using for temporary storage is ext3 and the system drive (where /dev/loop resides) is ext4. Could this be tricking the program?

I did manually mount and copy the contents to a working folder. I suppose I could tell yum to use that as an environment, but I really don't know how to go about doing that.


----------



## chenxiaolong

Quote:


> Originally Posted by *dizzy4*
> 
> The way livecd-creator works is that it uses a squashfs container to make the installation. My issue has nothing to do with booting the livecd either. The program is supposed to be able to open the livecd and add packages and features with a kickstart file. I am not having any issues mounting the .iso manually, but you actually might have hit the nail on the head. The loops might not be working because the partition I am using for temporary storage is ext3 and the system drive (where /dev/loop resides) is ext4. Could this be tricking the program?
> I did manually mount and copy the contents to a working folder. I suppose I could tell yum to use that as an environment, but I really don't know how to go about doing that.


Could you post the kickstart file you are using? Maybe I can give it a try.

I don't think it's the best idea to call yum manually, but If you want yum to install to a specific root/mount point, you would run:

Code:



Code:


yum --installroot=/tmp/chroot install <...>

The respective rpm command is:

Code:



Code:


rpm --root=/tmp/chroot -ivh <...>.rpm

If you do this, note that the repositories and distro version are taken from the host, not the chroot.


----------



## alcal

What kind of a performance hit do you take when gaming on a VM over gaming on a regular windows install? Is there any way for "MOAR CORES" to improve gaming performance past what you could get with your ordinary bottleneck-free k-sku processor?

Translation: Is there any good reason to do this if you aren't hellbent on using Linux? Is there an advantage for someone who is already happy with just plain windows?


----------



## dizzy4

Quote:


> Originally Posted by *chenxiaolong*
> 
> Could you post the kickstart file you are using? Maybe I can give it a try.
> I don't think it's the best idea to call yum manually, but If you want yum to install to a specific root/mount point, you would run:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> yum --installroot=/tmp/chroot install <...>
> 
> The respective rpm command is:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> rpm --root=/tmp/chroot -ivh <...>.rpm
> 
> If you do this, note that the repositories and distro version are taken from the host, not the chroot.


I am using the --base-on=livecd.iso flag which supposedly uses the existing live cd iso. The idea of this flag is mainly to add packages and features not included with the original live cd. There is no documentation of its usage or how to write a kickstart for it. From what I gather, the kickstart only needs to contain the extra packages, repos, flags and scripts. The kickstart is not the issue since I already have a working livecd.iso.... the issue is getting the livecd-creator to handle the mounting properly.


----------



## dizzy4

Quote:


> Originally Posted by *alcal*
> 
> What kind of a performance hit do you take when gaming on a VM over gaming on a regular windows install? Is there any way for "MOAR CORES" to improve gaming performance past what you could get with your ordinary bottleneck-free k-sku processor?
> Translation: Is there any good reason to do this if you aren't hellbent on using Linux? Is there an advantage for someone who is already happy with just plain windows?


Typical performance hit is less than 5%. In most cases it is well below 1% or the margin of error is too great to accurately see a performance hit at all. This makes it simple to keep machines completely separate, but more importantly lets you run them simultaneously. For instance: a gaming pc and an htpc running on the same hardware at the some time.

I also hate the whole "bottleneck" sensationalism... Most of the modern core processors will not create a huge difference in performance at all. My i7-2600 might have less ghz than a lot of overclocked k series chips, but it games really well still. Most of the performance issues in games are GPU based and then the only time bottleneck applies is to multi-gpu solutions where there are physically less connections that can be made. Even at that, pci-e [email protected] is almost always enough bandwidth and with 3.0 is always enough. Sure some games are more processor intensive so overclocking helps a bit, but those systems cant also be running another full gaming vm or htpc at the same time









So basically moar cores = more virtual machines. There issue really is in how many threads the game itself has and how many it can use. Paravirtualization might increase performance a little as you suggest, but is much harder to do right.

It's not for everyone, but it sure is cool technology. Besides, C2 stepping 2011 cpus can overclock and do this.


----------



## N0BOX

Quote:


> Originally Posted by *dizzy4*
> 
> The way livecd-creator works is that it uses a squashfs container to make the installation. My issue has nothing to do with booting the livecd either. The program is supposed to be able to open the livecd and add packages and features with a kickstart file. I am not having any issues mounting the .iso manually, but you actually might have hit the nail on the head. The loops might not be working because the partition I am using for temporary storage is ext3 and the system drive (where /dev/loop resides) is ext4. Could this be tricking the program?
> I did manually mount and copy the contents to a working folder. I suppose I could tell yum to use that as an environment, but I really don't know how to go about doing that.


Theoretically, Linux should be able to handle any number of different filesystem types mounted, regardless of whether they are a mix of ext3/4 and compressed container loop mounts... There are only two possible problems I can imagine having: doing a copy operation that preserves some kind of attribute that doesn't exist in the filesystem you are copying files to (say like trying to copy a file from an ext4 partition to an exfat partition while preserving file ownership and permissions?) and trying to copy files into the squashfs filesystem... I believe you have to extract the whole filesystem, chroot it, modify it, and then recompress it back to squashfs.

Other than that, I can't make a guess as to what could cause your problem.


----------



## dizzy4

Quote:


> Originally Posted by *N0BOX*
> 
> Theoretically, Linux should be able to handle any number of different filesystem types mounted, regardless of whether they are a mix of ext3/4 and compressed container loop mounts... There are only two possible problems I can imagine having: doing a copy operation that preserves some kind of attribute that doesn't exist in the filesystem you are copying files to (say like trying to copy a file from an ext4 partition to an exfat partition while preserving file ownership and permissions?) and trying to copy files into the squashfs filesystem... I believe you have to extract the whole filesystem, chroot it, modify it, and then recompress it back to squashfs.
> Other than that, I can't make a guess as to what could cause your problem.


yeah that's what I thought. It looks like I will just have to create a custom livecd with a full kickstart or just manually do it using yum and chroot into the right path. Hopefully I will resolve this soon.


----------



## powerhouse

Quote:


> Originally Posted by *alcal*
> 
> What kind of a performance hit do you take when gaming on a VM over gaming on a regular windows install? Is there any way for "MOAR CORES" to improve gaming performance past what you could get with your ordinary bottleneck-free k-sku processor?
> Translation: Is there any good reason to do this if you aren't hellbent on using Linux? Is there an advantage for someone who is already happy with just plain windows?


If you're happy with Windows, why would you want to change anything? Unless you dual-boot Windows and Linux.

But since you asked about advantages (if any), here is my take:

1. Porting your Windows VM to a new PC: Of course there are ways to clone native Windows drives, but the clones only work with the hardware Windows was installed on. If you run Windows in a VM (Xen domU), it is relatively easy to take the Windows VM to another PC. With a passed-through graphics adapter, you would just install the (old) adapter in your new PC and perhaps adopt USB host or SATA controller PCI IDs - a simple edit of the config file - and your Windows will run just fine.
2. Backup Windows: There are backup or cloning tools for Windows, but I cannot recall anything that comes close to the simplicity of a LVM backup. LVM is Logical Volume Management. Using the "snapshot" feature, you can backup a running Windows VM. LVM is a Linux feature.
3. Virtual drives: LVM (see above) virtualizes your drives. Windows has "dynamic volumes", their proprietary implementation of LVM. Let's make it short - LVM has several advantages such as compatibility, robustness, etc. Read http://en.wikipedia.org/wiki/Dynamic_disk to get some ideas of why not to use Windows dynamic volumes, and http://en.wikipedia.org/wiki/Logical_Volume_Manager_%28Linux%29 to see what you can do with LVM.
4. No more crappy hardware RAID: Although Windows 7 has dynamic volume which supports stripes (RAID0), mirror (RAID1) etc. - LVM does that too - many users still use hardware RAID, what I mean is the usually sloppy, proprietary BIOS RAID implementation found on many mother boards. Linux has a far superior software RAID option that's dominated (and still dominates) the storage array and server market.
5. Running 2 or more VMs on the same PC: You can run 2 or perhaps more gaming VMs on one PC. Great for LAN parties. Or as dizzy4 wrote, a gaming VM and a HTPC VM, or whatever you fancy.

Above are the immediate *advantages* I see (there are probably more, but I can't think about them now). As to performance, I don't see a difference between a native bare metal Windows installation and running Windows in a VM using VGA passthrough. I've done some practical performance tests such as running CPU and disk intensive tasks in Windows and Linux and I'm absolutely thrilled about how well CPU resources scale to different tasks. For example, I processed several hundred RAW photo files in Windows using Lightroom while at the same time I ripped a DVD under Linux using "handbrake". Monitoring both VMs you could watch the CPU resources being assigned to the processes that needed them, and released when a process was idle (for example during I/O). I have 10 of 12 CPU threads (6-core CPU) assigned to Windows, but Linux can grab them when Windows is idle. Vice versa, Windows can grab almost all CPU resources when Linux doesn't use them.

The only bottleneck I can see is with the applications that can't make full use of my hardware. As far as I can see, it's mainly the applications that are behind the times with regard to performance.

P.S.: There are PV drivers available for Windows, which I use. They make a huge difference for disk I/O performance.


----------



## Zombiechow

Has anybody attempted this with Fedora 17?


----------



## dizzy4

Quote:


> Originally Posted by *Zombiechow*
> 
> Has anybody attempted this with Fedora 17?


It works well. Not any different than fedora 16 other than a newer version of xen. I might actually be switching to F17 because centos is a pain to to make a live cd.


----------



## Zombiechow

Quote:


> Originally Posted by *dizzy4*
> 
> Quote:
> 
> 
> 
> Originally Posted by *Zombiechow*
> 
> Has anybody attempted this with Fedora 17?
> 
> 
> 
> It works well. Not any different than fedora 16 other than a newer version of xen. I might actually be switching to F17 because centos is a pain to to make a live cd.
Click to expand...

Thanks!

I'm in the LiveCD environment right now attempting to hash this out. Gonna put this 3970x to work







.


----------



## dizzy4

Quote:


> Originally Posted by *Zombiechow*
> 
> Thanks!
> I'm in the LiveCD environment right now attempting to hash this out. Gonna put this 3970x to work
> 
> 
> 
> 
> 
> 
> 
> .


You might need to use a live-usb to add the kernel and xen though. There is a livecd to usb windows tool from fedora.


----------



## Ex3c

Hello, i am interested in the power consumption of the graphics card, when used in the virtual machine.
For example does "Zero Core Power" work with an AMD GPU?
And what happens when the virtual machine is shut down: Is also the GPU shutdown, and consumes really zero power?


----------



## powerhouse

Just opened a thread with a list of VT-d compatible Intel desktop motherboards: http://www.overclock.net/t/1338063/vt-d-compatible-motherboards

You are all welcome to add further information on Intel and/or other motherboard manufacturers supporting VT-d.


----------



## powerhouse

Quote:


> Originally Posted by *Ex3c*
> 
> Hello, i am interested in the power consumption of the graphics card, when used in the virtual machine.
> For example does "Zero Core Power" work with an AMD GPU?
> And what happens when the virtual machine is shut down: Is also the GPU shutdown, and consumes really zero power?


Interesting questions. I've never thought about it before, and I'm quite sure I cannot answer all the questions, but I'll give it shot.

1. When the VM runs, the graphics driver of the VM (for example the AMD graphics driver that runs on Windows) takes direct control of the graphics card. As far as I understand, power saving features are part of the graphics driver, at least it was like that with my old Nvidia card. So while the VM runs the graphics driver should take care of power management.

2. Does "Zero Core Power" work with an AMD GPU? No idea, I'm using a Nvidia adapter for the VM.

3. What happens when the virtual machine is shut down: Is also the GPU shutdown, and consumes really zero power? I could check the fan on my Nvidia GPU to see what happens when the VM is shut down, but I doubt the graphics adapter is shut down. I can connect my PC to a Watt meter (I have one) and check the power consumption, but this would only allow indirect conclusions. Still, if the power consumption goes down after shutting down the VM, it may indicate that some sort of power management kicks in.

It will take a while until I get to play with that, but my wild guess is that nothing will happen, or that the power consumption will increase (since there is no VM / driver throttling the GPU). Hope someone here has more experience / knowledge. I too would like to know what happens.


----------



## Zombiechow

This is proving to be a challenge. I'm having difficulty compiling a kernel module for the Xen kernel. So far it's been a no go. I have two GTX 680's.


----------



## dizzy4

Quote:


> Originally Posted by *Zombiechow*
> 
> This is proving to be a challenge. I'm having difficulty compiling a kernel module for the Xen kernel. So far it's been a no go. I have two GTX 680's.


The upstream kernel should have continuing support for Xen. The current Xen package for distribution with yum is 4.1.3 and should include a kernel that is compatible. Another option is using a custom repository.

Code:



Code:


# yum install http://au1.mirror.crc.id.au/repo/el6/x86_64/kernel-xen-release-6-4.noarch.rpm
# yum install xen kernel-xen

It might give you issues since there is already a xen package listed. You will have to chose the right one.

Credit to Stephen Haigh for his repository and work.


----------



## Ex3c

Quote:


> Originally Posted by *powerhouse*
> 
> 3. What happens when the virtual machine is shut down: Is also the GPU shutdown, and consumes really zero power? I could check the fan on my Nvidia GPU to see what happens when the VM is shut down, but I doubt the graphics adapter is shut down. I can connect my PC to a Watt meter (I have one) and check the power consumption, but this would only allow indirect conclusions. Still, if the power consumption goes down after shutting down the VM, it may indicate that some sort of power management kicks in.
> It will take a while until I get to play with that, but my wild guess is that nothing will happen, or that the power consumption will increase (since there is no VM / driver throttling the GPU). Hope someone here has more experience / knowledge. I too would like to know what happens.


That would be great.
Another sign for wether the GPU is shut down or not may be if the fan of the gpu is still running, when the virtual machine is shut down.


----------



## dizzy4

Quote:


> Originally Posted by *Ex3c*
> 
> That would be great.
> Another sign for wether the GPU is shut down or not may be if the fan of the gpu is still running, when the virtual machine is shut down.


I don't think it can be 100% off if it is connected to PCI power. This would be like booting the computer with a different card. The other(s) are still turned on during boot, but in an idle mode. The difference is in initialization. Another reason I believe it stays on is because dom0 can always see the card and the PCI bus whether connected to a VM or not.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> I don't think it can be 100% off if it is connected to PCI power. This would be like booting the computer with a different card. The other(s) are still turned on during boot, but in an idle mode. The difference is in initialization. Another reason I believe it stays on is because dom0 can always see the card and the PCI bus whether connected to a VM or not.


I agree with you. Which is why I'm curious to see in which state the graphics card runs when not in use.

My old PC has a passively cooled Nvidia card (can't remember the model) and I always monitored its temperature with lmsensor. When I turned on the PC and left it at the login prompt for some time, the graphics card would get really hot (as seen right after logon). Only after the logon, and after loading the Nvidia driver would the temperature go down. The Nvidia driver has a power throttling feature which - in this case - only works while the driver is loaded.

So, what I hope for is that the graphics adapter is in a low power mode at default, and only activated to full power when using the driver. Or whatever power saving mechanism there is. Just the opposite of what the old graphics module and driver did.

On a side note: I've been doing some reading on the Xen Wiki site and found this interesting script: http://wiki.xen.org/wiki/Bind_lib.bash_:_A_script_that_makes_pciback_with_modules_easier_to_use. It's only relevant to xend / xm toolstack users (not the newest 4.2 hypervisor with xl toolstack). It makes me think about grabbing my Nvidia card (the one I use for the Windows domU) while not running Windows and to use it for Linux. The key sentence on that page is: "This script is tested with a Radeon 6770 card, using pciback, with "pci_passthru=0" and "stdvga=1". Using this method, you don't need to blacklist radeon. " Will have to try it.


----------



## caffeinescandal

Hey guys. Really interested in doing this to my sig rig. Had a couple questions though.

1. If I understand correctly, it's like running virtualbox over linux. Only difference is there is no emulation going on, so it'll be like running the OS natively.
2. Theoretically, If I have two graphics cards and enough hard drive space, CPU power and RAM, I can run two OS's (e.g. 2 instances of Windows 7) and run TF2 on one OS and watch a blu-ray on the other with little lag? Also I get to individually assign PCI devices like a RAID card to their own OS right?

Downtime isn't really an issue, I plan to save images of my current OS install and give this a shot. I plan to spend all day getting this to work.


----------



## dizzy4

Quote:


> Originally Posted by *caffeinescandal*
> 
> Hey guys. Really interested in doing this to my sig rig. Had a couple questions though.
> 1. If I understand correctly, it's like running virtualbox over linux. Only difference is there is no emulation going on, so it'll be like running the OS natively.
> 2. Theoretically, If I have two graphics cards and enough hard drive space, CPU power and RAM, I can run two OS's (e.g. 2 instances of Windows 7) and run TF2 on one OS and watch a blu-ray on the other with little lag? Also I get to individually assign PCI devices like a RAID card to their own OS right?
> Downtime isn't really an issue, I plan to save images of my current OS install and give this a shot. I plan to spend all day getting this to work.


There is still some emulation done. QEMU is the base device model, but the video and cpu hardware is not emulated. Most pci devices will work well, but some video cards (some nvidia) require some patches. I think the raid card should be good for a NAS and HTPC virtual machine. If you have enough resources the lag shouldn't be an issue.


----------



## powerhouse

Quote:


> Originally Posted by *caffeinescandal*
> 
> Hey guys. Really interested in doing this to my sig rig. Had a couple questions though.
> 1. If I understand correctly, it's like running virtualbox over linux. Only difference is there is no emulation going on, so it'll be like running the OS natively.
> 2. Theoretically, If I have two graphics cards and enough hard drive space, CPU power and RAM, I can run two OS's (e.g. 2 instances of Windows 7) and run TF2 on one OS and watch a blu-ray on the other with little lag? Also I get to individually assign PCI devices like a RAID card to their own OS right?
> Downtime isn't really an issue, I plan to save images of my current OS install and give this a shot. I plan to spend all day getting this to work.


With regard to hardware, the FX990 chipset supports IOMMU (the AMD equivalent of VT-d) - see here: http://wiki.xen.org/wiki/VTd_HowTo. BUT, unfortunately Asus does not support Linux, so you need to be very cautious with BIOS updates - one BIOS might work, the other not, and if it doesn't, you'll be on your own. Also note these two threads about your board:
http://forums.bit-tech.net/showthread.php?p=2965518 and http://vip.asus.com/forum/view.aspx?board_id=1&model=SABERTOOTH+990FX&id=20110904052005048&page=1&SLanguage=en-us.
You will need to enable SVM (AMD-V) and perhaps also IOMMU in your 990FX motherboard BIOS.

Your CPU should work, looking at the specs.

Graphics card: Your HD 6850 should be good.

As to performance, there are worlds between say Virtualbox and Xen with VGA passthrough. I think it's worth to install the GPLPV drivers in Windows once you successfully setup your Windows domU (VM).

Time: It took me a lot more than a day to get my machine working, but then I had tried with incompatible hardware and a different distro (Linux Mint). It was a good learning experience, though. If things don't work at first trial, DON'T GIVE UP. Xen is real cool and dizzy4 gives us the easiest how-to to get it working.

Good luck!


----------



## Pigpoo

Perfect! Just what I needed, thank you very much!


----------



## nyxcharon

So, I cheated a bit, but I got this working on Linux Mint as well. It should be said, I just copied my config file over from fedora and used the pre made logical volume I had with windows already on it.








Also, I didn't have to bridge my network again, just like in fedora which I thought was strange.

Question for those of you who are experienced with LVM: What happens if I format off fedora(which is where I made all my volumes)?
I'd much rather use Mint if possible. I'm going to make a new windows VM now to make sure everything is working, but If i can just format off fedora and use mint i'd be pretty happy.









EDIT: So, I made a new VM in Mint with windows and had no issues, and that naggy bluescreen from the last time I tried this is no where to be found. Guess I'm backing up some data and formatting off fedora.









Relevant LVM info;


Spoiler: Warning: Spoiler!



[email protected] ~ $ sudo pvdisplay
[sudo] password for nyxcharon:
--- Physical volume ---
PV Name /dev/sdb3
VG Name vg_dom0
PV Size 231.05 GiB / not usable 24.00 MiB
Allocatable yes (but full)
PE Size 32.00 MiB
Total PE 7393
Free PE 0
Allocated PE 7393
PV UUID WC1cKc-n9t9-7MqG-ocNb-oaC4-B2Y2-bVeZKK

--- Physical volume ---
PV Name /dev/sda2
VG Name vg_dom0
PV Size 146.61 GiB / not usable 20.00 MiB
Allocatable yes
PE Size 32.00 MiB
Total PE 4691
Free PE 211
Allocated PE 4480
PV UUID 9irRft-dSum-c860-D0Kt-NCEU-tFEF-DNH9CB

[email protected] ~ $ sudo lvdisplay
--- Logical volume ---
LV Path /dev/vg_dom0/domUa
LV Name domUa
VG Name vg_dom0
LV UUID vg3np4-561j-QSOG-Rcmc-bneg-j3tZ-YvUhcz
LV Write Access read/write
LV Creation host, time dom0.xenhost, 2012-11-12 13:28:31 -0500
LV Status available
# open 0
LV Size 182.19 GiB
Current LE 5830
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:0

--- Logical volume ---
LV Path /dev/vg_dom0/dom0
LV Name dom0
VG Name vg_dom0
LV UUID tyjtXB-gy8f-YVom-WRYN-0RoJ-zKRY-OcM2TO
LV Write Access read/write
LV Creation host, time dom0.xenhost, 2012-11-12 13:28:34 -0500
LV Status available
# open 0
LV Size 48.84 GiB
Current LE 1563
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:1

--- Logical volume ---
LV Path /dev/vg_dom0/lvol0
LV Name lvol0
VG Name vg_dom0
LV UUID L61w0u-znMD-ZxnX-PrZX-eg3e-faFP-Iv2Yj5
LV Write Access read/write
LV Creation host, time dom0.xenhost, 2012-12-09 11:52:27 -0500
LV Status available
# open 0
LV Size 140.00 GiB
Current LE 4480
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 252:2


----------



## jrl1357

I don't think it's worth it with F18 around the corner, but F16 is about 2 months away from EOL so the guide could be updated use Fedora 18 when it hits stable in a couple of weeks. Just a thought. You could even use the beta now so it would almost certainly work with stable (they are mainly just bug hunting, mostly on fedup and anaconda)


----------



## dizzy4

Quote:


> Originally Posted by *jrl1357*
> 
> I don't think it's worth it with F18 around the corner, but F16 is about 2 months away from EOL so the guide could be updated use Fedora 18 when it hits stable in a couple of weeks. Just a thought. You could even use the beta now so it would almost certainly work with stable (they are mainly just bug hunting, mostly on fedup and anaconda)


That is a good idea, but I am currently trying to migrate to centos 6.3. The problem I am having is getting a livecd created with all the proper packages and scripts. I am using livecd-tools, but kickstarts for centos 6.3 are almost impossible to find and the livecd-creator --based-on flag does not work for me. If you have any information on how this works, it would be great. If not, I may have to compile xen 4.2 from source in a f18 kickstart.

With the holidays and work I am pretty busy, but hopefully I will have something soon


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> So, I cheated a bit, but I got this working on Linux Mint as well. It should be said, I just copied my config file over from fedora and used the pre made logical volume I had with windows already on it.
> 
> 
> 
> 
> 
> 
> 
> 
> Also, I didn't have to bridge my network again, just like in fedora which I thought was strange.
> Question for those of you who are experienced with LVM: What happens if I format off fedora(which is where I made all my volumes)?
> I'd much rather use Mint if possible. I'm going to make a new windows VM now to make sure everything is working, but If i can just format off fedora and use mint i'd be pretty happy.
> 
> 
> 
> 
> 
> 
> 
> 
> EDIT: So, I made a new VM in Mint with windows and had no issues, and that naggy bluescreen from the last time I tried this is no where to be found. Guess I'm backing up some data and formatting off fedora.
> 
> 
> 
> 
> 
> 
> 
> 
> Relevant LVM info;
> 
> 
> Spoiler: Warning: Spoiler!
> 
> 
> 
> [email protected] ~ $ sudo pvdisplay
> [sudo] password for nyxcharon:
> --- Physical volume ---
> PV Name /dev/sdb3
> VG Name vg_dom0
> PV Size 231.05 GiB / not usable 24.00 MiB
> Allocatable yes (but full)
> PE Size 32.00 MiB
> Total PE 7393
> Free PE 0
> Allocated PE 7393
> PV UUID WC1cKc-n9t9-7MqG-ocNb-oaC4-B2Y2-bVeZKK
> 
> --- Physical volume ---
> PV Name /dev/sda2
> VG Name vg_dom0
> PV Size 146.61 GiB / not usable 20.00 MiB
> Allocatable yes
> PE Size 32.00 MiB
> Total PE 4691
> Free PE 211
> Allocated PE 4480
> PV UUID 9irRft-dSum-c860-D0Kt-NCEU-tFEF-DNH9CB
> 
> [email protected] ~ $ sudo lvdisplay
> --- Logical volume ---
> LV Path /dev/vg_dom0/domUa
> LV Name domUa
> VG Name vg_dom0
> LV UUID vg3np4-561j-QSOG-Rcmc-bneg-j3tZ-YvUhcz
> LV Write Access read/write
> LV Creation host, time dom0.xenhost, 2012-11-12 13:28:31 -0500
> LV Status available
> # open 0
> LV Size 182.19 GiB
> Current LE 5830
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 256
> Block device 252:0
> 
> --- Logical volume ---
> LV Path /dev/vg_dom0/dom0
> LV Name dom0
> VG Name vg_dom0
> LV UUID tyjtXB-gy8f-YVom-WRYN-0RoJ-zKRY-OcM2TO
> LV Write Access read/write
> LV Creation host, time dom0.xenhost, 2012-11-12 13:28:34 -0500
> LV Status available
> # open 0
> LV Size 48.84 GiB
> Current LE 1563
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 256
> Block device 252:1
> 
> --- Logical volume ---
> LV Path /dev/vg_dom0/lvol0
> LV Name lvol0
> VG Name vg_dom0
> LV UUID L61w0u-znMD-ZxnX-PrZX-eg3e-faFP-Iv2Yj5
> LV Write Access read/write
> LV Creation host, time dom0.xenhost, 2012-12-09 11:52:27 -0500
> LV Status available
> # open 0
> LV Size 140.00 GiB
> Current LE 4480
> Segments 1
> Allocation inherit
> Read ahead sectors auto
> - currently set to 256
> Block device 252:2


It doesn't matter if you create LVM volumes in Fedora, Ubuntu, Suse or whatever - they are all the same. As long as you make sure LVM support is available when booting. In Linux Mint 14 it should be available by default, in Linux Mint 13 it wasn't and required some workaround.

I suppose you had installed the required packages on Linux Mint? Which LM release are you running?

Also, did you need to create some links (ln -s ...)?

I like your approach of creating the Windows domU under Fedora and then just running it from Linux Mint (though I guess you already performed the necessary steps/package installations before running the Windows domU).


----------



## jrl1357

Quote:


> Originally Posted by *dizzy4*
> 
> Quote:
> 
> 
> 
> Originally Posted by *jrl1357*
> 
> I don't think it's worth it with F18 around the corner, but F16 is about 2 months away from EOL so the guide could be updated use Fedora 18 when it hits stable in a couple of weeks. Just a thought. You could even use the beta now so it would almost certainly work with stable (they are mainly just bug hunting, mostly on fedup and anaconda)
> 
> 
> 
> That is a good idea, but I am currently trying to migrate to centos 6.3. The problem I am having is getting a livecd created with all the proper packages and scripts. I am using livecd-tools, but kickstarts for centos 6.3 are almost impossible to find and the livecd-creator --based-on flag does not work for me. If you have any information on how this works, it would be great. If not, I may have to compile xen 4.2 from source in a f18 kickstart.
> 
> With the holidays and work I am pretty busy, but hopefully I will have something soon
Click to expand...

I've been avoiding working with kickstarts since I switched to fedora- I have absolutely no idea where to even start. I'm really counting on the new fedup thing, although I suppose I will have to buckle down and learn how to make a kickstart at some point.


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> It doesn't matter if you create LVM volumes in Fedora, Ubuntu, Suse or whatever - they are all the same. As long as you make sure LVM support is available when booting. In Linux Mint 14 it should be available by default, in Linux Mint 13 it wasn't and required some workaround.
> I suppose you had installed the required packages on Linux Mint? Which LM release are you running?
> Also, did you need to create some links (ln -s ...)?
> I like your approach of creating the Windows domU under Fedora and then just running it from Linux Mint (though I guess you already performed the necessary steps/package installations before running the Windows domU).


Yeah, I didn't really understand LVM, but after reading up a bit it all makes sense now.
As far as mint, I used Linux Mint 14 64bit Cinnamon. I followed your guide to get Xen up and running, minus the networking, since I already had Internet in the VM. I made another Windows VM for testing, and it worked fine, no bluescreens like last time.

Ended up formatting off fedora and then expanding my LVM windows volume. Pretty happy to be back in something in the Debian ecosystem.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> Yeah, I didn't really understand LVM, but after reading up a bit it all makes sense now.
> As far as mint, I used Linux Mint 14 64bit Cinnamon. I followed your guide to get Xen up and running, minus the networking, since I already had Internet in the VM. I made another Windows VM for testing, and it worked fine, no bluescreens like last time.
> Ended up formatting off fedora and then expanding my LVM windows volume. Pretty happy to be back in something in the Debian ecosystem.


I'm glad to hear that you made it work. Linux Mint 14 makes things a little easier, compared to LM13. I'm in the same boat as you are, and feel a lot more comfortable in the Debian/Ubuntu/LM environment.

I've been playing a little with this http://community.linuxmint.com/tutorial/view/918 today in the hope to produce a Linux Mint USB stick that boots right into Xen, sort of as a first step to make this whole thing a little easier. I know dizzy4 is working on something probably more advanced, but I still would like to give it a try with Linux Mint.

The first ISO/USB stick I produced boots fine, just not into Xen







. Now I'm on my second trial.

P.S.: If you or anyone else needs some backup scripts, or a domU start script that checks if the domU is already running or not (instead of just quitting when it does run), let me know and I post some links to them.


----------



## caffeinescandal

Whoever makes the live cd, i'll be the first to try it.









I tried my hand at getting this to run, my linux level isnt much as I mostly messed around the dd-wrt routers. I got it to install fine, got xen running (I think), restarted and chose xen. Created my vm and the only thing I couldnt get to work was PCI passthrough. I only have one graphics card atm, but I was planning on getting a second card so I can run two VM's with passthrough. Thats actually the main reason I tried this. lol

Anyway I couldnt get it to work after a day of tinkering, so for now I will be going back to windows.


----------



## dalastbmills

So if my motherboard doesn't support VT-d I will not be able to successfully attempt this?

(Great guide, by the way. Can't believe it's been floating around for a year and I am just seeing it for the first time







)

I'm definitely interested in getting a linux VM up and running. What are my options?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> I'm glad to hear that you made it work. Linux Mint 14 makes things a little easier, compared to LM13. I'm in the same boat as you are, and feel a lot more comfortable in the Debian/Ubuntu/LM environment.
> I've been playing a little with this http://community.linuxmint.com/tutorial/view/918 today in the hope to produce a Linux Mint USB stick that boots right into Xen, sort of as a first step to make this whole thing a little easier. I know dizzy4 is working on something probably more advanced, but I still would like to give it a try with Linux Mint.
> The first ISO/USB stick I produced boots fine, just not into Xen
> 
> 
> 
> 
> 
> 
> 
> . Now I'm on my second trial.
> P.S.: If you or anyone else needs some backup scripts, or a domU start script that checks if the domU is already running or not (instead of just quitting when it does run), let me know and I post some links to them.


I think i might play with getting this to work on Debian now. If I can, I might have to tinker with making a Live Cd of it after I finish working on my new Adonis Linux Beta. Sounds like It could be a fun project.
Quote:


> Originally Posted by *dalastbmills*
> 
> So if my motherboard doesn't support VT-d I will not be able to successfully attempt this?
> (Great guide, by the way. Can't believe it's been floating around for a year and I am just seeing it for the first time
> 
> 
> 
> 
> 
> 
> 
> )
> I'm definitely interested in getting a linux VM up and running. What are my options?


That is correct. Said to say, unless you want to tinker around with getting a modified bios for your mobo not much else you can do but get a new one. I'm using an Asrock Z77 Pro3 and I love it.
Quote:


> Originally Posted by *caffeinescandal*
> 
> Whoever makes the live cd, i'll be the first to try it.
> 
> 
> 
> 
> 
> 
> 
> 
> I tried my hand at getting this to run, my linux level isnt much as I mostly messed around the dd-wrt routers. I got it to install fine, got xen running (I think), restarted and chose xen. Created my vm and the only thing I couldnt get to work was PCI passthrough. I only have one graphics card atm, but I was planning on getting a second card so I can run two VM's with passthrough. Thats actually the main reason I tried this. lol
> Anyway I couldnt get it to work after a day of tinkering, so for now I will be going back to windows.


What graphics card where you using, and where you doing it with fedora, like in the guide?
And like I said above, might play with making a Live Cd myself.


----------



## caffeinescandal

Quote:


> Originally Posted by *nyxcharon*
> 
> I think i might play with getting this to work on Debian now. If I can, I might have to tinker with making a Live Cd of it after I finish working on my new Adonis Linux Beta. Sounds like It could be a fun project.
> That is correct. Said to say, unless you want to tinker around with getting a modified bios for your mobo not much else you can do but get a new one. I'm using an Asrock Z77 Pro3 and I love it.
> What graphics card where you using, and where you doing it with fedora, like in the guide?
> And like I said above, might play with making a Live Cd myself.


I was using fedora 16 at first and then i tried 17 as well. Pretty much the same. The problem I've been getting with passthrough was something along the lines of "pci backend and pci-stub dont own the device" or something along those lines.


----------



## nyxcharon

Quote:


> Originally Posted by *caffeinescandal*
> 
> I was using fedora 16 at first and then i tried 17 as well. Pretty much the same. The problem I've been getting with passthrough was something along the lines of "pci backend and pci-stub dont own the device" or something along those lines.


Was that when you where trying to attach them/start the vm?
I got that message as well for my motherboard audio and I just had to run the following command and it worked:

Code:



Code:


[[email protected] ~]$ sudo virsh nodedev-dettach pci_0000_01_00_0
Device pci_0000_01_00_0 detached


----------



## caffeinescandal

Quote:


> Originally Posted by *nyxcharon*
> 
> Was that when you where trying to attach them/start the vm?
> I got that message as well for my motherboard audio and I just had to run the following command and it worked:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> [[email protected] ~]$ sudo virsh nodedev-dettach pci_0000_01_00_0
> Device pci_0000_01_00_0 detached


Yeah it was when I was gonna start a VM. I did the detach command too for the gpu, it worked since my screen went black but then I couldn't see what I was doing.


----------



## dizzy4

Quote:


> Originally Posted by *caffeinescandal*
> 
> Yeah it was when I was gonna start a VM. I did the detach command too for the gpu, it worked since my screen went black but then I couldn't see what I was doing.


haha yeah, that's the thing. A script could handle this well I am sure. You can try:

Code:



Code:


sudo virsh nodedev-dettach pci_xxxx_xx_xx_x && sudo xm create win7a.sxp

the two ampersands separate the commands and will execute them both.


----------



## powerhouse

Quote:


> Originally Posted by *dalastbmills*
> 
> So if my motherboard doesn't support VT-d I will not be able to successfully attempt this?
> (Great guide, by the way. Can't believe it's been floating around for a year and I am just seeing it for the first time
> 
> 
> 
> 
> 
> 
> 
> )
> I'm definitely interested in getting a linux VM up and running. What are my options?


If you run Windows and want to have a Linux VM, you can always use VirtualBox (https://www.virtualbox.org/). That's probably the easiest way.

The guide presented by dizzy4 talks about installing Windows as a VM on a Linux/Xen machine. In the example given you will be running Fedora 16 as your main OS (administrative domain or dom0 in Xen talk), and Windows as a virtual machine or domU. The special thing about dizzy4's guide is that you will be able to run Windows with almost no performance penalty compared to a regular install. The trickiest part is getting Windows to directly communicate with the graphics card, a technology called VGA passthrough.

There are much simpler ways to test or get a better feel of a particular Linux distribution, just download a live CD/DVD ISO image and put it on a USB stick using unetbootin (also available for Windows). Then you can boot the USB stick and play with the distro - no need to install until you are settled with your choice.

I may have totally misunderstood you and I don't mean to offend you or anyone else here by explaining some basics, but since you asked about running a Linux VM I thought this could help.

Edit: Another way would be to use Microsoft Hyper-V and install Linux on that. Microsoft even published PV drivers for Linux - well, they sort of had to do it, since Microsoft (perhaps illegally) used open source GPL-ed code without adhering to the license agreement. Search for it - it's worth reading about.


----------



## powerhouse

So, I gave the bootable Xen USB stick another try. It even started booting into Xen (as I could see from the console lines), but eventually rebooted somewhere in the middle. Oh well, I knew I had to edit the initrd but was lazy, after it threw me some errors. Lesson learned - don't try to edit the initrd with a different Linux release (I'm running LM13 but was preparing a LM14 ISO).

One thing I discovered by "accident" was that my passed-through USB host/ports aren't active when the Windows domU isn't running. I tried to charge my iPhone on a passed-through USB port - while Windows was shut off - and there was nothing. Does that mean there is no voltage output on that USB port (i.e. it's completely shut down), or does the iPhone expect to interact with the OS? I have no idea.


----------



## dizzy4

I did get a live-usb to work. I copied the centos 6.3 livecd to a usb stick using liveusb-creator and created an overlay. Then I just installed the proper packages for xen (including xen). The trick to getting it working was really just dumb luck. I had created the liveusb on a drive that contained a few files (about 150mb) and found that the files were still there after the distro was loaded. This was perfect, because I could delete those files and fill the free space with the kernel, xen.gz and initramfs -- this portion is not part of the persistent overlay. I had to add mboot.c32 and edit my syslinux.cfg. You use mboot.c32 as the kernel and append xen.gz, pvops kernel and initramfs.

I found documentation somewhere, but my tablet died and I can't find it now. I hope this points you in the right direction.


----------



## caffeinescandal

Quote:


> Originally Posted by *dizzy4*
> 
> haha yeah, that's the thing. A script could handle this well I am sure. You can try:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> sudo virsh nodedev-dettach pci_xxxx_xx_xx_x && sudo xm create win7a.sxp
> 
> the two ampersands separate the commands and will execute them both.


Thanks. I'll give this a shot.

edit: before I do, I assume I have to detach both the gpu and audio part of my video card? so the script will go something like:

Code:



Code:


sudo virsh nodedev-dettach pci_xxxx_xx_xx_1 && sudo virsh nodedev-dettach pci_xxxx_xx_xx_2 && sudo xm create win7a.sxp


----------



## stolid

Has anyone tried this with Debian? I run Wheezy, and I'm considering trying this with a 5770 soon.


----------



## dizzy4

Quote:


> Originally Posted by *stolid*
> 
> Has anyone tried this with Debian? I run Wheezy, and I'm considering trying this with a 5770 soon.


Many people have gotten it working with Linux Mint (Debian/Ubuntu base). You should be able to get this working pretty easily -- you might want to look into development versions since Debian is pretty far behind on Xen releases.


----------



## Pip Boy

II installed xen, ubuntu base, there is now a bootloader option on grub should I need it?
also what viewing / management tool should I use?


----------



## nyxcharon

Quote:


> Originally Posted by *stolid*
> 
> Has anyone tried this with Debian? I run Wheezy, and I'm considering trying this with a 5770 soon.


I tried this on stable and testing and it failed horribly.
Linux Mint 14 was a breeze, as was Fedora 17.
Quote:


> Originally Posted by *phill1978*
> 
> II installed xen, ubuntu base, there is now a bootloader option on grub should I need it?
> also what viewing / management tool should I use?


You should be picking that xen grub entry on startup. I use virt-manager for management, as does the guide dizzy posted.


----------



## jrl1357

Quote:


> Originally Posted by *dizzy4*
> 
> Quote:
> 
> 
> 
> Originally Posted by *stolid*
> 
> Has anyone tried this with Debian? I run Wheezy, and I'm considering trying this with a 5770 soon.
> 
> 
> 
> Many people have gotten it working with Linux Mint (Debian/Ubuntu base). You should be able to get this working pretty easily -- you might want to look into development versions since Debian is pretty far behind on Xen releases.
Click to expand...

Quote:


> Originally Posted by *nyxcharon*
> 
> Quote:
> 
> 
> 
> Originally Posted by *stolid*
> 
> Has anyone tried this with Debian? I run Wheezy, and I'm considering trying this with a 5770 soon.
> 
> 
> 
> I tried this on stable and testing and it failed horribly.
> Linux Mint 14 was a breeze, as was Fedora 17.
> Quote:
> 
> 
> 
> Originally Posted by *phill1978*
> 
> II installed xen, ubuntu base, there is now a bootloader option on grub should I need it?
> also what viewing / management tool should I use?
> 
> Click to expand...
> 
> You should be picking that xen grub entry on startup. I use virt-manager for management, as does the guide dizzy posted.
Click to expand...

you might have to grab xen from sid.


----------



## nyxcharon

Guess I'll give that a go when I get the chance then. The closer I am to pure debian the happier I am









http://packages.debian.org/search?keywords=xen&searchon=names&suite=all&section=all
Stable is 4.0
Testing is 4.1
Experimental is 4.2


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> I did get a live-usb to work. I copied the centos 6.3 livecd to a usb stick using liveusb-creator and created an overlay. Then I just installed the proper packages for xen (including xen). The trick to getting it working was really just dumb luck. I had created the liveusb on a drive that contained a few files (about 150mb) and found that the files were still there after the distro was loaded. This was perfect, because I could delete those files and fill the free space with the kernel, xen.gz and initramfs -- this portion is not part of the persistent overlay. I had to add mboot.c32 and edit my syslinux.cfg. You use mboot.c32 as the kernel and append xen.gz, pvops kernel and initramfs.
> I found documentation somewhere, but my tablet died and I can't find it now. I hope this points you in the right direction.


Thanks dizzy4. I got as far as adding mboot.c32 and edited the syslinux.cfg. Just for reference, could you share your syslinux.cfg?

My issue was that I wasn't able to run update-grub successfully after installing the Xen hypervisor, so grub as well as initramfs wasn't updated. I also didn't manage to edit the initramfs manually.

I found some comments on the Linux Mint forum that I need to try.

What do you mean by "created an overlay"?

I understand from you that dom0 runs a pvops kernel, not the standard kernel?

xen.gz is your hypervisor?


----------



## dizzy4

Code:



Code:


label XEN
        kernel mboot.c32
        append xen-4.2.0.gz iommu=1 --- vmlinuz-3.6.2-1.el6xen.x86_64 root=live:UUID=7402-1F02 rootfstype=vfat rw liveimg overlay=UUID=7402-1F02 3 quiet nodiskmount novolumemount rhgb vga=791 rd.luks=0 rd.md=0 rd.dm=0 --- initramfs-3.6.2-1.el6xen.x86_64.img

there is the relevant .cfg entry. For root I copied the default from the other entry.

Since this is a live-usb a persistent storage chunk is created in the space not needed. This allows packages to be added etc.

I am using xen-4.2.0.gz to be exact, but any version should work.

I am not editing my initramfs file. I just included the one I use as well as my kernel into the syslinux folder.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> label XEN
> kernel mboot.c32
> append xen-4.2.0.gz iommu=1 --- vmlinuz-3.6.2-1.el6xen.x86_64 root=live:UUID=7402-1F02 rootfstype=vfat rw liveimg overlay=UUID=7402-1F02 3 quiet nodiskmount novolumemount rhgb vga=791 rd.luks=0 rd.md=0 rd.dm=0 --- initramfs-3.6.2-1.el6xen.x86_64.img
> 
> there is the relevant .cfg entry. For root I copied the default from the other entry.
> Since this is a live-usb a persistent storage chunk is created in the space not needed. This allows packages to be added etc.
> I am using xen-4.2.0.gz to be exact, but any version should work.
> I am not editing my initramfs file. I just included the one I use as well as my kernel into the syslinux folder.


Thanks! That's very helpful and I will try that. I still need to fix the update-grub issue but as I said the Linux Mint forum has some pointers.

Is there any difference between xl and xm (xend)? Ease of use, performance, compatibility? Just a question if you noticed something.


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> Thanks! That's very helpful and I will try that. I still need to fix the update-grub issue but as I said the Linux Mint forum has some pointers.
> Is there any difference between xl and xm (xend)? Ease of use, performance, compatibility? Just a question if you noticed something.


xend is being phased out and xl is now the default as of 4.2.0. The main reason I like xl is because you no longer need to use virsh (libvirt) to hot swap pci devices. Xend is not even needed anymore, but is still enabled by default so xm can still be used.

Sorry I hate GRUB 2... I avoid it if I can


----------



## jrl1357

Quote:


> Originally Posted by *dizzy4*
> 
> Quote:
> 
> 
> 
> Originally Posted by *powerhouse*
> 
> Thanks! That's very helpful and I will try that. I still need to fix the update-grub issue but as I said the Linux Mint forum has some pointers.
> Is there any difference between xl and xm (xend)? Ease of use, performance, compatibility? Just a question if you noticed something.
> 
> 
> 
> xend is being phased out and xl is now the default as of 4.2.0. The main reason I like xl is because you no longer need to use virsh (libvirt) to hot swap pci devices. Xend is not even needed anymore, but is still enabled by default so xm can still be used.
> 
> Sorry I hate GRUB 2... I avoid it if I can
Click to expand...

I know the feeling- get it working and for god sakes never touch it


----------



## flash0151

Helly everyone,

I've got a Fedora 16 build working but as soon as I add any pci device and try to start the machine i get the "pci: PCI Backend and pci-stub don't own device xx:xx.x" error i've used the guide on the first post without a hitch but i cant pass anything. My mobo and cpu have vt-d enabled installed (I'm a ex-debian user) can someone point me in the right direction?


----------



## jrl1357

Quote:


> Originally Posted by *flash0151*
> 
> Helly everyone,
> 
> I've got a Fedora 16 build working but as soon as I add any pci device and try to start the machine i get the "pci: PCI Backend and pci-stub don't own device xx:xx.x" error i've used the guide on the first post without a hitch but i cant pass anything. My mobo and cpu have vt-d enabled installed (I'm a ex-debian user) can someone point me in the right direction?


it's possible it's an selinux issue. I would post how to disable it but a google search provides everything in the first couple hits. I would try that first


----------



## flash0151

Quote:


> it's possible it's an selinux issue. I would post how to disable it but a google search provides everything in the first couple hits. I would try that first


I had selinux set to "permissive" as per the guide I've also set it to "disabled" but no help. Everything works as advertised but i cannot add any pci divices because I get the "pciBack doen't own this device" error....I'm sure it has to be something simple but i've been reading other posts and everyong has something different I used this guide because i almost have identical hardware yet pci passthrough still not working


----------



## flash0151

There is another problem I encounter while following the guide again there is a lot of great information here but after reading 40+ pages of replies I did not find a solution so here it goes. While trying to start virt-manager remotely following the instructions of the OP i get the following error while using google i discovered that this is a very common error which leads me to the question how come no one here has address it yet?

has anyone run into this issue before?


----------



## Pigpoo

Flash0151, it appears you're trying to run virt-manager in PuTTY. PuTTY is text-based and cannot write objects on-screen which is why you're getting that error. You either need to use VNC/tunnel X to see a GUI or use xm/xl.


----------



## powerhouse

Quote:


> Originally Posted by *flash0151*
> 
> There is another problem I encounter while following the guide again there is a lot of great information here but after reading 40+ pages of replies I did not find a solution so here it goes. While trying to start virt-manager remotely following the instructions of the OP i get the following error while using google i discovered that this is a very common error which leads me to the question how come no one here has address it yet?
> 
> has anyone run into this issue before?


Edit: I agree with Pigpoo and deleted my original reply.


----------



## flash0151

Quote:


> Originally Posted by *powerhouse*
> 
> Edit: I agree with Pigpoo and deleted my original reply.


Thanks for answering guys but i"m using putty with xming as suggested by the OP on page 1 of this forum section 6a which leads to believe it has been tested and therefore supported. What else should i try? This has been a 4weeks deal trying to set this machine up to work with my security camara pci card which doesnt support FLR hence the "PCI Backend and pci-stub don't own this device" error and now this. The steps are very clear yet the results have not been.

Thanks again guys
semper fi


----------



## Pigpoo

Quote:


> Originally Posted by *flash0151*
> 
> Thanks for answering guys but i"m using putty with xming as suggested by the OP on page 1 of this forum section 6a which leads to believe it has been tested and therefore supported.


What Distro are you using? Do you have a GUI installed? If you don't, install the Gnome meta-package or similar. Otherwise, it may be that you don't have the correct dependencies installed (seems like a GTK+ import error).


----------



## flash0151

Quote:


> Originally Posted by *Pigpoo*
> 
> What Distro are you using? Do you have a GUI installed? If you don't, install the Gnome meta-package or similar. Otherwise, it may be that you don't have the correct dependencies installed (seems like a GTK+ import error).


I have F16 just like the OP suggested, sorry to sound like a recording but i made sure I had the identical (close) hardware and followed the OP's instructions to the T....anywho i'll install the gnome metadata package as you suggested and see what happens...also any idean why i can not passthrough any of my pci devices? its seems like none of them is owned by pciback or pci-stub weird? isnt' that supposed to happen out of the box with F16?


----------



## Chetyre

Hello,

I've been trying to get vga passtrough working for a while now and I found this excelent topic, so I was wondering if anyone could help me.

First, my hardware is the following:

- Asrock Z77 Extreme4
- Intel I7 3770
- 2gb ram (temporary, more is on the way)
- Nvidia GTS250 (which is a rebranded 9800, which I know some people managed to get working
- Corsair force GT 90gb
- two other conventional HDs that I will also try to passthrough (can I even passthrough HDs? I could never find an answer to this)

I am running CentOS 6.3 without a graphical environment.

My problem is: whenever I try to activate gfx passthru, I just get no signal whatsoever on the monitor. Not even from a bios, and the windows VM doesn't seem to start either as I'm unable to ping it or anything. If I disable the option I'm able to work with it just fine.

I don't have my config with me right now, but I basically copied it from the config on the first post.

I'd really appreciate if any of you could help me with this.


----------



## dizzy4

Quote:


> Originally Posted by *Chetyre*
> 
> Hello,
> I've been trying to get vga passtrough working for a while now and I found this excelent topic, so I was wondering if anyone could help me.
> First, my hardware is the following:
> - Asrock Z77 Extreme4
> - Intel I7 3770
> - 2gb ram (temporary, more is on the way)
> - Nvidia GTS250 (which is a rebranded 9800, which I know some people managed to get working
> - Corsair force GT 90gb
> - two other conventional HDs that I will also try to passthrough (can I even passthrough HDs? I could never find an answer to this)
> I am running CentOS 6.3 without a graphical environment.
> My problem is: whenever I try to activate gfx passthru, I just get no signal whatsoever on the monitor. Not even from a bios, and the windows VM doesn't seem to start either as I'm unable to ping it or anything. If I disable the option I'm able to work with it just fine.
> I don't have my config with me right now, but I basically copied it from the config on the first post.
> I'd really appreciate if any of you could help me with this.


That is very similar hardware to what I have and I also run headless centos 6.3 as a dom0. A little more info would be good. Are you running Xen 4.1.2 like the original post or have you managed to get 4.2.0 to work? If not, I suggest going here: https://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/

He also has a small repository for Xen and keeps it up to date.

Something else to try is xl list (or xm list if you are before 4.2.0) to see what state the machine is in. If it is in -b---- it means that it is stuck in a boot state (usually storage or network issues). Can you post your VM configuration file?


----------



## fergalom

This may be a stupid question so apologies in advance.

I have similar hardware with F17 Dom0 and Xen 4.1.2 (all working ok)

Can I upgrade my F17 Dom0 to Xen 4.2 by following the instructions in the link you posted?

https://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/

Or can someone point me towards the correct kernel for F17?


----------



## Chetyre

Quote:


> Originally Posted by *dizzy4*
> 
> That is very similar hardware to what I have and I also run headless centos 6.3 as a dom0. A little more info would be good. Are you running Xen 4.1.2 like the original post or have you managed to get 4.2.0 to work? If not, I suggest going here: https://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/
> He also has a small repository for Xen and keeps it up to date.
> Something else to try is xl list (or xm list if you are before 4.2.0) to see what state the machine is in. If it is in -b---- it means that it is stuck in a boot state (usually storage or network issues). Can you post your VM configuration file?


I managed to get xen 4.2 to work by following this guide: http://www.howtoforge.com/virtualization-with-xen-on-centos-6.3-x86_64-paravirtualization-and-hardware-virtualization which does use the repo you mentioned.

I think I might have forgotten to disable selinux though. I can work on it today and I'll try to post my config file and see the state the machine is in.


----------



## powerhouse

Quote:


> Originally Posted by *flash0151*
> 
> I have F16 just like the OP suggested, sorry to sound like a recording but i made sure I had the identical (close) hardware and followed the OP's instructions to the T....anywho i'll install the gnome metadata package as you suggested and see what happens...also any idean why i can not passthrough any of my pci devices? its seems like none of them is owned by pciback or pci-stub weird? isnt' that supposed to happen out of the box with F16?


Not sure if it's the same on Fedora, but you could try to see what dmesg | grep pciback gives you. Here is my output as an example:

Code:



Code:


dmesg | grep pciback
[   21.647922] xen-pciback: backend is passthrough
[   28.053848] pciback 0000:02:00.0: seizing device
[   28.053884] pciback 0000:02:00.0: enabling device (0000 -> 0003)
[   28.053924] pciback 0000:02:00.0: PCI INT A -> GSI 40 (level, low) -> IRQ 40
[   28.053929] pciback 0000:02:00.0: PCI INT A disabled
[   28.267071] pciback 0000:02:00.1: seizing device
[   28.267150] pciback 0000:02:00.1: PCI INT B -> GSI 44 (level, low) -> IRQ 44
[   28.267159] pciback 0000:02:00.1: PCI INT B disabled
[   28.879674] pciback 0000:00:1a.0: seizing device
[   28.879753] pciback 0000:00:1a.0: PCI INT A -> GSI 23 (level, low) -> IRQ 23
[   28.879761] pciback 0000:00:1a.0: PCI INT A disabled
[   28.946573] pciback 0000:09:00.0: seizing device
[   28.946630] pciback 0000:09:00.0: PCI INT A -> GSI 17 (level, low) -> IRQ 17
[   28.946634] pciback 0000:09:00.0: PCI INT A disabled

There are several possible reasons why your pci devices aren't owned by pciback. I had problems with the nouveau driver (Nvidia) that was installed automatically for my secondary GPU (the one I wanted to pass through). I had to blacklist the nouveau driver.

I can't say much about the method dizzy4 described here in this post, as I use a Debian/Ubuntu based distro and a little different approach for the setup. However, here is a checklist you may want to go through:

1. Did you enable VT-d (IOMMU) in the BIOS? It's usually disabled by default. Don't confuse it with VT-x (often specified as Intel Virtualization or so in the BIOS), it's NOT the same - this latter option is usually enabled by default.
2. Check your grub file and make sure you ran grub2-mkconfig after editing it.
3. Instead of using virt-manager in step 7 of dizzy4's guide to detach the pci devices, you could download and try this script http://wiki.xen.org/wiki/Bind_lib.bash_:_A_script_that_makes_pciback_with_modules_easier_to_use.

See if any of this works. Good luck!


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> Hello,
> I've been trying to get vga passtrough working for a while now and I found this excelent topic, so I was wondering if anyone could help me.
> First, my hardware is the following:
> - Asrock Z77 Extreme4
> - Intel I7 3770
> - 2gb ram (temporary, more is on the way)
> - Nvidia GTS250 (which is a rebranded 9800, which I know some people managed to get working
> - Corsair force GT 90gb
> - two other conventional HDs that I will also try to passthrough (can I even passthrough HDs? I could never find an answer to this)
> I am running CentOS 6.3 without a graphical environment.
> My problem is: whenever I try to activate gfx passthru, I just get no signal whatsoever on the monitor. Not even from a bios, and the windows VM doesn't seem to start either as I'm unable to ping it or anything. If I disable the option I'm able to work with it just fine.
> I don't have my config with me right now, but I basically copied it from the config on the first post.
> I'd really appreciate if any of you could help me with this.


Looks like your VGA passthrough is kinda working. When you activate gfx passthrough the graphics card is literally disconnected from Linux/dom0 and attached to pciback, that means you should see nothing on that display. If you are passing a secondary GPU, your primary should still be available. Else you need to connect to your PC via ssh -X, VNC or similar from another PC.

Run xm create ... and afterwards virt-manager, click on your Windows VM and open the console. Since you have Windows installed, it might display some messages relating to your new graphics card, or just boot into Windows and ask for driver installation. Until you install the Windows driver for your graphics card and reboot, you most likely won't be able to see any output on the screen attached to that graphics card. virt-manager has a built-in VNC client that makes it easy to connect to the VM.

P.S.: Make sure your /etc/xen/xend-config.sxp file contains:

Code:



Code:


(xend-unix-server yes)

without #

Regarding HD passthrough, you cannot pass through individual drives, but the controller(s). Most motherboards have several SATA controllers; mine has 3: the Intel X79 SATA controller, an ASMedia SATA controller for two external eSATA ports, and a useless Marvell SATA controller that doesn't work under Xen. I added another ASMedia controller card to get more SATA ports for internal drives (currently 1 SSD, 5 HDD drives + 1 DVD R/W).

BUT, in my case there is a catch with the two ASMedia controllers: If I pass through one of them, I cannot access/use the other.

Actually, I don't see a reason to pass through a SATA controller, except perhaps for external eSATA media (e.g. removable backup disks). For good disk I/O performance, get the GPLPV drivers for Windows here: http://wiki.univention.de/index.php?title=Installing-signed-GPLPV-drivers. They work wonders, especially for disk drives! They also provide a 10Gb bridge to the dom0, so Samba file transfers between Windows and Linux (dom0) are running nicely and the limit seems to be the read write speed of the drives. I take it for granted that you use LVM drives?

Hope it helps.


----------



## Chetyre

Quote:


> Originally Posted by *powerhouse*
> 
> Looks like your VGA passthrough is kinda working. When you activate gfx passthrough the graphics card is literally disconnected from Linux/dom0 and attached to pciback, that means you should see nothing on that display. If you are passing a secondary GPU, your primary should still be available. Else you need to connect to your PC via ssh -X, VNC or similar from another PC.
> Run xm create ... and afterwards virt-manager, click on your Windows VM and open the console. Since you have Windows installed, it might display some messages relating to your new graphics card, or just boot into Windows and ask for driver installation. Until you install the Windows driver for your graphics card and reboot, you most likely won't be able to see any output on the screen attached to that graphics card. virt-manager has a built-in VNC client that makes it easy to connect to the VM.
> P.S.: Make sure your /etc/xen/xend-config.sxp file contains:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> (xend-unix-server yes)
> 
> without #
> Hope it helps.


I'm passing a secondary graphics card, the primary one is the integrated one in the processor, HD graphics 4000 IIRC. the one that has no signal is the monitor that should be connected to windows. BTW, if I VNC to the machine I just get a black screen that says "no serial" or something like that.

I'm using xl, not xm.


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> I'm passing a secondary graphics card, the primary one is the integrated one in the processor, HD graphics 4000 IIRC. the one that has no signal is the monitor that should be connected to windows. BTW, if I VNC to the machine I just get a black screen that says "no serial" or something like that.
> I'm using xl, not xm.


Is that what Windows throws at you when booting? Or some VNC/console message?

Can you post your domU config file (win7.cfg or whatever) found in /etc/xen/ ?


----------



## Chetyre

Quote:


> Originally Posted by *powerhouse*
> 
> Is that what Windows throws at you when booting? Or some VNC/console message?
> Can you post your domU config file (win7.cfg or whatever) found in /etc/xen/ ?


It's a VNC message.

I've disabled the gfx_passthru option and I could VNC into windows and see the nvidia device. Still no signal though. I will try to install the card's drivers and see what happens.

my config is as follows:

Code:



Code:


builder='hvm'
memory=1024
vcpus=8
pae=1
vif=['bridge=xenbr0,'mac=<random mac here>']
disk=['phy://dev/vg_xen/windows,hda,w']
boot='dc'
pci=['01:00.0']
vnc=1
vnclisten="0.0.0.0"
vncpasswd=""
usbdevice='tablet'
localtime=1


----------



## Stealth Pyros

This sort of setup intrigues me and I'd love to experiment with it in some sort of LAN center setup. It may be very expensive in the start-up phase (maybe not even that since you can buy the hardware in bulk) but I think long term it'll offer good cost reductions and improved efficiency. Maintenance would be easier in my eyes, and you wouldn't have to interrupt your customers since you can use VMotion/etc. tools to move VMs between "servers" on the fly. I also see it being a more secure setup since the customers wouldn't have access to all the beefy hardware.

If I want a LAN center with say 64 "terminals" that my customers will use, how powerful would those 64 terminals have to be? Assume that I'd have some monstrous hardware in the back with 256GB of RAM or even more, and enough HD7950s (or whatever GPU, say even 2 7950s per gaming terminal) to make anyone on OCN cry in joy, along with whatever processors would be appropriate (racks of IB-Es anyone?). I'm just being theoretical here for the sake of curiosity. I'd love to start up a business some day and it consists of a LAN center for PC gaming.

Would love to hear back from anyone that's experienced such a setup.


----------



## Chetyre

Quote:


> Originally Posted by *Stealth Pyros*
> 
> This sort of setup intrigues me and I'd love to experiment with it in some sort of LAN center setup. It may be very expensive in the start-up phase (maybe not even that since you can buy the hardware in bulk) but I think long term it'll offer good cost reductions and improved efficiency. Maintenance would be easier in my eyes, and you wouldn't have to interrupt your customers since you can use VMotion/etc. tools to move VMs between "servers" on the fly. I also see it being a more secure setup since the customers wouldn't have access to all the beefy hardware.
> 
> If I want a LAN center with say 64 "terminals" that my customers will use, how powerful would those 64 terminals have to be? Assume that I'd have some monstrous hardware in the back with 256GB of RAM or even more, and enough HD7950s (or whatever GPU, say even 2 7950s per gaming terminal) to make anyone on OCN cry in joy, along with whatever processors would be appropriate (racks of IB-Es anyone?). I'm just being theoretical here for the sake of curiosity. I'd love to start up a business some day and it consists of a LAN center for PC gaming.
> 
> Would love to hear back from anyone that's experienced such a setup.


To do what you want it would probably be more appropriate to research streaming solutions together with the virtualization one. just making VMs and assigning them video cards won't magically transfer into a lot of gaming speed over a network, even on a LAN. You probably need something like what OnLive does.

As for the terminals, any thin client should work just fine. They wouldn't need to be powerful at all, just enough to drive the monitor at the right resolution and to use the software that will be receiving the stream from the server, whatever it could be.


----------



## Stealth Pyros

Quote:


> Originally Posted by *Chetyre*
> 
> Quote:
> 
> 
> 
> Originally Posted by *Stealth Pyros*
> 
> This sort of setup intrigues me and I'd love to experiment with it in some sort of LAN center setup. It may be very expensive in the start-up phase (maybe not even that since you can buy the hardware in bulk) but I think long term it'll offer good cost reductions and improved efficiency. Maintenance would be easier in my eyes, and you wouldn't have to interrupt your customers since you can use VMotion/etc. tools to move VMs between "servers" on the fly. I also see it being a more secure setup since the customers wouldn't have access to all the beefy hardware.
> 
> If I want a LAN center with say 64 "terminals" that my customers will use, how powerful would those 64 terminals have to be? Assume that I'd have some monstrous hardware in the back with 256GB of RAM or even more, and enough HD7950s (or whatever GPU, say even 2 7950s per gaming terminal) to make anyone on OCN cry in joy, along with whatever processors would be appropriate (racks of IB-Es anyone?). I'm just being theoretical here for the sake of curiosity. I'd love to start up a business some day and it consists of a LAN center for PC gaming.
> 
> Would love to hear back from anyone that's experienced such a setup.
> 
> 
> 
> To do what you want it would probably be more appropriate to research streaming solutions together with the virtualization one. just making VMs and assigning them video cards won't magically transfer into a lot of gaming speed over a network, even on a LAN. You probably need something like what OnLive does.
> 
> As for the terminals, any thin client should work just fine. They wouldn't need to be powerful at all, just enough to drive the monitor at the right resolution and to use the software that will be receiving the stream from the server, whatever it could be.
Click to expand...

Thanks @ streaming solution. I figured the terminals wouldn't need to be too powerful. Low budget Atoms with 2-4GB RAM should handle it just fine.


----------



## dizzy4

Quote:


> Originally Posted by *Stealth Pyros*
> 
> This sort of setup intrigues me and I'd love to experiment with it in some sort of LAN center setup. It may be very expensive in the start-up phase (maybe not even that since you can buy the hardware in bulk) but I think long term it'll offer good cost reductions and improved efficiency. Maintenance would be easier in my eyes, and you wouldn't have to interrupt your customers since you can use VMotion/etc. tools to move VMs between "servers" on the fly. I also see it being a more secure setup since the customers wouldn't have access to all the beefy hardware.
> 
> If I want a LAN center with say 64 "terminals" that my customers will use, how powerful would those 64 terminals have to be? Assume that I'd have some monstrous hardware in the back with 256GB of RAM or even more, and enough HD7950s (or whatever GPU, say even 2 7950s per gaming terminal) to make anyone on OCN cry in joy, along with whatever processors would be appropriate (racks of IB-Es anyone?). I'm just being theoretical here for the sake of curiosity. I'd love to start up a business some day and it consists of a LAN center for PC gaming.
> 
> Would love to hear back from anyone that's experienced such a setup.


I have often thought about this too. What I was thinking would be good is something like an 8-core AMD (to keep cost down) and boards with plenty of PCI-e slots (like 3 or 4) so you would be able to run 3 gaming computers in one. Another (and probably better) solution would be to get a really expensive server card and use Xen's XCP (Xen Cloud Platform). Quadros and FirePros both have models suited for virtual machines over the network. Then all you would need is a thin client at each station which could cost as little as $60. You might even be able to use consumer grade cards to accelerate the VMs over the network, but you would have to look into it.


----------



## dizzy4

Small update on overcommitting CPU resources:

I had a chance to run some wPrime today and here is what I came up with:

32m and 1024m results are each in 6, 8 and 20 vCPUs from top to bottom.



As you can see, not using all the cores available is not the quickest, but severely overcommitting or splitting CPU resources has little effect. This leads me to believe that performance and load balancing being done in Xen 4.2.0 is superior to older versions. When I tried similar things with the original setup (even just 8 vCPUs) everything ran slow including dom0 and the domU. Even Idle performance was terrible.

I also have a second theory. I have moved on to windows 8 and I looked at all the added features and there is a slough of features that are virtualization related -- including hyper-v. I also set viridian=1 in my .sxp for my virtual machine and windows 8 acknowledges that it is a virtual machine. Either way, this is good news for virtualization and this guide. I am going to officially recommend that readers switch to windows 8









Oh here is a little pic just for fun: QEMU HVM CPU limit is 32 I think (I tried to assign 40) and it thinks I have 2 sockets











So I did some wPrime tests with 32 vCPUs and when I set the benchmark to only use 8 vCPUs the speeds were back in the 10 second range. I am really liking how xen is handling all this with W8


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> It's a VNC message.
> 
> I've disabled the gfx_passthru option and I could VNC into windows and see the nvidia device. Still no signal though. I will try to install the card's drivers and see what happens.
> 
> my config is as follows:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> builder='hvm'
> memory=1024
> vcpus=8
> pae=1
> vif=['bridge=xenbr0,'mac=<random mac here>']
> disk=['phy://dev/vg_xen/windows,hda,w']
> boot='dc'
> pci=['01:00.0']
> vnc=1
> vnclisten="0.0.0.0"
> vncpasswd=""
> usbdevice='tablet'
> localtime=1


I've disabled gfx_passthru too, and I believe that it's the setting that works with many/most graphics cards / guest OSes. It's a good sign that Windows detects the Nvidia graphics card.

Here is my win7.cfg for reference:

Code:



Code:


kernel = "/usr/lib/xen-default/boot/hvmloader"
builder='hvm'
memory = 24576
name = "win7"
vcpus=10
pae=1
acpi=1
apic=1
on_xend_stop="shutdown"
vif = [ 'vifname=win7,type=ioemu,mac=00:16:3e:68:07:07,bridge=xenbr0' ]
disk = [ 'phy:/dev/mapper/lm13-win7,hda,w' , 'phy:/dev/mapper/photos-photo_stripe,hdb,w' , 'phy:/dev/mapper/original-photo_raw,hdc,w' ]
device_model = '/usr/lib/xen-default/bin/qemu-dm'
boot="c"
sdl=0
opengl=1
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
viridian=1
#soundhw='all'
usb=1
usbdevice='tablet'
gfx_passthru=0
pci=[ '02:00.0', '02:00.1' , '00:1a.0' , '0a:00.0' ]


----------



## Chetyre

Quote:


> Originally Posted by *powerhouse*
> 
> I've disabled gfx_passthru too, and I believe that it's the setting that works with many/most graphics cards / guest OSes. It's a good sign that Windows detects the Nvidia graphics card.
> 
> Here is my win7.cfg for reference:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> kernel = "/usr/lib/xen-default/boot/hvmloader"
> builder='hvm'
> memory = 24576
> name = "win7"
> vcpus=10
> pae=1
> acpi=1
> apic=1
> on_xend_stop="shutdown"
> vif = [ 'vifname=win7,type=ioemu,mac=00:16:3e:68:07:07,bridge=xenbr0' ]
> disk = [ 'phy:/dev/mapper/lm13-win7,hda,w' , 'phy:/dev/mapper/photos-photo_stripe,hdb,w' , 'phy:/dev/mapper/original-photo_raw,hdc,w' ]
> device_model = '/usr/lib/xen-default/bin/qemu-dm'
> boot="c"
> sdl=0
> opengl=1
> vnc=1
> vncpasswd=''
> stdvga=0
> serial='pty'
> tsc_mode=0
> viridian=1
> #soundhw='all'
> usb=1
> usbdevice='tablet'
> gfx_passthru=0
> pci=[ '02:00.0', '02:00.1' , '00:1a.0' , '0a:00.0' ]


Some questions:

What is viridian useful for? I read on it a little bit and apparently it is microsoft's solution to make the OS virtualization aware or something, but does it actually affect anything?

Why do you need opengl=1? Doesn't the graphics card take care of opengl in the hvm?

What does serial and tsc do?

Do you really need pae if your system is already x64 (I'm supposing)?


----------



## powerhouse

Quote:


> Originally Posted by *Stealth Pyros*
> 
> This sort of setup intrigues me and I'd love to experiment with it in some sort of LAN center setup. It may be very expensive in the start-up phase (maybe not even that since you can buy the hardware in bulk) but I think long term it'll offer good cost reductions and improved efficiency. Maintenance would be easier in my eyes, and you wouldn't have to interrupt your customers since you can use VMotion/etc. tools to move VMs between "servers" on the fly. I also see it being a more secure setup since the customers wouldn't have access to all the beefy hardware.
> 
> If I want a LAN center with say 64 "terminals" that my customers will use, how powerful would those 64 terminals have to be? Assume that I'd have some monstrous hardware in the back with 256GB of RAM or even more, and enough HD7950s (or whatever GPU, say even 2 7950s per gaming terminal) to make anyone on OCN cry in joy, along with whatever processors would be appropriate (racks of IB-Es anyone?). I'm just being theoretical here for the sake of curiosity. I'd love to start up a business some day and it consists of a LAN center for PC gaming.
> 
> Would love to hear back from anyone that's experienced such a setup.


In addition to what's been said already, one of the bottlenecks is the network performance. Steve Perlman's Onlive has developed some unique hardware and software to get their online gaming service working, as well as providing remote desktop services (DaaS = Desktop as a Service). Here is some interesting stuff, though the editor seems preoccupied with licensing stuff: http://www.brianmadden.com/blogs/gabeknuth/archive/2012/01/25/Breaking-down-OnLive-Desktop-_2D00_-Why-this-is-not-the-desktop-virtualization-solution-you_2700_re-looking-for.aspx

VMware favors PCoIP, which essentially compresses the data before it's send over the network. It's much more complicated, though. Teradici has developed custom chips that run PCoIP which are used by many zero client manufacturers. VMware can support those zero clients via software or - better - via dedicated PCoIP PCIe boards installed in the servers. Microsoft has yet other technologies - RDP and RemoteFX. Citrix is well known for HDX, many years and perhaps still the leader in VDI and remote desktops.

In the Linux world there is NX, and lately also SPICE, a technology developed by Qumranet (sounds familiar? should be) and acquired by Redhat - see here http://en.wikipedia.org/wiki/SPICE_%28protocol%29. Though Redhat dropped Xen in favor of KVM/qemu I sincerely hope that Xen will be able to profit from SPICE.

This is as far as my limited knowledge goes. Some year or two ago I played a little with NX and other remote desktop protocols to see how they perform. My objective was to set up a home server with zero clients for everybody, running their applications in VMs on the server. I soon realized that the remote desktop part would be a major headache, in particular since I insist on *zero clients* (well, I guess a real thin Linux thin-client would be fine too). Alternatively (or in addition to that), the remote desktop should run on iPADs.

Good luck with your LAN center.


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> Some questions:
> 
> What is viridian useful for? I read on it a little bit and apparently it is microsoft's solution to make the OS virtualization aware or something, but does it actually affect anything?
> 
> Why do you need opengl=1? Doesn't the graphics card take care of opengl in the hvm?
> 
> What does serial and tsc do?
> 
> Do you really need pae if your system is already x64 (I'm supposing)?


Excellent questions! Frankly, I had to look it up again - see here for more info: http://wiki.xen.org/wiki/XenConfigurationFileOptions.

viridian: see http://old-list-archives.xen.org/archives/html/xen-users/2009-07/msg00661.html - not sure if that is relevant to my system, but so far I don't see that this option hurts. I think you are right about this option exposing virtualization to the VM, see also http://digitaldj.net/2010/08/25/a-possible-fix-xen-hvm-windows-2008/. In the end, I didn't make any comparison test viridian=1 vs. viridian=0.

pae: pae=1 is the default if not explicitly set to 0. I inserted this line as a reminder to change if things go bad or don't work well. I haven't tried pae=0, though. Both dom0 and domU are 64bit, by the way.

opengl=1 : should provide graphics acceleration in the (VNC) console window, as long as the graphics card drivers of the host are present (if I got this right, and I'm not sure about it, the VNC console uses the graphics card driver of the host/dom0). See here for some hint: http://www.2virt.com/blog/?p=151. In my system it seems to work fine, though again I didn't compare that with opengl=0. This option has no impact once the VGA is passed thru.

serial='pty' : This option enables a serial console. See https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Troubleshooting_Xen-Guest_configuration_files.html. It's actually only relevant for Linux HVMs, not for Windows. It's probably best to delete this option, as I can't see any benefit.

tsc_mode=0 : This is the default. This option is actually quite involving. An in-depth explanation on the settings can be found here: http://svn.openfoundry.org/xenids/xen-4.0.0/docs/misc/tscmode.txt. Again, I use this entry as a reminder on what to check/modify when things go wrong.

None of the settings you asked about are critical or likely to influence whether or not your Windows guest will start and run with VGA passthru. What I tried to do is get a list of all potentially relevant options and list them in the config file with their default values (mostly). In some cases, like opengl and viridian, I chose to enable the feature in the hope it would bring benefits. I haven't yet compared those settings enabled versus not enabled, so I can't say if they actually are beneficial.

About gfx_passthru:


Spoiler: gfx_passthru=BOOLEAN



gfx_passthru=BOOLEAN

Enable graphics device PCI passthrough. This option makes an assigned PCI graphics card become primary graphics card in the VM. The QEMU emulated graphics adapter is disabled and the VNC console for the VM will not have any graphics output. All graphics output, including boot time QEMU BIOS messages from the VM, will go to the physical outputs of the passedthrough physical graphics card.

The graphics card PCI device to passthrough is chosen with pci option, exactly in the same way as normal Xen PCI device passthrough/assignment is done. Note that gfx_passthru does not do any kind of sharing of the GPU, so you can only assign the GPU to one single VM at a time.

gfx_passthru also enables various legacy VGA memory ranges, BARs, MMIOs, and ioports to be passed thru to the VM, since those are required for correct operation of things like VGA BIOS, text mode, VBE, etc.

Enabling gfx_passthru option also copies the physical graphics card video BIOS to the guest memory, and executes the VBIOS in the guest to initialize the graphics card.

Most graphics adapters require vendor specific tweaks for properly working graphics passthrough. See the XenVGAPassthroughTestedAdapters http://wiki.xen.org/wiki/XenVGAPassthroughTestedAdapters wiki page for currently supported graphics cards for gfx_passthru.

gfx_passthru is currently only supported with the qemu-xen-traditional device-model. Upstream qemu-xen device-model currently does not have support for gfx_passthru.

Note that some graphics adapters (AMD/ATI cards, for example) do not necessarily require gfx_passthru option, so you can use the normal Xen PCI passthrough to assign the graphics card as a secondary graphics card to the VM. The QEMU-emulated graphics card remains the primary graphics card, and VNC output is available from the QEMU-emulated primary adapter.

More information about Xen gfx_passthru feature is available on the XenVGAPassthrough http://wiki.xen.org/wiki/XenVGAPassthrough wiki page.


----------



## powerhouse

@dizzy4: Thanks for the wPrime results and your tests re VCPU over-provisioning. Looks like the 4.2 hypervisor handles scheduling nicely. You say that with your "original setup (even just 8 vCPUs) everything ran slow including dom0 and the domU. Even Idle performance was terrible." I assume you refer to the Xen 4.1 hypervisor. Do you mean that everything was slow while running the benchmark, or normal operation?

I downloaded and installed the Prime95 benchmark/application - both under Linux and Windows. I reserve 2 VCPUs for dom0, the rest (10) for Windows. I never noticed any slowdown or bad performance using the 4.1.2 hypervisor. Here my xm info output:


Spoiler: xm info



xm info
host : woody
release : 3.2.0-35-generic
version : #55-Ubuntu SMP Wed Dec 5 17:42:16 UTC 2012
machine : x86_64
nr_cpus : 12
nr_nodes : 1
cores_per_socket : 6
threads_per_core : 2
cpu_mhz : 3200
hw_caps : bfebfbff:2c100800:00000000:00003f40:13bee3bf:00000000:00000001:00000000
virt_caps : hvm hvm_directio
total_memory : 32740
free_memory : 2391
free_cpus : 0
xen_major : 4
xen_minor : 1
xen_extra : .2
xen_caps : xen-3.0-x86_64 xen-3.0-x86_32p hvm-3.0-x86_32 hvm-3.0-x86_32p hvm-3.0-x86_64
xen_scheduler : credit
xen_pagesize : 4096
platform_params : virt_start=0xffff800000000000
xen_changeset : unavailable
xen_commandline : placeholder iommu=1 dom0_mem=6G,max:6G vga=mode-0x031A console=vga
cc_compiler : gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5)
cc_compile_by : marc.deslaurier
cc_compile_domain : ubuntu.com
cc_compile_date : Tue Dec 11 16:32:07 UTC 2012
xend_config_format : 4



Here my Prime95 results under Linux dom0 (while Windows domU was running idle):
Timing FFTs using 12 threads on 6 physical CPUs.
Best time for 61 bit trial factors: 2.101 ms.
Best time for 62 bit trial factors: 2.137 ms.
Best time for 63 bit trial factors: 2.407 ms.
Best time for 64 bit trial factors: 2.472 ms.
Best time for 65 bit trial factors: 2.873 ms.
Best time for 66 bit trial factors: 3.378 ms.
Best time for 67 bit trial factors: 3.354 ms.
Best time for 75 bit trial factors: 3.261 ms.
Best time for 76 bit trial factors: 3.271 ms.
Best time for 77 bit trial factors: 3.268 ms.

Here the same test under Windows domU using 10 VCPUs:
Best time for 61 bit trial factors: 2.675 ms.
Best time for 62 bit trial factors: 2.618 ms.
Best time for 63 bit trial factors: 3.204 ms.
Best time for 64 bit trial factors: 3.284 ms.
Best time for 65 bit trial factors: 4.314 ms.
Best time for 66 bit trial factors: 5.367 ms.
Best time for 67 bit trial factors: 5.343 ms.
Best time for 75 bit trial factors: 5.237 ms.
Best time for 76 bit trial factors: 5.228 ms.
Best time for 77 bit trial factors: 5.224 ms.

I did several runs of Prime95 and results vary. The above is about average. Obviously the 2 missing VCPUs do have some impact. However, I also noticed that the Windows CPU usage meter went down from initially 100% to some 20% as CPU cores were added. Strange, isn't it?


----------



## powerhouse

Quote:


> Originally Posted by *Ex3c*
> 
> Hello, i am interested in the power consumption of the graphics card, when used in the virtual machine.
> For example does "Zero Core Power" work with an AMD GPU?
> And what happens when the virtual machine is shut down: Is also the GPU shutdown, and consumes really zero power?


It took me some time to set up my wattmeter. My PC is now connected to a digital wattmeter that continually measures and displays the power consumption in Watt. Here some results:

Linux dom0 only, without any open windows/apps: 115W
Linux dom0 and Windows domU, Firefox runs on Linux with this page open: 117W
Prime95 *benchmark* on Windows: ~233W at start, then down to 182W at the end when all 10 threads are used (looks like a problem with the Windows version)
Prime95 (calculating primes) on Windows: 253W (constant), with Windows CPU meter on 100%
Prime95 (calculating primes) on Linux (w/o Windows domU running): 307W !!!
Prime95 on Linux (see above), then starting Windows domU: Up to 320W during and shortly after booting Windows, then 307W
Prime95 on both Linux dom0 and Windows domU: 293W (Windows CPU meter at maximum)

I could do more, for example file transfers via Samba or other checks.

All in all I'm surprised how low the power consumption is.

My software:
Xen hypervisor 4.1.2 with Linux Mint 13 Mate 64bit dom0 running a graphical desktop
Windows Pro 64bit as domU

Hardware powered by the Seasonic 660W X-Series Gold PSU:
Intel 3930K CPU
Asus Sabertooth X79
ATI 6450 GPU (for dom0)
Nvidia Quadro 2000 (for domU)
3x WD20EARS Green 2TB drives
1x WD10?? Green 1TB drive
1x WD5000AAK? 500GB drive
1x Sandisk Extreme 120GB SSD
1x LG DVD R/W
1x Transcend PDC3 USB 3.0 / SATA-III combo card
1x Edimax USB KVM switch
1x Microsoft 600 (wired) keyboard
1x Microsoft basic USB mouse
Corsair 500R with 3x120mm fans, and 1x250mm side fan
Noctua NH-D14 SE2011 CPU Cooler with 2 fans

Amazing how much stuff is connected to a PC power supply.


----------



## powerhouse

Another performance test, this time networking. I finally got to install netperf 2.5.0. Here some results:

Linux dom0 to Windows 7 domU using a xen bridge:

Code:



Code:


netperf -H my_guest_OS_IP
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to my_guest_OS_IP (my_guest_OS_IP) port 0 AF_INET : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec

 65535  16384  16384    10.00    5188.43

Power consumption during test: 200W

Linux dom0 to remote Linux Mint laptop via Cisco / Linksys EA3500 router (Gigabit links):

Code:



Code:


netperf -H my_remote_laptop_IP
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to my_remote_laptop_IP (my_remote_laptop_IP) port 0 AF_INET : demo
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.01     601.65

Power consumption during test: 118W (like idle)

Remote Linux Mint laptop to Windows domU via Cisco router (Gigabit):

Code:



Code:


netperf -H my_guest_OS_IP
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to my_guest_OS_IP (my_guest_OS_IP) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec

 65535  16384  16384    10.00     523.79

Power consumption during test: 138-153W (several reruns)

Remote Linux Mint laptop to Linux Mint dom0 via Cisco router (Gigabit):

Code:



Code:


netperf -H my_host_dom0_IP
MIGRATED TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to my_host_dom0_IP (my_host_dom0_IP) port 0 AF_INET
Recv   Send    Send                          
Socket Socket  Message  Elapsed              
Size   Size    Size     Time     Throughput  
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.01     534.31

Power consumption during test: 125W

Network throughput looks good under netperf. So I'm just a little closer to explaining why samba or nfs file transfers to *remote* PCs are slow, extremely slow!

By the way, Samba file transfers between dom0 and domU are REAL fast, as you can expect when looking at the throughput (it's like a 10Gig link). But the power consumption also points to CPU resources used.


----------



## Chetyre

Quote:


> Originally Posted by *powerhouse*
> 
> Excellent questions! Frankly, I had to look it up again - see here for more info: http://wiki.xen.org/wiki/XenConfigurationFileOptions.
> 
> viridian: see http://old-list-archives.xen.org/archives/html/xen-users/2009-07/msg00661.html - not sure if that is relevant to my system, but so far I don't see that this option hurts. I think you are right about this option exposing virtualization to the VM, see also http://digitaldj.net/2010/08/25/a-possible-fix-xen-hvm-windows-2008/. In the end, I didn't make any comparison test viridian=1 vs. viridian=0.
> 
> pae: pae=1 is the default if not explicitly set to 0. I inserted this line as a reminder to change if things go bad or don't work well. I haven't tried pae=0, though. Both dom0 and domU are 64bit, by the way.
> 
> opengl=1 : should provide graphics acceleration in the (VNC) console window, as long as the graphics card drivers of the host are present (if I got this right, and I'm not sure about it, the VNC console uses the graphics card driver of the host/dom0). See here for some hint: http://www.2virt.com/blog/?p=151. In my system it seems to work fine, though again I didn't compare that with opengl=0. This option has no impact once the VGA is passed thru.
> 
> serial='pty' : This option enables a serial console. See https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/sect-Virtualization-Troubleshooting_Xen-Guest_configuration_files.html. It's actually only relevant for Linux HVMs, not for Windows. It's probably best to delete this option, as I can't see any benefit.
> 
> tsc_mode=0 : This is the default. This option is actually quite involving. An in-depth explanation on the settings can be found here: http://svn.openfoundry.org/xenids/xen-4.0.0/docs/misc/tscmode.txt. Again, I use this entry as a reminder on what to check/modify when things go wrong.
> 
> None of the settings you asked about are critical or likely to influence whether or not your Windows guest will start and run with VGA passthru. What I tried to do is get a list of all potentially relevant options and list them in the config file with their default values (mostly). In some cases, like opengl and viridian, I chose to enable the feature in the hope it would bring benefits. I haven't yet compared those settings enabled versus not enabled, so I can't say if they actually are beneficial.
> 
> About gfx_passthru:
> 
> 
> Spoiler: gfx_passthru=BOOLEAN
> 
> 
> 
> gfx_passthru=BOOLEAN
> 
> Enable graphics device PCI passthrough. This option makes an assigned PCI graphics card become primary graphics card in the VM. The QEMU emulated graphics adapter is disabled and the VNC console for the VM will not have any graphics output. All graphics output, including boot time QEMU BIOS messages from the VM, will go to the physical outputs of the passedthrough physical graphics card.
> 
> The graphics card PCI device to passthrough is chosen with pci option, exactly in the same way as normal Xen PCI device passthrough/assignment is done. Note that gfx_passthru does not do any kind of sharing of the GPU, so you can only assign the GPU to one single VM at a time.
> 
> gfx_passthru also enables various legacy VGA memory ranges, BARs, MMIOs, and ioports to be passed thru to the VM, since those are required for correct operation of things like VGA BIOS, text mode, VBE, etc.
> 
> Enabling gfx_passthru option also copies the physical graphics card video BIOS to the guest memory, and executes the VBIOS in the guest to initialize the graphics card.
> 
> Most graphics adapters require vendor specific tweaks for properly working graphics passthrough. See the XenVGAPassthroughTestedAdapters http://wiki.xen.org/wiki/XenVGAPassthroughTestedAdapters wiki page for currently supported graphics cards for gfx_passthru.
> 
> gfx_passthru is currently only supported with the qemu-xen-traditional device-model. Upstream qemu-xen device-model currently does not have support for gfx_passthru.
> 
> Note that some graphics adapters (AMD/ATI cards, for example) do not necessarily require gfx_passthru option, so you can use the normal Xen PCI passthrough to assign the graphics card as a secondary graphics card to the VM. The QEMU-emulated graphics card remains the primary graphics card, and VNC output is available from the QEMU-emulated primary adapter.
> 
> More information about Xen gfx_passthru feature is available on the XenVGAPassthhttp://www.overclock.net/t/1348236/please-need-your-help-to-avoid-formatrough http://wiki.xen.org/wiki/XenVGAPassthrough wiki page.


TSC is indeed very interesting, but more for server use than for us, so I guess 0 or 1 are the best. Thanks for the links, they were very informative. I Think I'll write my config enterely with all the defaults and commented so I remember all these.

I'd like to know more about viridian though. If it makes the hvm aware of virtualization, it might lead to better integration and drivers. It's an interesting subject.

I haven't had time to mess with my VM lately, but what should I expect exactly if I'm not using gfx_passthru? That option makes the graphics card primary on the domU instead of secundary, so can I just make it primary in the domU? I wonder if that is why my monitor has no signal


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> TSC is indeed very interesting, but more for server use than for us, so I guess 0 or 1 are the best. Thanks for the links, they were very informative. I Think I'll write my config enterely with all the defaults and commented so I remember all these.
> 
> I'd like to know more about viridian though. If it makes the hvm aware of virtualization, it might lead to better integration and drivers. It's an interesting subject.
> 
> I haven't had time to mess with my VM lately, but what should I expect exactly if I'm not using gfx_passthru? That option makes the graphics card primary on the domU instead of secundary, so can I just make it primary in the domU? I wonder if that is why my monitor has no signal


I think the gfx_passthru option is a bit misleading. A good explanation can be found here: http://wiki.xen.org/wiki/XenVGAPassthrough#The_effect_of_gfx_passthru.3D_option. I quote:
Quote:


> When you specify "gfx_passthru=1" the passthru graphics card will be made the primary graphics card in the VM, and the Xen Qemu-dm emulated Cirrus graphics card is disabled.
> 
> If you use "gfx_passthru=0", or don't have gfx_passthru= option at all, then the Xen Qemu-dm emulated Cirrus graphics card will be the primary in the VM, and the passthru graphics card will be secondary.


That same link a little further down at "Status of VGA graphics passthru in Xen" says with regard to Xen 4.1.1:
Quote:


> Passing thru AMD/ATI Radeon/FirePro/FireGL adapters as secondary to the VM should work out-of-the-box (you need latest ATI gfx drivers in the VM). Secondary means the Xen Qemu-dm virtual Cirrus adapter is the primary where you see the VM BIOS etc when powering on the VM, and the passthru adapter is secondary, so you can enable/use it after the OS in the VM has started and you've installed drivers for the adapter.


Up until now I misunderstood the meaning of "secondary adapter". The way I understand it now (reading the above), it's not your onboard or PCIe #1 slot adapter being the primary adapter versus the add-on VGA card or PCIe #2 slot adapter being the secondary one. Instead, it's from the perspective of the Windows guest:
When booting the (Windows) guest:
1. gfx_passthru=0 means that Windows boots with the qemu-dm emulated Cirrus graphics adapter. Once Windows has started, the user can then install and activate the driver for his/her graphics card. Access to Windows is via VNC, there is no graphics output to the passed thru VGA adapter.
Upon reboot of the guest, Windows again boots the Cirrus adapter, but later loads the driver for the secondary (physical) graphics adapter and switches the video output to that adapter (provided it is set up correctly under Windows).

2. gfx_passthru=1 means that Windows is not presented with the emulated Cirrus adapter, but boots directly using the physical (passed-thru) adapter and hopefully manages to to use it. But there is more to this option: "Xen copies the VGA BIOS to the HVM guest memory and re-executes it there to initialize the graphics card". In some cases vendor specific tools are needed to copy the VGA BIOS to file to be loaded from there. (See above link for more.)

In my case option 1 works, option 2 doesn't. When booting Windows I can connect to the Cirrus emulated display via VNC and I see the typical Windows boot animation until the graphics driver loads and switches to my physical display. Here a screen shot of the VNC window:


----------



## wowoteur

Hello,
sorry for my english... i am french.
I make a xen with fedora like the tutorial.
I give my cg to a vm for make a HTPC.
When i start the vm where ubuntu is, i see the video card, when i write lspci in a terminal.
But i can active in display.
And nothing arrive in my TV.

Second question, what is the solution for configuration of bridge.
because i do :

for em1:
BRIDGE=virbro

for virbr0:
DEVICE=virbr0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0

but nothing....

Thank you for your helping


----------



## Chetyre

Quote:


> Originally Posted by *wowoteur*
> 
> Hello,
> sorry for my english... i am french.
> I make a xen with fedora like the tutorial.
> I give my cg to a vm for make a HTPC.
> When i start the vm where ubuntu is, i see the video card, when i write lspci in a terminal.
> But i can active in display.
> And nothing arrive in my TV.
> 
> Second question, what is the solution for configuration of bridge.
> because i do :
> 
> for em1:
> BRIDGE=virbro
> 
> for virbr0:
> DEVICE=virbr0
> TYPE=Bridge
> BOOTPROTO=dhcp
> ONBOOT=yes
> DELAY=0
> 
> but nothing....
> 
> Thank you for your helping


your configuration for em1 reads BRIDGE=virbro, it should be BRIDGE=virbr0. Are you sure your device is em1 btw?

Do you want your VM to be an Ubuntu HTPC? Sorry I couldn't understand very well


----------



## wowoteur

Yes my nic is em1.
And it's virbr0, i do a mistake when i write my post.
I want to create a vm for a htpc with ubuntu and plex.


----------



## Rezz

I finally got my HTPC/gaming/general rig going, with thanks to dizzy4 and powerhouse for their excellent guides.

The rig comprises of:

*Case:* Thermaltake Element Q (added a chassis fan instead of a vertical mount HDD)
*Mainboard:* Asrock Z77E-ITX (Bios version 1.40)
*Memory:* gSkill Ripjaws 8GB DDR3 - 1600MHz (2 x 4GB kit)
*CPU:* Xeon E3 1225V2
*CPU Cooler:* Xigmatek Praeton LD963
*GPU:* HD P4000 (Dom0), Powercolor 7750 1GB DDR5 (DomU)
*HDD:* Samsung 830 128GB SSD and WD 1TB Green
*Keyboard and Mouse:* Microsoft Entertainment 7000 (Dom0), iPazzport 2.4GHz Mini Keyboard (DomU)
*TV Tuner:* Leadtek DTV USB Dongle Dual DVB-T (DomU)
*DVD/CD:* Pioneer DVD RW (Dom0)
*OS/Main software:* Xen 4.1.3 64 bit, Mint 14 Cinnamon 64 bit (Kernel 3.7) (Dom0), Windows 8 Pro 64 bit + XBMC Frodo (DomU)
*Display:* 32" Full HD LCD (Dom0, using DVI), 58" Full HD Plasma (Dom0, using HDMI)

*CPU Geometry:* Dom0 4/2 Cores, DomU 2 Cores
*RAM Geometry:* Dom0 (and system) 4096MB, DomU 4096MB
*HDD Geometry (LVM):* Samsung 830 (Dom0: 1GB boot, 15GB root, 45GB home) (DomU: remainder NTFS), WD 1TB (Dom0: 10GB swap, 215GB storage) (DomU: remainder NTFS)

DomU config:

Code:



Code:


kernel = '/usr/lib/xen-4.1/boot/hvmloader'
builder='hvm'
memory=4096
name="win8"
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'type=ioemu,mac=00:16:3e:##:##:##,bridge=xenbr0' ] # Insert your own MAC
disk = [ 'phy:/dev/mapper/lm14-win8,hda,w' , 'phy:/dev/mapper/hdd-spaceU,hdb,w' ]
device_model = '/usr/lib/xen-4.1/bin/qemu-dm'
on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'
boot='c'
sdl=0
opengl=1
vnc=1
vncpasswd=''
stdvga=0
serial='pty'
tsc_mode=0
viridian=1
gfx_passthru=0
pci=[ '01:00.0', '01:00.1' , '00:1a.0' , '05:00.0' ] # 01:00.0 and 01:00.1 is HDMI A/V (Powercolor 7750), 00:1a.0 is onboard USB controller #2, 05:00.0 is the Asmedia USB 3.0 controller
xen_platform_pci=1
localtime=1

Problems and how they were overcome:

"Kernel panic" and hang when booting Mint 14 Cinnamon (or Mate) with Xen 4.1.3 (or 4.1.4). Overcome by updating the Linux kernel to 3.7 (3.8 rc1 works as well).

Flaky USB keyboard/mouse performance with Win 8 DomU and the TV Tuner not recognised at boot (a very, very frustrating problem). Overcome by disabling "fast startup" and "hibernate" in Power Options/System Settings in Win 8. It is *not* a Xen/Dom0 problem. There has been no difference to boot or shutdown times with "fast startup" and "hibernate" disabled.

No text displayed in XBMC when the 7750 driver is upgraded in Win 8. Overcome by just using the 7700 series driver Win 8 detects/uses during install. The current, updated ATI drivers (from the AMD website) have a known issue with XBMC (no text).
*Do not ever try to use the driver/software installer exe or iso/disc that came with the card or from the AMD website, it will crash your rig* and you will have to reinstall nearly everything. Instead, extract the actual driver folder/files from the disc or iso and let Windows find the drivers in Device Manager if you have to update for any reason.

Lost Win 8 activation when GPLPV is installed. Overcome by phoning Microsoft for a new "key".
Flaky USB and ethernet performance with GPLPV. Overcome by disabling USB passthrough (it doesn't appear to be needed when USB controllers are being passed through) and disabling network function when installing GPLPV. I'm now using gplpv_Vista2008x64_0.11.0.369.msi from http://www.meadowcourt.org/private/gplpv_Vista2008x64_0.11.0.369.msi .
It is now working very well, with 7.9 (and sometimes 8.1) for HDD performance in Win 8. The overall result is 6.9 (due to the graphics card), which is fine for me.

Lost Win 8 activation after installing Windows Media Centre for Win 8 (yes, I had a legitimate WMC key emailed to me).
Don't use Windows Media Centre. It is truly a pain. Even despite being unable to activate, it would always hang when attempting to scan for TV channels.

There were several other minor problems with Mint 14 and the Xen install that I don't really recall (like auto-booting DomU), but they were relatively easily solved by a bit of quick googling.
Win 8 also had a few other minor problems (that weren't anything to do with Xen or being virtualised) that had to be sorted.

Future upgrades and mods:

Improve DomU/Win 8 boot times (it currently takes several minutes to boot from a cold start)
Xen 4.2
Kernel 3.8
Better PSU (450W SFX)......just in case (the included 220W PSU has been ok so far)
USB Bluray for DomU
Larger capacity and faster mechanical HDD
Icybox 3.5" cardreader and USB hub module
Decent IR remote to use with the iPazzport keyboard for DomU
Game controller and maybe a better single slot GPU (if ever needed)
Better sound system (might build an all valve/tube one when time permits)


----------



## powerhouse

Hello Rezz, great news. Looks like a nice rig!

Thanks for summarizing your installation - your information is very helpful. In particular the networking part, which I couldn't entirely solve for my domU. I also noticed I forgot the localtime=1 entry.

You'll need some more HDDs for your media center. If you're looking for speed, may I suggest getting 2 identical low-cost "green" HDD and running them LVM striped - it works beautifully, even between domU/dom0. GPLPV driver for the HDD is necessary, though. By the way, you can get signed drivers from the univention website (if you plan to replace the drivers, you may have to get the uninstall batch file from meadows and read the comments!!!).

Several minutes boot time is a lot! My rig takes about 35 sec. cold boot into Linux Mint 13 dom0, then another 30 sec. to load Windows 7 from the moment I press the launcher.


----------



## dizzy4

Great Post Rezz! I had to do the same thing with Microsoft over the ohone on friday because of GPLPV drivers. I am going to put a notice about GPLPV in the first post too! Credits to all of you in the next release. I am thinking of making it a Mint release due to the centOS issues I am having getting a working live-cd image.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Great Post Rezz! I had to do the same thing with Microsoft over the ohone on friday because of GPLPV drivers. I am going to put a notice about GPLPV in the first post too! Credits to all of you in the next release. I am thinking of making it a Mint release due to the centOS issues I am having getting a working live-cd image.


Oh wow - that's quite a change, from Fedora/CentOS to Mint. I don't know how much experience you have with Mint, so here a few thoughts you may want to consider:
1. Linux Mint is a desktop OS - they don't have a server edition. That said it uses the Ubuntu repository and has a wealth of packages, including Xen (4.1.x with both LM13 and LM14).
2. It doesn't come with the latest kernel and packages.
3. pciback is NOT compiled into the kernel, which makes things a bit different and perhaps limits the ability to use it for primary VGA passthru. On the other hand, there should be a way to grab the VGA and assign it to pciback before the graphics driver takes control, or simply blacklist the driver. Anyway, you should be aware of that.
4. No LVM install option, but the latest LM 14 release has at least some LVM support out of the box. It's not a big deal to get it working, and I can send you the link to my how-to if needed.

I'm very prejudiced about Mint, but wanted to acknowledge its shortcomings.


----------



## Chetyre

Ok, so I finally got to mess with my VM for a little bit again...

Well, it doesn't work. Device manager says it is working properly, the drivers are installed, but it is BSODing a huge lot, and if I try to access the settings, it says I am not using a display that is attached to a nvidia GPU (I am). No idea what I can do now...


----------



## dizzy4

Quote:


> Originally Posted by *Chetyre*
> 
> Ok, so I finally got to mess with my VM for a little bit again...
> 
> Well, it doesn't work. Device manager says it is working properly, the drivers are installed, but it is BSODing a huge lot, and if I try to access the settings, it says I am not using a display that is attached to a nvidia GPU (I am). No idea what I can do now...


It might have to do with you using an Nvida card. There are patches for better compatibility if you care to compile xen from source. However, it seems that you are close.

Try setting nographic=1 in the VM config and try disabling the other adapter in device manager. Hopefully that will solve the problems.
Quote:


> Originally Posted by *powerhouse*
> 
> Oh wow - that's quite a change, from Fedora/CentOS to Mint. I don't know how much experience you have with Mint, so here a few thoughts you may want to consider:
> 1. Linux Mint is a desktop OS - they don't have a server edition. That said it uses the Ubuntu repository and has a wealth of packages, including Xen (4.1.x with both LM13 and LM14).
> 2. It doesn't come with the latest kernel and packages.
> 3. pciback is NOT compiled into the kernel, which makes things a bit different and perhaps limits the ability to use it for primary VGA passthru. On the other hand, there should be a way to grab the VGA and assign it to pciback before the graphics driver takes control, or simply blacklist the driver. Anyway, you should be aware of that.
> 4. No LVM install option, but the latest LM 14 release has at least some LVM support out of the box. It's not a big deal to get it working, and I can send you the link to my how-to if needed.
> 
> I'm very prejudiced about Mint, but wanted to acknowledge its shortcomings.


Yeah it's not a big deal which distro is used, but I am looking for something that I can easily package into a livecd or live-usb (livecd + persistent storage) to make it easier for users.


----------



## Chetyre

Quote:


> Originally Posted by *dizzy4*
> 
> It might have to do with you using an Nvida card. There are patches for better compatibility if you care to compile xen from source. However, it seems that you are close.
> 
> Try setting nographic=1 in the VM config and try disabling the other adapter in device manager. Hopefully that will solve the problems.


I am using nographic=1 and I have disabled the other adapter. I think I'll try using windows 7 instead of 8, maybe I'll have better luck.

But I don't mind compiling it from source. Could you direct me to the relevant patches?


----------



## dizzy4

Links to the patches can be found at the bottom of this page:

http://wiki.xen.org/wiki/Xen_VGA_Passthrough

There are several and I have no clue what the status is currently.

It seems like you are getting close. Can you get anything at all to display on the Nvidia card? If it is displaying and BSODing later there might be some driver updates you could try. Are you also passing through the Audio portion (if it has one)?


----------



## Chetyre

It is not displaying anything, it just BSODs randomly connected to VNC which is the only thing available for me. There is no audio portion to this card, the only audio device I have is my onboard one.

What does happen though is that my monitor glitches on the analog part while the VM is running. I am using the same monitor. VGA is onboard, and DVI is the nvidia card. I don't think that should happen, since they are separate display adaptors.


----------



## dizzy4

Quote:


> Originally Posted by *Chetyre*
> 
> It is not displaying anything, it just BSODs randomly connected to VNC which is the only thing available for me. There is no audio portion to this card, the only audio device I have is my onboard one.
> 
> What does happen though is that my monitor glitches on the analog part while the VM is running. I am using the same monitor. VGA is onboard, and DVI is the nvidia card. I don't think that should happen, since they are separate display adaptors.


I notice that happens when I try to pass through my HD 2000 onboard graphics. I Assume that it isn't patched properly so I think that patching would be the best bet. A second thought is that it isn't releasing all the resources properly from the host. It might be trying to assign memory from the IGP in a weird way or getting confused with linux drivers. Try booting in text mode or generic VGA mode and seeing what happens before you try patching.


----------



## Chetyre

I'm running a headless system. I can try changing the runtime to single user, but I don't think that will do anything.

The soonest I can guarantee I can play with my pc again will be on Thursday, so until then I'd appreciate any ideas of things to try.

EDIT: I went to my motherboard's (Asrock Z77 Extreme4) page and there is a bios update that mentions something about VT-D, but it sounds very engrish to me.

Quote:


> 1. Support UEFI PXE boot.
> 2. Modify VT-d Capability will be unsupported after flashing BIOS P2.50.


I am running bios version 2.50. Maybe that update broke VT-D support somehow? I'll try finding more about it.


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> I'm running a headless system. I can try changing the runtime to single user, but I don't think that will do anything.
> 
> The soonest I can guarantee I can play with my pc again will be on Thursday, so until then I'd appreciate any ideas of things to try.
> 
> EDIT: I went to my motherboard's (Asrock Z77 Extreme4) page and there is a bios update that mentions something about VT-D, but it sounds very engrish to me.
> I am running bios version 2.50. Maybe that update broke VT-D support somehow? I'll try finding more about it.


According to http://www.asrock.com/mb/Intel/Z77%20Extreme4/?cat=Download&os=BIOS, VT-d is unsupported with BIOS 2.70, if I understand that correctly. Anyway, best to contact Asrock and find out. Also, I would be most interested in knowing why they broke VT-d, as well as when they are going to fix it.

EDIT: There are lots of references in Xen related websites and wikis that testify that this board supports VT-d. If Asrock has dropped that support, this information needs to be updated and users should be warned NOT to buy that board.

EDIT 2: The same goes for the Z77 Extreme6 board - same BIOS upgrade note.


----------



## Chetyre

This is worrisome. I have contacted Asrock and asked for clarifications on the matter. I'll keep you updated.


----------



## powerhouse

Quote:


> Originally Posted by *Chetyre*
> 
> This is worrisome. I have contacted Asrock and asked for clarifications on the matter. I'll keep you updated.


Since you are running BIOS version 2.50 you should not be affected, as long as you don't upgrade. Still, it is worrisome and getting some explanations from the vendor would help. The note of the BIOS release 2.70 is also a bit vague. I could as well read it this way:

1. If VT-d was enabled prior to the BIOS upgrade, it will be enabled after the upgrade (but cannot be disabled).
2. If VT-d was disabled prior to the upgrade, you cannot enable it after the BIOS upgrade.

Let's put this way: I wouldn't want to be the guinea pig to test it.


----------



## dizzy4

I think the VT-d wording in their statements only means that the user will not have the ability to turn it on or off. I think that means it is automatically on if the CPU supports it and off if not.


----------



## BulliteShot

Success!

*Does anyone know how to get the DomU to send sound to Dom0? Or is this a job for a PCI sound card?*

I played about with a GTX 670 for ages, could not get it to work. Gave up and bought an ATI card.

I would like to report that I got an MSI TwinFrozr 3GDS R7950 working out of the box.

Here are my system specs.

Graphics Card: MSI TwinFrozr 3GDS R7950
Motherboard: Gigabyte B75-D3V
Fedora 16 64-bit
Kernel 3.6.11
Xen 4.4.1
DomU: Windows 7 64-bit with Latest Catalyst Driver 12-10
In the BIOS, I have set my integrated graphics from AUTO to ENABLED. I also changed the default adapter from AUTO to IGFX.

So, I'm using the integrated graphics for the linux host on one screen and I have windows 7 running on my right screen.

I'm using Synergy to transfer the mouse to and from windows. It works well with minimal input lag while gaming. Note that I had to enable the relative mouse movement in synergy to get minecraft input working. This is the only game I have tested so far.
Synergy does NOT work with UAC (User Account Control) so I disabled that. There is a work around, where you can use the VNC session to control the mouse during a UAC dialog. The video on my VNC does not work after the graphics driver is loaded (It just shows Windows is starting) This isn't a problem, but the mouse input still works to click on the UAC dialog









These commands are run on startup to unbind the graphics card for xen

Code:



Code:


#VGA
echo "1002 679a" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.0 > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers/pci-stub/bind

#VGA Audio
echo "1002 aaa0" > /sys/bus/pci/drivers/pci-stub/new_id
echo 0000:01:00.1 > /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo 0000:01:00.1 > /sys/bus/pci/drivers/pci-stub/bind


----------



## powerhouse

@BulliteShot: I suppose you don't have a HDMI connection to your second screen (for Windows), and built-in speakers or headphone plugs.

I tried pulseaudio, but gave up when I couldn't get the ancient Windows drivers to forward sound. After spending several hours on it in vain, I went to the next computer shop and bought a $5 USB sound stick, plugged it into a passed-thru USB port, and wired the headphone output to the line in of the motherboard. This worked out of the box. Of course, you could get a PCI sound card and pass it thru to Windows, again with a wire from the sound card to the on-board line-in port.

Thanks for your input on synergy - I had about the same experience. Supposedly the "elevate" option should solve the admin login, but I couldn't make it work. In the end I settled for a USB KVM switch, which works great in all situations, with the disadvantage that I don't have copy-paste between Windows and Linux. If I need that, I run the remote desktop from within Windows to my Linux dom0.

Could you share your win7.cfg file?

Last question: How did you configure networking? This is my last real obstacle - my Windows 7 domU doesn't play Youtube without getting stuck every few seconds, nor does it play video files residing on my networked media tank (using VLC). Any observations on that?


----------



## Chetyre

well I tried with a windows XP HVM, same thing. Thing doesn't start with a code 10... Guess I'll just bite the bullet and buy an ATI card. I really wish I could keep using nvidia, but all my project are ruined if I can't get this to work.


----------



## dizzy4

Quote:


> Originally Posted by *Chetyre*
> 
> well I tried with a windows XP HVM, same thing. Thing doesn't start with a code 10... Guess I'll just bite the bullet and buy an ATI card. I really wish I could keep using nvidia, but all my project are ruined if I can't get this to work.


Ruined how? I am not sure what you are trying to do, but have you tried patching Xen and compiling from source? Don't give up yet


----------



## nbhusain

Hi, i managed to create a gaming vm using xen-4.1.4 and uefi duet booting directly off my sata controller (native ahci boot) with windows 7 X64 in uefi mode, my setup is:


AMD-FX 8120
CROSSHAIR V FORMULA 1703 bios
12gb ram
RADEON 5450 primary
RADEON 6950 secondary
OS: archlinux
Basically what im doing is passing 1 of my sata controllers to the vm (the MB has 2), booting xen with a DUET disk image, duet provides drivers for usb2, usb3 and ahci, so it detects my sata and usb controllers, keyboard, mouse, pendrives etc, after that it boots windows 7 in uefi mode from one disk connected to the controller. The Windows installer did not work from duet so i had to install it natively from a pendrive (i believe you can convert a mbr setup to uefi if your mobo doesnt suport it), after that i launched the vm and continued with the windows installation.

i also patched xen to enable gfx_passthru option in xen, that means you can see the windows loading logo and desktop even if you dont have the drivers installed.

These are the patches i used.
http://www.filesend.net/download.php?f=7967f17f8c538b860246275360953424
http://www.filesend.net/download.php?f=07f4558e2b1f868dfc5824154058a8cc

The first patch is for loading your vga bios rom if your card is secondary, otherwise you'll only need the second patch

You must copy your vga bios to tools/firmware/vgabios/vgabios-pt.bin when building xen ( only needed using the first patch )

A precompiled duet image i created can be found here: http://www.filesend.net/download.php?f=3d9ec1033c6658c0c77a8a7fafbeafc9
If your interested in building it yourself the source code can be found here: https://edk2.svn.sourceforge.net/svnroot/edk2/trunk/edk2/ you might need to tweak a few things tho.

More info on duet: http://www.rodsbooks.com/bios2uefi/
Convert WIN7X64 MBR to UEFI: http://gitorious.org/tianocore_uefi_duet_builds/pages/Windows_x64_BIOS_to_UEFI


----------



## powerhouse

@ nbhusain: Interesting approach. I have some questions.

1. Was there a reason for choosing UEFI DUET boot? Faster Windows boot?

2. Does UEFI DUET boot only work when directly booting from disk (AHCI)?

3. How about disk performance of the passed-thru disk(s) (AHCI)? Is it faster than the regular LVM based disk access, using the GPLPV drivers? N.B.: As far as I have seen, the GPLPV drivers (under Windows 7) greatly affect disk I/O speed.

4. You say you boot Xen and afterwards Windows? Though I know that dom0 is nothing but a privileged VM, don't you need to run the dom0 (Linux) to start a domU (Windows)? I'm a bit confused.

5. Do you care to share the Windows cfg file?

Forgive me for asking so many questions, but it's the first time I read about this option. And thanks for sharing this info.


----------



## Rezz

Quote:


> Originally Posted by *powerhouse*
> 
> You'll need some more HDDs for your media center. If you're looking for speed, may I suggest getting 2 identical low-cost "green" HDD and running them LVM striped - it works beautifully, even between domU/dom0. GPLPV driver for the HDD is necessary, though. By the way, you can get signed drivers from the univention website (if you plan to replace the drivers, you may have to get the uninstall batch file from meadows and read the comments!!!).


Thanks powerhouse.
I will eventually get an additional USB 3 external storage hdd (I might run into a bit of a temperature problem if I install another larger HDD in the case I'm using and another simply won't fit).

The version of GPLPV is the very latest I could find (.369 as opposed to .356 or .357) and appears to be signed. It seems to run a lot smoother and didn't cause a problem with Win 8 activation.
Quote:


> Originally Posted by *powerhouse*
> 
> Several minutes boot time is a lot! My rig takes about 35 sec. cold boot into Linux Mint 13 dom0, then another 30 sec. to load Windows 7 from the moment I press the launcher.


Yes. 2 or 3 minutes (after Mint boots) is very slow for Win 8. I'm still trying to solve this.
Quote:


> Originally Posted by *dizzy4*
> 
> I had to do the same thing with Microsoft over the ohone on friday because of GPLPV drivers.


Have you tried the .369? It seems to not cause the activation headaches I had with .357.


----------



## BulliteShot

@powerhouse
Would you be up to putting together a dedicated website/forum for this kind of discussion?

Synergy actually copies the clipboard across the Linux and Windows hosts for me.

*/etc/synergy.conf*

Code:



Code:


section: screens
linux:
windows:
end

section: aliases
windows:
Josh-PC
linux:
josh-fedora
end

section: links
linux:
right = windows
windows:
left = linux
end

section: options
screenSaverSync = false
relativeMouseMoves = true
# My KVM uses Scroll Lock to switch screens, so set the
# hotkey to lock the cursor to the screen to something else
keystroke(f12) = lockCursorToScreen(toggle)
end

*win7.cfg*
A few notes about this. I'm using LVM to manage the partitioning. I have the LV /dev/passport/cleaninstall as the default installation with the ATI drivers, USB and synergy drivers installed. /dev/passport/win7 is a snapshot of cleaninstall. If windows stops working (which it does for me, every few months) I can delete the /dev/passport/win7 snapshot and create a new snapshot from cleaninstall. No more reinstalling through VNC








/dev/passport/persistent is a secondary drive in the VM, where steam and all my games are reinstalled. Since my internet connections is useless, in the event that I delete the snapshot, I don't have to redownload all the games.

Code:



Code:


kernel = "hvmloader"
builder='hvm'
memory = 4096
name = "win7"
vcpus=2
pae=1
acpi=1
apic=1
vif = [ 'bridge=br0' ]
disk = [ 'phy:/dev/cdrom,hdc:cdrom,r', 'phy:/dev/passport/win7,hda,w', 'phy:/dev/passport/persistent,hdb,w' ]
device_model = 'qemu-dm'
boot="dc"
sdl=0
opengl=1
serial='pty'
tsc_mode=0
soundhw='all'

vnc=1
vncpasswd=''
usb=1
usbdevice='tablet' # fixes the cursor offset in a VNC session

#graphic card only
pci=[ '01:00.0', '01:00.1' ]

# use the command usb-devices to find USB BUS PCI
#pci=[ '01:00.0', '01:00.1', '00:14.0' ]

# usb only
#pci=[ '00:14.0' ]

*Networking*
I'm not using libvirtd ... it doesn't do anything that xm create, shutdown, destroy and xm top can't do. Plus the default NAT network configuration, as you have witnessed, is poor.
I setup a bridge network. First disable the NetworkManager service, since it doesn't support network bridging through the GUI yet. Tutorial to setup the bridge: http://www.virtpanel.com/docs-virtualbox-installation-centos/

*Sound*
I did check out pulseaudio, but it doesn't seem to support windows very well. People who have it working then report problems with a delay in the audio being received. Not good for gaming.
Did you passthrough the PCI of the USB hub, or did it work using the Xen USB emulation for the device?
I would like to avoid passing through the entire USB hub, because my system only has 2 and i'm thinking replacing my HTPC using another VM and GPU on my second PCI-E slot.


----------



## nyxcharon

So, I'm back.









I recently acquired a Nvidia GT 640. I planned on using it on my Host (LM14) to play Steam on Linux games, however I've run into some issues. If I boot into xen with the card and no proprietary drivers, I can get into LM though Cinnamon crashes. If I install the drivers and setup my xorg, I just get a black screen on startup though I lose no video signal (it's just a black screen). I also can't switch to another TTY either.
The card and drivers however do work absolutely fine when not booting with Xen, so I've narrowed down the problem slightly.

Any ideas on what to do?


----------



## powerhouse

@ BulliteShot:

1. Dedicated forum: I'm still in the process of learning about this and there are more qualified people around here. On the other hand, I try to share as much as I can since I know how hard it can be sometimes to find information. Maybe one day I will open a forum.

2. Synergy: Yes, copying worked for me too. Thanks for the synergy config file! I vaguely remember something about an "elevate" option, but couldn't make it work.

3. win.cfg: Thanks for sharing your config file. Did you write the comment on using "usb-devices" to find the PCIs for the USB? It took me a while to figure that out.

You say your Windows crashes every couple of months. Was that prior to using a snapshot? Because I wouldn't be surprised that this happens when you use a snapshot. When doing a snapshot, the original LV is frozen and no changes are applied to it. Instead any changes to the files/folders are written to the snapshot. Depending on how large of a snapshot (snapshot size) you create, it probably will fill up eventually. When it does fill up, LVM automatically discards the snapshot which could crash Windows.

It's an interesting concept to use a snapshot as the regular Windows VM disk. I on the other hand use snapshots to create backups. After major changes to the Windows VM (installation of software, major Windows updates, etc.) I backup the Windows LV (using snapshot). I always keep the last 2 backups, plus a backup of the original installation. The same goes for Linux. Here is my backup script for the Windows VM:

Code:



Code:


#!/bin/sh
mv /media/tmp_stripe/lm13-win7.img /media/tmp_stripe/lm13-win7.img.old
lvcreate --size 10G --snapshot -n win7-snap /dev/lm13/win7
echo Creating backup copy of win7...
pv /dev/mapper/lm13-win7-snap | dd of=/media/tmp_stripe/lm13-win7.img
lvremove /dev/lm13/win7-snap
echo Finished

4. Networking: I too disabled the network manager (actually, I removed it I believe) and created a bridge configuration in the /etc/network/interfaces file. I don't see libvirtd running on my machine. I tried several things but nothing really fixed the issues, although I managed to improve Samba file transfer speed a little bit. I will check the link you posted.

5. Sound: I passed through a USB host (got plenty of them) and never tried a Xen emulated USB device. I suppose you get only 2 lines when running "lspci | grep -i usb". You may want to install a sound PCI card and pass that thru, if you have a spare slot. My gut feeling is that using an emulated USB for sound isn't a good idea. Again, I didn't try it and $4-5 for a USB sound stick is not a lot of money.


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> So, I'm back.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> I recently acquired a Nvidia GT 640. I planned on using it on my Host (LM14) to play Steam on Linux games, however I've run into some issues. If I boot into xen with the card and no proprietary drivers, I can get into LM though Cinnamon crashes. If I install the drivers and setup my xorg, I just get a black screen on startup though I lose no video signal (it's just a black screen). I also can't switch to another TTY either.
> The card and drivers however do work absolutely fine when not booting with Xen, so I've narrowed down the problem slightly.
> 
> Any ideas on what to do?


Sounds familiar to me. I had the same issues with using a Nvidia card with proprietary driver under LM13 Mate. Booting a normal LM kernel worked fine with the Nvidia driver.

Did you install the Nvidia driver while running Xen?

By the way, I didn't have these issues with the proprietary ATI/catalyst driver.


----------



## BulliteShot

@powerhouse
I don't want a USB sound device. I can't find one that doesn't have reports of poor sound quality.
My GPU seems to support Audio over HDMI. I have been searching around for devices which extract the audio from a HDMI cable, but could only find devices in the region of €300+
I found this though, which I am going to try: eBay Item
It converts the HDMI to VGA while extracting the audio for only €15. My monitor supports both VGA and DVI and I personally don't see the difference between the two running at 1920x1080 resolution.

Yes, I found out that the usb-devices lists the PCI device ID of the USB hub through a quick google search. lsusb only seems to list the attached USB devices.

When I say windows stops working, I mean it just breaks. It's little things like a certain game won't start, and the only thing to fix it is a fresh install of windows. My most recent problem with windows was when triple clicking on a word in Chrome wouldn't select the entire paragraph. I could only fix it with a reinstall. (which was what made me switch to linux for my primary OS)


----------



## nbhusain

@ powerhouse
Quote:


> @ nbhusain: Interesting approach. I have some questions.
> 
> 1. Was there a reason for choosing UEFI DUET boot? Faster Windows boot?
> 
> 2. Does UEFI DUET boot only work when directly booting from disk (AHCI)?
> 
> 3. How about disk performance of the passed-thru disk(s) (AHCI)? Is it faster than the regular LVM based disk access, using the GPLPV drivers? N.B.: As far as I have seen, the GPLPV drivers (under Windows 7) greatly affect disk I/O speed.
> 
> 4. You say you boot Xen and afterwards Windows? Though I know that dom0 is nothing but a privileged VM, don't you need to run the dom0 (Linux) to start a domU (Windows)? I'm a bit confused.
> 
> 5. Do you care to share the Windows cfg file?
> 
> Forgive me for asking so many questions, but it's the first time I read about this option. And thanks for sharing this info.


1)

My idea was to create 2 different VM's both capable of gaming/multimedia without affecting eachother under load, so my dom0 is my primary Linux desktop using gnome and 2 usb controllers, and my windows domU is my secondary desktop with 2 other usb controllers, and 1 sata controller, i chose DUET because i wanted to boot windows directly from an ahci disk instead of lvm/file/blktap, i also tried to load the option rom for the sata controller directly from hvmloader but it messes up the boot table on rombios, my setup is like this:

Dom0

2 pinned cpus
3850MB ram
2 usb controller
1 sata controller (asmedia ASM1062 integrated)
RADEON 5450 primary
Internal audio
my grub.cfg file:

Code:



Code:


        multiboot /xen-4.1.4.gz  xsave=1 dom0_max_vcpus=2 dom0_vcpus_pin noselfballooning  dom0_mem=3850M  cpuidle
        module   /vmlinuz-linux root=/dev/mapper/vg0-root ro xen-pciback.permissive xen-pciback.hide=(00:11.0)(00:16.0)(00:16.2)(04:00.0)(05:00.0)(06:00.0)(07:00.0)(07:00.1) 
        module  /initramfs-linux.img

DomU

6 pinned cpus
8GB ram
1 usb3 controller and 1 usb2 controller
1 sata controller (amd sb integrated)
RADEON 6950 secondary (primary on guest OS)
usb audio
my domU cfg:

Code:



Code:


kernel = "hvmloader"
builder='hvm'
memory = 8192
name = "hvm"
cpus = [ "2", "3", "4", "5", "6", "7" ]
vcpus = 6
localtime = 1
vif = [ 'bridge=br0' ]
disk = [ 'file:/root/xen/img/duet.img,xvda,w' ]
pci_msitranslate=0
gfx_passthru=1
pci_power_mgmt=1
viridian=1
acpi_s3=1
acpi_s4=1
hpet=1
xen_platform_pci = 1
pci = [ '0000:07:00.*', '00:11.0', '04:00.0', '05:00.0', '06:00.0', '0000:00:16.*', ]

2) No DUET can boot qemu emulated disks and cdroms too, it even boots my native ahci cdrom connected to the controller, since DUET contains efi drivers, it can also boot windows/linux/etc from usb devices connected to the pass-thru controller.

Since its uefi it might be able to boot a hackintosh build but since im using an amd cpu so i didnt even try.
(if u wanna try you might also want to look at Clover bootloader http://sourceforge.net/projects/cloverefiboot/)

3) Yes, its faster as it completly bypasses the dom0 and gplpv drivers, i do not pass the the real disk to the vm only the duet image.

4) No, windows boots like any other vm except it boots directly from the sata controller instead of the qemu emulated one. Xen creates the vm then it boots DUET from the qemu emulated disk, then DUET loads the ahci drives and finds \EFI\boot\bootx64.efi from the native ahci drive and boots from it.

I hope this answers your questions.

EDIT: Attached are some screenshots of the boot process


----------



## nbhusain

I'd also like to point that seabios (kvm, xen-4.3-unstable) also containts ahci "drivers" so you can boot directly from a native controller without the need of a qemu emulated controller and/or gplpv drivers (and snd-hda-intel for audio passthrough) but unfortunately kvm doesnt allow you to reserve cpus for the "dom0" and xen-4.3-unstable is too buggy at this point. Also since both use the new qemu (1.3.0 atm) none of them support vga pasthrough as a primary device tho you can pass them as secondary. Work is being done to support this using vfio-pci but unfortunately its wip atm and not usable at all
http://lists.nongnu.org/archive/html/qemu-devel/2013-01/msg00959.html


----------



## KeltecRFB

Wow, great guide Dizzy4! It's about time for a better solution than Dual Booting, Wine, or paid version of Wine.

Quick question, has the guide been edited to your most recent version?


----------



## nyxcharon

Quote:


> Originally Posted by *powerhouse*
> 
> Sounds familiar to me. I had the same issues with using a Nvidia card with proprietary driver under LM13 Mate. Booting a normal LM kernel worked fine with the Nvidia driver.
> 
> Did you install the Nvidia driver while running Xen?
> 
> By the way, I didn't have these issues with the proprietary ATI/catalyst driver.


I did indeed install the driver while running Xen. Not sure how to go about troubleshooting this, since all I get is a black screen and I can't find anything in my logs.


----------



## Chetyre

Asrock has confirmed that VT-D will no longer work in their bios 2.70 update for both Z77 Extreme4 and Z77 Extreme6.

So people looking for those two boards and maybe others need to be careful about the bios version shipped.


----------



## powerhouse

Quote:


> Originally Posted by *BulliteShot*
> 
> @powerhouse
> I don't want a USB sound device. I can't find one that doesn't have reports of poor sound quality.
> My GPU seems to support Audio over HDMI. I have been searching around for devices which extract the audio from a HDMI cable, but could only find devices in the region of €300+
> I found this though, which I am going to try: eBay Item
> It converts the HDMI to VGA while extracting the audio for only €15. My monitor supports both VGA and DVI and I personally don't see the difference between the two running at 1920x1080 resolution.
> 
> Yes, I found out that the usb-devices lists the PCI device ID of the USB hub through a quick google search. lsusb only seems to list the attached USB devices.
> 
> When I say windows stops working, I mean it just breaks. It's little things like a certain game won't start, and the only thing to fix it is a fresh install of windows. My most recent problem with windows was when triple clicking on a word in Chrome wouldn't select the entire paragraph. I could only fix it with a reinstall. (which was what made me switch to linux for my primary OS)


Yes, I agree with the USB audio sticks, though I must say mine isn't too bad. For me it's good enough for Windows use, since all my multimedia stuff resides on Linux anyway. (I'm actually one of these old fashioned guys who still listens to music from CD, playing in a real CD player.)

I do see a difference between DVI and VGA, but that's probably because I did this whole VGA passthru setup solely to be able to do photo editing, so I'm looking closely at the picture quality. VGA should be more than satisfactory for most uses.

The only question is whether this HDMI-VGA converter doesn't introduce noise or distortions. It seems worth trying, if it can help improve audio quality.


----------



## powerhouse

@nbhusain: Thanks for your detailed answers. This is most helpful. If I hadn't got already everything up and running, I would try your method.

Do you think DUET boot would work with a regular installation of Windows 7 on disk? I mean, can you dual boot your PC with Windows 7? I guess that would be asking too much.


----------



## nbhusain

@ powerhouse

Yea i dual boot my pc, linux and windows in UEFI mode, i have 2 disk each connected to a different sata controller, one of them contains my linux install, the other a regular UEFI install of windows 7, i use DUET on xen to boot the windows UEFI install directly from my secondary sata controller.

If you have a regular UEFI windows 7 installation on disk you can boot it with xen and duet. If your windows 7 install is MBR then no, but its possible to convert it to UEFI.


----------



## KeltecRFB

For newbies, here are a couple of ways to validate virtualization is truly enabled.

Linux = www.cyberciti.biz/faq/linux-xen-vmware-kvm-intel-vt-amd-v-support/

Windoze = http://www.microsoft.com/en-us/download/details.aspx?id=592


----------



## powerhouse

Quote:


> Originally Posted by *nyxcharon*
> 
> I did indeed install the driver while running Xen. Not sure how to go about troubleshooting this, since all I get is a black screen and I can't find anything in my logs.


Unfortunately I can't try on my own system now, since I use my Nvidia card for domU exclusively. Here is what I found, in the hope it gives you some ideas:

An old post but a good description of the process, albeit for Gentoo: http://legroom.net/2008/06/22/running-binary-nvidia-drivers-under-xen-host
For CentOS: http://wiki.centos.org/HowTos/Xen/NvidiaWithXen
For OpenSuse: http://www.nvnews.net/vbulletin/showpost.php?p=2435805&postcount=114
And finally Ubuntu: http://nathanosullivan.com/blog/computing/using-ubuntu-904-as-a-xen-dom0-host-with-nvidia-driver/

This may be helpful too: http://www.nvnews.net/vbulletin/showthread.php?t=122900&highlight=xen&page=2

I would try the Ubuntu version, perhaps cross-check with the CentOS wiki. What bothers me is that all those how-tos are for rather old kernels. See here: http://www.nvnews.net/vbulletin/showthread.php?t=174219

So, the above solutions may work with pre-3.x kernels, but I haven't seen a single success report for newer kernels/hypervisors. See http://wiki.xen.org/wiki/Xen_Kernel_Feature_Matrix for the difference between older and newer kernels/hypervisor.


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> @ powerhouse
> 
> Yea i dual boot my pc, linux and windows in UEFI mode, i have 2 disk each connected to a different sata controller, one of them contains my linux install, the other a regular UEFI install of windows 7, i use DUET on xen to boot the windows UEFI install directly from my secondary sata controller.
> 
> If you have a regular UEFI windows 7 installation on disk you can boot it with xen and duet. If your windows 7 install is MBR then no, but its possible to convert it to UEFI.


That's great news! I believe that makes things a lot easier for many dual-boot users. A friend of mine recently got compatible hardware and I will ask him to let me give this a try.
Quote:


> Originally Posted by *Chetyre*
> 
> Asrock has confirmed that VT-D will no longer work in their bios 2.70 update for both Z77 Extreme4 and Z77 Extreme6.
> 
> So people looking for those two boards and maybe others need to be careful about the bios version shipped.


Thanks for checking. I updated http://wiki.xen.org/wiki/VTd_HowTo.


----------



## nbhusain

@ powerhouse

Good luck, remember when installing windows, make sure the hard drive you're installing to is the only one connected, otherwise windows might create the efi partition it needs to boot on another disk or use a previous one it finds, this happened to me, windows found my linux efi partition and copied the boot files there. After that connect all your disks again.


----------



## Rezz

Quote:


> Originally Posted by *Chetyre*
> 
> Asrock has confirmed that VT-D will no longer work in their bios 2.70 update for both Z77 Extreme4 and Z77 Extreme6.
> 
> So people looking for those two boards and maybe others need to be careful about the bios version shipped.


I guess we can then safely say that this will also be the case for the Z77E-ITX with Bios updates 1.60 and 1.70......?

*1.40 (currently) being the last known Bios version to support VT-d for the Z77E-ITX.*

Hopefully Asrock will re-enable VT-d support for all their Z77 boards in future.

In the Z77 ITX form factor, it was one thing that gave them a big advantage over other OEM's.


----------



## Chetyre

I'm going to give this another try and compile 4.2 from source with this patch http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 but it needs my vga bios.

I'm stumped on how to extract that though. I see a lot of discussion lists mentioning nvtools, but I just can't find anything about what it is or how to use it. I found this site http://cgit.freedesktop.org/~lb/nvtools/ but it fails to compile.

Any help?


----------



## nbhusain

Quote:


> I'm going to give this another try and compile 4.2 from source with this patch http://www.davidgis.fr/download/xen-4.2_rev25240_gfx-passthrough-patchs.tar.bz2 but it needs my vga bios.
> 
> I'm stumped on how to extract that though. I see a lot of discussion lists mentioning nvtools, but I just can't find anything about what it is or how to use it. I found this site http://cgit.freedesktop.org/~lb/nvtools/ but it fails to compile.
> 
> Any help?


You can download ir from here http://www.techpowerup.com/vgabios/


----------



## dizzy4

Quote:


> Originally Posted by *Chetyre*
> 
> Asrock has confirmed that VT-D will no longer work in their bios 2.70 update for both Z77 Extreme4 and Z77 Extreme6.
> 
> So people looking for those two boards and maybe others need to be careful about the bios version shipped.


What was it that they said exactly? I am curious because I have heard differing reports.


----------



## Chetyre

Quote:


> Originally Posted by *nbhusain*
> 
> You can download ir from here http://www.techpowerup.com/vgabios/


Thanks, I'll give it a try. But still how would I extract it myself if ever needed?

This is my card http://www.techpowerup.com/vgabios/index.php?page=1&architecture=NVIDIA&manufacturer=XFX&model=GTS+250&interface=PCI-E&memSize=512

as you can see there are 5 different bioses... I don't know how to find out which one is mine
Quote:


> Originally Posted by *dizzy4*
> 
> What was it that they said exactly? I am curious because I have heard differing reports.


This is directly from my email:
Quote:


> Originally Posted by *Asrock*
> 
> -My Name-;
> 
> After you update the BIOS with the 2.70 the VT-D not longer works
> 
> ASRock America Support
> 
> BTW , you do not need to update the BIOS if you do not have problems
> 
> ASRock America Support


Their English is pretty terrible. I asked why they removed support too, but they never answered that.


----------



## nbhusain

Get gpu-z and check your bios version there, you could also try to match the default clocks to the bios posted there, to extract it just unzip it, it should contain a .bin or .rom file, rename that to vgabios-pt.bin and put it on /tools/firmware/vgabios/vgabios-pt.bin


----------



## Chetyre

I managed to do it with nvflash. I compiled an older version for the yet-unstable 4.2 version because the patches are all very old. they date to almost a year ago.

Anyway, it compiled, but I couldn't get pciback to work, so still no deal...


----------



## nbhusain

I sugest you use xen-4.1.4 with xend, im using radeon cards atm but i managed to passthrough a nvidia card (9800gt) not so long ago with xen 4.1


----------



## nbhusain

BTW what distro are you using?


----------



## Chetyre

CentOS 6.3


----------



## nbhusain

If pciback is compiled as a module, you have to include it on the intramfs, if its not even built as module, you might have to rebuild your kernel, anyway, i recomend archlinux, its quite easy to create custom made packages for it.


----------



## Chetyre

I have arch for everyday use on my laptop, but for my dom0 I was aiming for a more stable distro. It was either CentOS or Debian


----------



## Bobicon

Any reason as to why you recommend Fedora instead of a distro with a smaller foot print like Debian?

Because if it were me I want all the extra preformance I could get if I were gaming on a VM.


----------



## nbhusain

Yeah but stable also means outdated you might have to rebuild your kernel anyway


----------



## nbhusain

@ Bobicon
It doesnt matter which distro you use, dom0 is just a privileged VM in the end, the cpu's you see are actually threads on the xen hypervisor as far as i understand, the only advantage as you point out might be memory footprint, but you can limit dom0 max memory anyway.


----------



## nbhusain

I'd like to make a correction, the windows installer under DUET on EFI mode only crashes if 'xen_platform_pci = 1' y changed it to 0 and it worked







, after that i reverted it to 1 and windows is happy with it, so you dont need a uefi mobo to install and run windows on uefi mode after all.


----------



## powerhouse

@nbhusain: Thanks for the update - this sounds more and more intriguing.

@Bobicon: I think this is both a matter of preference and available hardware. If you have enough RAM and a modern 4+ core CPU, the resources that the dom0 uses are negligible, regardless of the distro. Memory is cheap so giving the dom0 2GB (or more) of RAM shouldn't cost too much.

For I/O performance it is advisable to give dom0 enough memory and CPU (vcpu) to make sure it doesn't turn into a bottleneck.

It makes perhaps sense to have a small "console only" dom0 and run both the Linux DE and Windows in VMs, if only for security reasons. But in that case you would probably need to passthru 2 VGAs to get good graphics performance for Linux and Windows.


----------



## nbhusain

Quote:


> It makes perhaps sense to have a small "console only" dom0 and run both the Linux DE and Windows in VMs, if only for security reasons. But in that case you would probably need to passthru 2 VGAs to get good graphics performance for Linux and Windows.


Yup, done that under gentoo, the downside is you will always have to allocate at least 1 cpu to dom0 and if something goes wrong, you're on blind mode, thats why im doing this experiment with duet, i always have access to my dom0 and even under heavy load it doesnt affect windows at all.


----------



## Chetyre

Quote:


> Originally Posted by *nbhusain*
> 
> Yeah but stable also means outdated you might have to rebuild your kernel anyway


I have a 3.7 kernel


----------



## nbhusain

Did you build pciback as a module?


----------



## Chetyre

nope, build-in.

In anycase when I have time again I'll try building 4.1 and see what happens


----------



## nbhusain

check if this folder exists /sys/bus/pci/drivers/pciback/


----------



## nbhusain

Hi, i made more progress using duet, i modified the source to boot always from the hdd (duet doesnt keep efi variables) and installed rEFInd on the image, this allows me select the media i want to boot from a pretty menu, here are some screenshots i took, i had to disable gpu passthru ( gfx_passthru=0 ) to take the shots, but it works just fine when its enabled, usb keyboard and mouse work too

Here's duet recognizing my usb pendrive with windows 7 installer:



Booting windows 7 installer from the pendrive (xen_platform_pci = 0):



Duet recognizing my sata dvd drive:



Booting Fedora 17 beta installer from the sata drive:



Fedora 17 on UEFI mode, on XEN.



My cfg file:

Code:



Code:


kernel = "hvmloader"
builder='hvm'
memory = 8192
boot = 'c'
name = "hvm"
cpus = [ "2", "3", "4", "5", "6", "7" ]
vcpus = 6
localtime = 1
vif = [ 'bridge=br0' ]
disk = [ 'file:/root/xen/img/duet.img,hda,w' ]
pci_msitranslate=0
gfx_passthru=1
pci_power_mgmt=1
viridian=1
acpi_s3=1
acpi_s4=1
hpet=1
xen_platform_pci = 1
pci = [ '0000:07:00.*', '00:11.0', '04:00.0', '05:00.0', '06:00.0', '0000:00:12.*', '0000:00:16.*' ]

EDIT: Sucess!, i managed to dump the AMD RAID UEFI driver from memory, it loads just fine on DUET and i can boot windows from the controller on raid mode on xen!, here's
the .efi file for anyone interested.

RaidX64.efi.zip 79k .zip file


duet.img.zip 3410k .zip file


----------



## WorldExclusive

I just bought a i7 3770.

What are the basic features and benefits of VM?

I tried reading some articles online, but most of them lost me with their explanation.


----------



## Chetyre

Quote:


> Originally Posted by *WorldExclusive*
> 
> I just bought a i7 3770.
> 
> What are the basic features and benefits of VM?
> 
> I tried reading some articles online, but most of them lost me with their explanation.


The most important benefit in our case, in my opinion, is that it eliminates the need for dual booting for those of us who wish to use Linux as their primary system, but still want to game.

But there are many other benefits, such as making it easy to backup entire systems. It also eases the development of new programs since you can test it on a number of OSs with very little trouble, and another benefit is that services can be separated from one another, minimizing possible conflicts since one service can't interfere with the other.

There are other benefits too, such as testing beta software without compromising your main system, and using it as a sandbox against viruses and other malware (windows mostly)


----------



## powerhouse

I had mentioned before that my local network file transfer speeds were sub-optimal. I've tried many different configurations, and finally it seems to work. I've written down the steps (to the best of my knowledge) here:
http://forums.linuxmint.com/viewtopic.php?f=42&t=112013&p=679098#p679098

From 4.5Mbyte file transfer speed via a Gigabit link I'm now up to around 25-26MByte. The Fast Ethernet link to my media streamer shows also a nice improvement. Hope this helps anyone.


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> Hi, i made more progress using duet, i modified the source to boot always from the hdd (duet doesnt keep efi variables) and installed rEFInd on the image, this allows me select the media i want to boot from a pretty menu, here are some screenshots i took, i had to disable gpu passthru ( gfx_passthru=0 ) to take the shots, but it works just fine when its enabled, usb keyboard and mouse work too
> 
> Here's duet recognizing my usb pendrive with windows 7 installer:
> 
> 
> 
> Booting windows 7 installer from the pendrive (xen_platform_pci = 0):
> 
> 
> 
> Duet recognizing my sata dvd drive:
> 
> 
> 
> Booting Fedora 17 beta installer from the sata drive:
> 
> 
> 
> Fedora 17 on UEFI mode, on XEN.
> 
> 
> 
> My cfg file:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> kernel = "hvmloader"
> builder='hvm'
> memory = 8192
> boot = 'c'
> name = "hvm"
> cpus = [ "2", "3", "4", "5", "6", "7" ]
> vcpus = 6
> localtime = 1
> vif = [ 'bridge=br0' ]
> disk = [ 'file:/root/xen/img/duet.img,hda,w' ]
> pci_msitranslate=0
> gfx_passthru=1
> pci_power_mgmt=1
> viridian=1
> acpi_s3=1
> acpi_s4=1
> hpet=1
> xen_platform_pci = 1
> pci = [ '0000:07:00.*', '00:11.0', '04:00.0', '05:00.0', '06:00.0', '0000:00:12.*', '0000:00:16.*' ]
> 
> EDIT: Sucess!, i managed to dump the AMD RAID UEFI driver from memory, it loads just fine on DUET and i can boot windows from the controller on raid mode on xen!, here's
> the .efi file for anyone interested.
> 
> RaidX64.efi.zip 79k .zip file
> 
> 
> duet.img.zip 3410k .zip file


That looks very nice! Not sure I asked this question before, but do you see a difference in performance of Windows under Xen versus normal boot? Such as disk / file transfer speed, graphics performance, etc.


----------



## nbhusain

@ Powerhouse

If by normal boot you mean passing the raw disk (ex: disk = [ phy:/dev/sdb ] ) yes its a lot faster than using an emulated controller with gplpv, like i already mentioned, it bypasses the dom0, since the controller ( in my case its device 00:11.0 amd sb controller) is managed by the domU, and its using native drivers on windows.

As for native boot vs xen boot, no, i dont see any difference at all except in the cpu performance, as i have 2 cores pinned to dom0, and 6 cores pinned to domU, this setup is as close to metal as possible with the exception of the net device, i guess you could passthrough a net device and it will be almost 100% native performance.


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> @ Powerhouse
> 
> If by normal boot you mean passing the raw disk (ex: disk = [ phy:/dev/sdb ] ) yes its a lot faster than using an emulated controller with gplpv, like i already mentioned, it bypasses the dom0, since the controller ( in my case its device 00:11.0 amd sb controller) is managed by the domU, and its using native drivers on windows.
> 
> As for native boot vs xen boot, no, i dont see any difference at all except in the cpu performance, as i have 2 cores pinned to dom0, and 6 cores pinned to domU, this setup is as close to metal as possible with the exception of the net device, i guess you could passthrough a net device and it will be almost 100% native performance.


Actually I meant booting directly into Windows versus Windows VM on Xen. The way I understood you is that you have Windows installed on disk and can dual boot into Windows, as well as boot into Linux/Xen and run Windows in a VM. Or have I completely lost you?

My Windows is installed on LVM, that is disk = [ 'phy:/dev/mapper/lm13-win7,hda,w'...], same as yours except LVM. According to the Xen wiki and other documentation, LVM is supposed to offer the best performance. Well, I can't complain, though I've nothing to compare with since all my disks/partitions are LVM, except /boot of course. Some disks are LVM striped for RAID0 performance, which works very well.


----------



## nbhusain

Maybe it was confusing so ill try to explain again, i have 2 sata controllers integrated on my mobo, one is the amd sb one, and the other one is an asm1062. My first disk is connected to the asmedia controller, that disk contains my linux installation with xen, the second disk i have contains my windows installation, and its connected to the amd controller along with a sata dvd drive.

Code:



Code:


kernel = "hvmloader"
builder='hvm'
memory = 8192
boot = 'c'
name = "hvm"
cpus = [ "2", "3", "4", "5", "6", "7" ]
vcpus = 6
localtime = 1
vif = [ 'bridge=br0' ]
disk = [ 'file:/root/xen/img/duet.img,hda,w' ]
pci_msitranslate=0
gfx_passthru=1
pci_power_mgmt=1
viridian=1
acpi_s3=1
acpi_s4=1
hpet=1
xen_platform_pci = 1
pci = [ '0000:07:00.*', '00:11.0', '04:00.0', '05:00.0', '06:00.0', '0000:00:12.*', '0000:00:16.*' ]

07:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman PRO [Radeon HD 6950]
07:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
00:16.0 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 USB EHCI Controller

As u can see from my cfg file,what i do is, i pass the duet image (disk = [ 'file:/root/xen/img/duet.img,hda,w' ]) as a disk, this disk DOES NOT contain my windows installation, it just contains duet and refind which is an efi bootloader, i do not pass my real disk to xen, what i do is, i pass the entire controller (00:11.0 which like i said contains a REAL disk with windows installed + a sata dvd drive). Duet then picks up the controller, finds my disk which contains windows and the dvd drive and boots from it. My setup is just one disk im not using lvm or raid, but like i mentioned i managed to dump from memory the driver for my controller when i switched it to raid mode, and it works on duet, it finds the real disk and i can boot from it, tho i did not try to create a raid0 setup with it because i only got 1 disk. As for performance this method should be NATIVE fast since the entire controller is managed by the domU, i hope this answers your questions







.

And yes with this setup you can dual boot.


----------



## powerhouse

@ nbhusain: Thanks for the detailed explanation. I think I got it now







.

I already got my Windows VM running on an LVM volume and see no reason to change that. But your method looks like a good way to convert a Windows PC or dual boot PC into a Xen box with Windows running on a VM. Hope to be able to try that on a friend's PC (he's got the right hardware for it). Need to do some reading on duet and how to read the SATA controller ROM.

I'm a little familiar with refind and used it before, until I discovered that Xen doesn't support UEFI (until recently - kernel 3.7 is supposed to solve that). Rod the developer/maintainer is a great guy, always there to help.


----------



## nbhusain

You dont need the rom for the sata controller as long as it is on ahci mode and even then, you would need an efi driver, one easy way to check if this works for you is using the duet image i provided and passing your sata controller to the vm, if it picks up your disk your good to go.
Quote:


> I'm a little familiar with refind and used it before, until I discovered that Xen doesn't support UEFI (until recently - kernel 3.7 is supposed to solve that). Rod the developer/maintainer is a great guy, always there to help.


Xen does actually support uefi, im using xen-4.1.4 and kernel-3.6.11 on arch and i can boot just fine, or you could just boot linux in legacy mode and with duet you'll still be able to boot windows on xen on uefi mode with duet.


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> You dont need the rom for the sata controller as long as it is on ahci mode and even then, you would need an efi driver, one easy way to check if this works for you is using the duet image i provided and passing your sata controller to the vm, if it picks up your disk your good to go.
> Xen does actually support uefi, im using xen-4.1.4 and kernel-3.6.11 on arch and i can boot just fine, or you could just boot linux in legacy mode and with duet you'll still be able to boot windows on xen on uefi mode with duet.


Good to know about the AHCI mode and not needing the ROM.

About Xen supporting UEFI, that's new to me. Actually, I believe I read some release note to the 3.7x kernel that mentioned UEFI support for Xen. I had tried it with a 3.2 kernel and Xen 4.1.2. I recently installed Linux Mint 14 which comes with a 3.5 kernel and Xen 4.1.3, but didn't try UEFI. Is there any advantage to UEFI, except perhaps a faster boot (I'm not sure on that)?


----------



## nbhusain

Quote:


> Is there any advantage to UEFI, except perhaps a faster boot (I'm not sure on that)?


I guess one advantage is booting gpt disks but i think you can do that with hybrid mbr, as for boot time, for me is about the same.


----------



## przemoc

Is here any ASRock Z77 Extreme4 user with BIOS version >= 2.70 or ASRock Z77 Extreme6 user with BIOS version >= 2.40? If yes, could you confirm whether VT-d works for you or not? Could you also state whether you bought your mobo with such BIOS version or whether you upgraded it yourself?

The info on the BIOS change version:
Quote:


> Modify VT-d Capability will be unsupported after flashing BIOS P2.XX


could be also read as "turning off possibility to change its enable state". After all these sites are known for bad English. (Well, there are naming mistakes in BIOSes too, like I showed once in case of Abit KN9, thus double or triple checking anything is almost always required.)

Also front-line support people are often incompetent, that's why I doubt credibility of ASRock America Support response to Chetyre quoted here.

I'm considering buying one of these two mobos, that's why I'm trying to confirm one way or the other.


----------



## powerhouse

Quote:


> Originally Posted by *przemoc*
> 
> Is here any ASRock Z77 Extreme4 user with BIOS version >= 2.70 or ASRock Z77 Extreme6 user with BIOS version >= 2.40? If yes, could you confirm whether VT-d works for you or not? Could you also state whether you bought your mobo with such BIOS version or whether you upgraded it yourself?
> 
> The info on the BIOS change version:
> could be also read as "turning off possibility to change its enable state". After all these sites are known for bad English. (Well, there are naming mistakes in BIOSes too, like I showed once in case of Abit KN9, thus double or triple checking anything is almost always required.)
> 
> Also front-line support people are often incompetent, that's why I doubt credibility of ASRock America Support response to Chetyre quoted here.
> 
> I'm considering buying one of these two mobos, that's why I'm trying to confirm one way or the other.


See http://www.overclock.net/t/1338063/vt-d-compatible-motherboards, specifically post #20. It seems a Japanese guy made it work with the latest BIOS. My suggestion is to contact Asrock technical support to find out. Please post your findings.


----------



## Chetyre

nbhusain, could you help me compile xen on archlinux? the pkgbuild on aur fails for me for some reason on the stubdom.

I'm also trying to mess with uefi for now. I just managed to make it boot with rEFInd, but how would I boot the xen kernel? I googled around and found that xen has a xen.efi, but how do I pass the kernel parameters?

sorry, uefi is very new to me.

EDIT: managed to install it. one of the dependencies was broken. now I need to boot it...


----------



## nbhusain

Quote:


> nbhusain, could you help me compile xen on archlinux? the pkgbuild on aur fails for me for some reason on the stubdom.


Sure, here's my xen-4.1.4 pkg based on xen-4.2 AUR pkg, it contains patches for AMD vga passthrough, if you have an AMD card and you wish to enable these patches, you have to copy your bios file to vgabios-pt.bin on the root folder (replace mine), otherwise you can disable it by commenting out these lines:

patch -Np1 -i ../xen-4.1.3.ati-passthrough.patch
patch -Np1 -i ../loadbios.patch
cp ../../vgabios-pt.bin tools/firmware/vgabios/vgabios-pt.bin

xen.zip 52k .zip file

Quote:


> I'm also trying to mess with uefi for now. I just managed to make it boot with rEFInd, but how would I boot the xen kernel? I googled around and found that xen has a xen.efi, but how do I pass the kernel parameters?


Assuming you already installed arch on uefi mode, you can boot xen from grub by creating a grub cfg file on /etc/grub.d/

Here's mine:

Code:



Code:


cat /etc/grub.d/05_xen

Code:



Code:


#!/bin/sh                                                                                                                                                                                                                                                                      
exec tail -n +3 $0

menuentry 'XEN' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-core repo kernel-true-89ae0638-a44f-45be-b983-d65ac23e13b4' {                                                                                                             
        load_video                                                                                                                                                                                                                                                             
        set gfxpayload=keep                                                                                                                                                                                                                                                    
        insmod gzio                                                                                                                                                                                                                                                            
        insmod part_gpt                                                                                                                                                                                                                                                        
        insmod ext2                                                                                                                                                                                                                                                            
        set root='hd1,gpt2'                                                                                                                                                                                                                                                    
        if [ x$feature_platform_search_hint = xy ]; then                                                                                                                                                                                                                       
          search --no-floppy --fs-uuid --set=root --hint-bios=hd1,gpt2 --hint-efi=hd1,gpt2 --hint-baremetal=ahci1,gpt2  8f5355af-14eb-4f86-930a-de7ece6a8114                                                                                                                   
        else                                                                                                                                                                                                                                                                   
          search --no-floppy --fs-uuid --set=root 8f5355af-14eb-4f86-930a-de7ece6a8114                                                                                                                                                                                         
        fi                                                                                                                                                                                                                                                                     
        echo    'Loadin XEN'                                                                                                                                                                                                                                                   
        multiboot /xen-4.1.4.gz xsave=1 dom0_max_vcpus=2 dom0_vcpus_pin noselfballooning  dom0_mem=3850M  cpuidle                                                                                                                        
        echo    'Loading Linux core repo kernel ...'                                                                                                                                                                                                                           
        module   /vmlinuz-linux root=/dev/mapper/vg0-root ro xen-pciback.permissive xen-pciback.hide=(00:11.0)(00:12.0)(00:12.2)(00:16.0)(00:16.2)(04:00.0)(05:00.0)(06:00.0)(07:00.0)(07:00.1) nomce                                                                          
        echo    'Loading initial ramdisk ...'                                                                                                                                                                                                                                  
        module  /initramfs-linux.img                                                                                                                                                                                                                                           
}

Booting XEN on uefi at this point is kind of pointless, as xen provides its own memory map for dom0, and you cant access efivars from it.

EDIT:

This is how i got xen-4.2.1 efi to boot from grub2:

Code:



Code:


cat /etc/grub.d/04_xen

Code:



Code:


menuentry 'XEN' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-core repo kernel-true-89ae0638-a44f-45be-b983-d65ac23e13b4' {
        load_video
        set gfxpayload=keep
        insmod gzio
        insmod part_gpt
        insmod ext2
        set root='hd1,gpt2'
        if [ x$feature_platform_search_hint = xy ]; then
          search --no-floppy --fs-uuid --set=root --hint-bios=hd1,gpt2 --hint-efi=hd1,gpt2 --hint-baremetal=ahci1,gpt2  8f5355af-14eb-4f86-930a-de7ece6a8114
        else
          search --no-floppy --fs-uuid --set=root 8f5355af-14eb-4f86-930a-de7ece6a8114
        fi
        echo    'Loadin XEN'
        chainloader /xen-4.2.1.efi -cfg=xen.cfg

}

Code:



Code:


cat /boot/xen.cfg

Code:



Code:


[global]
default=arch

[arch]
options=loglvl=all xsave=1 dom0_max_vcpus=2 dom0_vcpus_pin noselfballooning  dom0_mem=3850M  cpuidle
kernel=vmlinuz-linux root=/dev/mapper/vg0-root ro xen-pciback.permissive xen-pciback.hide=(00:11.0)(00:12.0)(00:12.2)(00:16.0)(00:16.2)(04:00.0)(05:00.0)(06:00.0)(07:00.0)(07:00.1) nomce
ramdisk=initramfs-linux.img

EDIT2: Quick update, you need to build binutils with "--enable-targets=x86_64-pep" for xen.efi to build, here's an edited binutils-multilib pkg:

binutils.zip 2k .zip file


----------



## Chetyre

Quote:


> Originally Posted by *nbhusain*
> 
> EDIT2: Quick update, you need to build binutils with "--enable-targets=x86_64-pep" for xen.efi to build, here's an edited binutils-multilib pkg:
> 
> binutils.zip 2k .zip file


Build fails when I use the modified binutils. nevermind my shell.

Code:



Code:


install -d -m0755 -p "/tmp/yaourt-tmp-arfoire/aur-xen/pkg/usr/lib/xen/boot"
install -m0644 -p mini-os-x86_32-grub/mini-os.gz "/tmp/yaourt-tmp-arfoire/aur-xen/pkg/usr/lib/xen/boot/pv-grub-x86_32.gz"
make[1]: Leaving directory `/tmp/yaourt-tmp-arfoire/aur-xen/src/xen-4.2.1/stubdom'
mv: cannot stat ���etc/init.d���: No such file or directory
==> ERROR: A failure occurred in package().
    Aborting...
==> ERROR: Makepkg was unable to build xen.
==> Restart building xen ? [y/N]
==> ----------------------------
==>

I'm using yaourt, but it does the same thing with makepkg. This is the package https://aur.archlinux.org/packages/xen/


----------



## nbhusain

Try this quick fix:

Code:



Code:


...
    if [[ -d usr/lib64 ]]; then
        cd usr/
        cp -r lib64/* lib/
        rm -rf lib64
    fi
...

change it to:

Code:



Code:


...
    if [[ -d usr/lib64 ]]; then
        cd usr/
        cp -r lib64/* lib/
        rm -rf lib64
    fi

   cd "$pkgdir"
...

Why are you building stubdom anyway? you dont need it at all, specially if your gonna use gplpv drivers.


----------



## keroberos7

Quote:


> Originally Posted by *przemoc*
> 
> Is here any ASRock Z77 Extreme4 user with BIOS version >= 2.70 or ASRock Z77 Extreme6 user with BIOS version >= 2.40? If yes, could you confirm whether VT-d works for you or not? Could you also state whether you bought your mobo with such BIOS version or whether you upgraded it yourself?


VT-d is working on ASRock Z77 Extreme6 with BIOS P2.40.
Check out my blog.
http://mo.kerosoft.com/0198


----------



## Chetyre

Quote:


> Originally Posted by *nbhusain*
> 
> Try this quick fix:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> ...
> if [[ -d usr/lib64 ]]; then
> cd usr/
> cp -r lib64/* lib/
> rm -rf lib64
> fi
> ...
> 
> change it to:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> ...
> if [[ -d usr/lib64 ]]; then
> cd usr/
> cp -r lib64/* lib/
> rm -rf lib64
> fi
> 
> cd "$pkgdir"
> ...
> 
> Why are you building stubdom anyway? you dont need it at all, specially if your gonna use gplpv drivers.


Thanks, I'll try it as soon as I can. I'm building it because it's part of the package. And I might need it someday, not everything is going to use gplpv drivers. I'll also have some pv guests

EDIT: It worked. I was just going to edit the pkg with the absolute path to the directory, but using the variable is a lot easier indeed.

Wow, that went a lot smother than I thought it would. I'm using REFInd, not grub, so I wasn't sure where I would put the config file, but it worked beautifully on my first try. Thanks.


----------



## powerhouse

Quote:


> Originally Posted by *keroberos7*
> 
> VT-d is working on ASRock Z77 Extreme6 with BIOS P2.40.
> Check out my blog.
> http://mo.kerosoft.com/0198


Thanks for sharing your information here! This is very useful.


----------



## Chetyre

nbhusain,

I used your DUET image to install windows 8 from the sata controller, but I'm unable to modify the bootloader entry to change its order. I tried doing it from the shell, but it didn't work. How do you do it? I want it to start automatically

EDIT: Nevermind. I disabled the efis in the tools folder and now the only entries are the windows ones.


----------



## nbhusain

Quote:


> Originally Posted by *Chetyre*
> 
> nbhusain,
> 
> I used your DUET image to install windows 8 from the sata controller, but I'm unable to modify the bootloader entry to change its order. I tried doing it from the shell, but it didn't work. How do you do it? I want it to start automatically
> 
> EDIT: Nevermind. I disabled the efis in the tools folder and now the only entries are the windows ones.


You can find refind.conf inside the image i created, and you can edit it to only load manual entries:

Change this

Code:



Code:


#scanfor manual,external,optical
scanfor internal,external,optical

to this

Code:



Code:


scanfor manual,external,optical
#scanfor internal,external,optical

Then you need to create an entry for windows, i made a sample on refind.conf, for example for my windows 7 install i use:

Code:



Code:


menuentry "Windows 7" {
       icon /EFI/boot/refind_gris/os_win.icns
       volume WINEFI
       loader \EFI\boot\bootx64.efi
       enabled
}

You might need to give your windows ESP a label in my case "WINEFI". The ESP is the first windows fat32 partition.

You might also wanna change the timeout from 20 to 1


----------



## powerhouse

I had installed the GPLPV drivers release 0.11.0.356 under Windows 7 (domU) and while they greatly improved disk I/O performance, there was an issue with Youtube video as well as viewing a video file that resides on a network drive. I wasn't able to fix this, no matter what I tried (short of removing the GPLPV drivers).

Today I downloaded and installed the latest GPLPV drivers from http://www.meadowcourt.org/downloads/. As of this writing the latest drivers are 0.11.0.357.

Now everything works as expected.

Hope this helps.


----------



## Rezz

Quote:


> Originally Posted by *powerhouse*
> 
> Today I downloaded and installed the latest GPLPV drivers from http://www.meadowcourt.org/downloads/. As of this writing the latest drivers are 0.11.0.357.


There are later GPLPV drivers that might be worth trying as well:

0.11.0.369 :

http://www.meadowcourt.org/private/gplpv_Vista2008x64_0.11.0.369.msi

and 0.11.0.372:

http://xen.1045712.n5.nabble.com/Problem-using-PoD-on-Xen-4-2-1-win-domU-crash-tp5714137p5714143.html


----------



## powerhouse

Quote:


> Originally Posted by *Rezz*
> 
> There are later GPLPV drivers that might be worth trying as well:
> 
> 0.11.0.369 :
> 
> http://www.meadowcourt.org/private/gplpv_Vista2008x64_0.11.0.369.msi
> 
> and 0.11.0.372:
> 
> http://xen.1045712.n5.nabble.com/Problem-using-PoD-on-Xen-4-2-1-win-domU-crash-tp5714137p5714143.html


Thanks for the information. I just visited http://www.meadowcourt.org/downloads/ and the most recent 0.11.0.372 drivers are listed. Will give them a try.

EDIT: Installed the 0.11.0.372 release of GPLPV drivers and they seem to work fine.


----------



## plmnb

There is one fundamental question by me... Is there possibility to make some VMs (for example 2 VMs on one "host" pc) and use full potential of this "host" + run it in lan.
example:
one host pc running 2vms ,and two laptops using each one vm ,playing games in two different places in home (of course with enough 1gbit lan bandwidth).


----------



## lloyd mcclendon

I finally got a VT-D compatible motherboard and got back to this. I do have PCI & USB passthrough working fine using KVM. However, VGA passthrough is a little more complicated. These NVIDIA cards do not support it, since they lack FLR (function level reset) capabilities. I have the card showing up in the guest, but the nvidia drivers on the other side aren't able to use the card as it has to be FLR'd first.







I wish I would have known this 6 months ago. Only the nvidia professional type cards will do this, and it's spotty at best.

So it looks like I may have to ditch these cards and switch to ATI







. I may then find that KVM as a hypervisor is still incapable of VGA passthrough and have to jump to XEN at that point which isn't a big deal. Ideal setup to me would look like:

[ onboard vga ] -- drive the physical host machine's console, for the most part not really used.

[ card 1 ] -- some sort of ATI, lower end, maybe a gen or two old but has to support VT-D & FLR. Sent to a linux VM used for normal desktop purposes.

[ card 2 ] -- again some sort of ATI, higher end, also has to support VTD & FLR. Sent to windows VM for gaming. Just use the switch on the monitor or kvm switch.
[ card 3 ] -- if card 2 isn't powerful enough, crossfired with card 2

What are some good candidates for card 1 and card 2? I'm really not in the ATI space at all, no idea what's out there.

Card 1 should be lower power consumption, but still good enough to run youtube etc - nothing over the top but still capable with dual monitors.

Card 2 should be about top of the line, ideally powerful enough that card 3 wouldn't be necessary to get good FPS. Both should support passthrough, and I'd like to water cool them with some sort of nice bridge setup. I hope there are some water blocks out there for two different types of cards where the fittings line up so a bridge setup is supported, or I may have to run an overkill card in slot 1.

Any thoughts, what cards would you recommend to passthrough if you were starting from scratch?


----------



## Chetyre

Today I discovered that having a nvidia card in my system was breaking a lot more things than it just not working. I wasn't able to auto-start VMs in /etc/xen/auto for example, but as soon as I removed my nvidia card (because honestly it wasn't doing anything anyway) everything worked perfectly and correctly.

Another thing that fixed itself is I have finally managed to passthrough the 3770's graphics card (HD4000)(At least apparently). It never worked before, but when I tried removing my nvidia card from the system it started working, even with gfx_passthrough set to ! So I can see the boot screen corretly.

I'm going to give it to windows because, honestly, I'm almost never at the console (I do pretty much everything remotely all the time), and because my wife has been bugging me that she can't ever use the computer because it never works (she doesn't like linux much).

Which brings me to my problem: nbhusain, you are the best candidate to help me with this again, when I passthrough the graphics card to the VM, DUET stops being able to load the sata controller. It simply can't find anything. At all. I'm booted directly to the shell where the only disk it finds is the img disk containing DUET. lspci confirms the device is owned by pciback, and there are no errors during the creation of the machine, so I really am not sure what is wrong...


----------



## nbhusain

@ lloyd mcclendon

Im using XEN with 2 radeon cards, my primary card is a radeon 5450 that use for dom0/kde desktop, the other one is a radeon 6950 that i passthrough to a windows 7 vm with gfx_passthru=1

@ Chetyre

Could you provide your logs, cfg file and lspci output, can you reproduce this every time you pass the card, and then without it? Also once you're dropped to the efi shell, try the command "pci" and check if your controller appears on the list, and "map" to check if your partitions are mapped.


----------



## lloyd mcclendon

I did pick up a 7870, and yes, the passthrough works.























I was using KVM at the time, and much to my surprise (I think the docs are out of date), it really just worked out of the box! WEI showed 7.8 / 7.9 (not counting the disk of course ..I will address that later).

However... I'm having some odd mouse issues, that are probably solvable, but more importantly, once I shut off the guest, if I try to turn it on again the entire host bricks.







that's not going to work. Reading around it appears it's still a limitation of KVM for the time being.










So, I am in the process of piloting XEN instead. I have the dom0 kernel all setup, and as far as I can tell I did everything right. The plan is to keep using the GTX570 with the proprietary nvidia-drivers on dom0, and send the 7870 to domU. However (again), I'm having some issues.

*Is anyone here using the proprietary nvidia-drivers on dom0?
*

The nvidia module did install just fine into the xen kernel. There's all kinds of (older) guides out there about adjusting the build to install it into the xen tree, but I'm pretty sure that is no longer necessary as the xen code is merged into the main tree. insmod nvidia works fine.

After that, startx fails to a black screen, and xorg.log shows some error about the *keyboard... but digging around for this, add "nopat" to the kernel boot line, and now it starts. I'm trying to make some sense of that, it is due to a bug in nvidia with vesafb. ok

Now I have X started, but glxgears shows 2 or 3 FPS. LOL. And if I kill the xserver or chvt, it's a blank screen. Apparently this can be fixed by using uvesafb, but I haven't found anything why I have no FPS.

Perhaps now I should just ditch this second nvidia card and go get a second 7870. *How is fglrx working out for you guys on dom0?* Nvidia is not on my favorites list lately, it's odd because a few years ago ATI was in this spot and the nvidia linux support was so much better, and here we are a complete 180.


----------



## lloyd mcclendon

Quote:


> Originally Posted by *nyxcharon*
> 
> So, I'm back.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> I recently acquired a Nvidia GT 640. I planned on using it on my Host (LM14) to play Steam on Linux games, however I've run into some issues. If I boot into xen with the card and no proprietary drivers, I can get into LM though Cinnamon crashes. If I install the drivers and setup my xorg, I just get a black screen on startup though I lose no video signal (it's just a black screen). I also can't switch to another TTY either.
> The card and drivers however do work absolutely fine when not booting with Xen, so I've narrowed down the problem slightly.
> 
> Any ideas on what to do?


[I'm searching back through the thread now]

If you still have any hair left and are still trying this, the black screen issue can be worked around with the nopat boot parameter or disabling it in the kernel config under processor --> MTRR -> X86_PAT (something like that, you'll see it) http://www.gossamer-threads.com/lists/xen/users/251308#251308 < the debian link there still doesn't explain why this works, though the kernel help item says to disable for bad video cards...

The chvt issue can apparently be fixed by uvesafb http://www.nvnews.net/vbulletin/showthread.php?p=2435805#post2435805 . though I haven't verified this yet, I'm trying to figure out the FPS issue...

Perhaps if you're still working on this, try nopat, and X should at least start. Tell us if you get FPS, though I doubt chvt will work.

I'm not sure how I feel about the way XEN works, it seems to have too deep of implications than I'd care to deal with. It's like booting into another OS almost.. KVM was much nicer in this regard.


----------



## nbhusain

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> I did pick up a 7870, and yes, the passthrough works.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> I was using KVM at the time, and much to my surprise (I think the docs are out of date), it really just worked out of the box! WEI showed 7.8 / 7.9 (not counting the disk of course ..I will address that later).
> 
> However... I'm having some odd mouse issues, that are probably solvable, but more importantly, once I shut off the guest, if I try to turn it on again the entire host bricks.
> 
> 
> 
> 
> 
> 
> 
> that's not going to work. Reading around it appears it's still a limitation of KVM for the time being.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> So, I am in the process of piloting XEN instead. I have the dom0 kernel all setup, and as far as I can tell I did everything right. The plan is to keep using the GTX570 with the proprietary nvidia-drivers on dom0, and send the 7870 to domU. However (again), I'm having some issues.
> 
> *Is anyone here using the proprietary nvidia-drivers on dom0?
> *
> 
> The nvidia module did install just fine into the xen kernel. There's all kinds of (older) guides out there about adjusting the build to install it into the xen tree, but I'm pretty sure that is no longer necessary as the xen code is merged into the main tree. insmod nvidia works fine.
> 
> After that, startx fails to a black screen, and xorg.log shows some error about the *keyboard... but digging around for this, add "nopat" to the kernel boot line, and now it starts. I'm trying to make some sense of that, it is due to a bug in nvidia with vesafb. ok
> 
> Now I have X started, but glxgears shows 2 or 3 FPS. LOL. And if I kill the xserver or chvt, it's a blank screen. Apparently this can be fixed by using uvesafb, but I haven't found anything why I have no FPS.
> 
> Perhaps now I should just ditch this second nvidia card and go get a second 7870. *How is fglrx working out for you guys on dom0?* Nvidia is not on my favorites list lately, it's odd because a few years ago ATI was in this spot and the nvidia linux support was so much better, and here we are a complete 180.


Well i tried kvm in the past, one of the issues i found was irregular performance while gaming, i think it had to do with the fact that you cant pin specific cpus to a vm, as for the crashes i did happen to me back then the solution was to build the kernel with pci-stub built-in and use "pci-stub.ids=" kernel parameter

ex:

My radeon card:
lspci

Code:



Code:


01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman PRO [Radeon HD 6950]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]

lspci -n

Code:



Code:


01:00.0 0300: 1002:6719
01:00.1 0403: 1002:aa80

So in my case i would need to add this as a kernel boot parameter

Code:



Code:


pci-stub.ids=1002:6719;1002:aa80

As for your problems with nvidia drivers + dom0 im pretty sure they work because i used to have a nvidia card + a radeon card and they both worked on dom0, but i know i had to change something in the kernel configuration, unfortunately i dont remember right now (i think last time i had it working was with the gentoo genkernel config file). As for fglrx drivers, well they're bad, i would suggest you keep trying with the nvidia card.


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> I did pick up a 7870, and yes, the passthrough works.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> I was using KVM at the time, and much to my surprise (I think the docs are out of date), it really just worked out of the box! WEI showed 7.8 / 7.9 (not counting the disk of course ..I will address that later).
> 
> However... I'm having some odd mouse issues, that are probably solvable, but more importantly, once I shut off the guest, if I try to turn it on again the entire host bricks.
> 
> 
> 
> 
> 
> 
> 
> that's not going to work. Reading around it appears it's still a limitation of KVM for the time being.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> So, I am in the process of piloting XEN instead. I have the dom0 kernel all setup, and as far as I can tell I did everything right. The plan is to keep using the GTX570 with the proprietary nvidia-drivers on dom0, and send the 7870 to domU. However (again), I'm having some issues.
> 
> *Is anyone here using the proprietary nvidia-drivers on dom0?
> *
> 
> The nvidia module did install just fine into the xen kernel. There's all kinds of (older) guides out there about adjusting the build to install it into the xen tree, but I'm pretty sure that is no longer necessary as the xen code is merged into the main tree. insmod nvidia works fine.
> 
> After that, startx fails to a black screen, and xorg.log shows some error about the *keyboard... but digging around for this, add "nopat" to the kernel boot line, and now it starts. I'm trying to make some sense of that, it is due to a bug in nvidia with vesafb. ok
> 
> Now I have X started, but glxgears shows 2 or 3 FPS. LOL. And if I kill the xserver or chvt, it's a blank screen. Apparently this can be fixed by using uvesafb, but I haven't found anything why I have no FPS.
> 
> Perhaps now I should just ditch this second nvidia card and go get a second 7870. *How is fglrx working out for you guys on dom0?* Nvidia is not on my favorites list lately, it's odd because a few years ago ATI was in this spot and the nvidia linux support was so much better, and here we are a complete 180.


1. Never tried KVM - found the documentation to be severely lacking. There also isn't much about PCI / VGA passthrough.

2. Tried nvidia proprietary driver but failed to make it work. Read about nvidia not supporting Xen. Lately I saw somewhere a workaround, but never tried nor remember where I saw it. 2-3 FPS looks like it's not using the proprietary driver, or something went seriously wrong.

3. About getting a second AMD card, just don't buy the 6450 - this card sucks even for basic web browsing. I will probably get a 6570 or so as I don't need much graphics power under Linux. By the way, I use a Nvidia card for Windows domU, a Quadro 2000. It's for photo editing and works nice. The Nvidia multi-OS graphics cards like the Quadro 2000 should work out of the box under Xen domU, but they are expensive and kinda special purpose (CAD/CAM, graphics workstations, video / picture editing). I wish Nvidia would make more graphics cards compatible with VGA passthrough.


----------



## lloyd mcclendon

Well I decided to just try the idea of ditching nvidia altogether, and using the 7870 for dom0 and probably another one for domU. I may go with a less powerful card for dom0...

And after some fighting, it did eventually work. There's a bug with fglrx+iommu that was worked around, but the fglrx performance is about 30% of what nvidia gave. Once I got that working w/o xen, I turned that on, and had to again disable PAT (temporarily I hope) due to a panic when starting X. It seems like I have a number of performance things to tune as it's a bit jumpy and seems a bit off, likely no PAT is one of them.

So, as was really stated long ago and many times in here, for this to work (and be healthy), you need Xen (not KVM yet..) and ATI cards all around. Unless you want to get into using nvflash or using their pro series cards, nvidia is just going to waste your time.

I guess I'm off to look into xen performance settings and see what I'm missing, test this setup out for a week and if all goes well, pick up a second ATI card.


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> I guess I'm off to look into xen performance settings and see what I'm missing, test this setup out for a week and if all goes well, pick up a second ATI card.


Within Windows, install the latest GPLPV drivers, in particular the disk driver. It improved my WEI index by a full 1.0 point to 7.8 using a SSD. My LVM striped HDDs are pretty speedy too.

Good luck!


----------



## lloyd mcclendon

Quote:


> Originally Posted by *powerhouse*
> 
> Within Windows, install the latest GPLPV drivers, in particular the disk driver. It improved my WEI index by a full 1.0 point to 7.8 using a SSD. My LVM striped HDDs are pretty speedy too.
> 
> Good luck!


yep, i did that last night. it didn't help WEI due to still being on a spinning HD. Well, I thought I installed it... then I found out you have to edit the bootloader config for it to load them.

I have successfully passed the VGA and sound card to the win7 domU using xen. This all worked just about the same as KVM, but I can gracefully shut off the guest without bricking the host.









There's one problem left though, I really can't figure this out. I am unable to pass a PCI usb controller to the guest. Well, it passes ok, the lights on the mouse shutoff so I know it's not attached to dom0 anymore. I've installed the drivers in the guest device manager shows the "ASPEED USB Controller", seems fine, but the mouse and the keyboard just do not come on. Even if I replug them in of course... I have two usb controllers in this board:

00:1d:00.0 echi, all kinds of stuff going on with this bus slot - staying in dom0 for now.
08:00:00.0 xhci, NOTHING else associated with this one at all. Very isolated and easy to pass over.

I send 08:00, it clicks off, windows makes noises and shows green icons and nice happy things that it's working fine. But no activity across the controller. Any thoughts? Did anyone run into this?

It's either
A) I'm missing something obvious or some implication about passing usb (or usb3). I can't imagine what as the sound and VGA are "harder" to do and worked real easily
B) A quirk with this particular controller or the way my board uses it -- will try to pick up a cheap PCI card one
C) A quirk with windows 7 and this controller -- will try a linux domU
D) A kernel bug in 3.7.10. Win7 did the same thing when I tried this with KVM, usb passthrough never worked despite what I thought I said.









Tried this again with just a jump drive in the controller, after "xl pci-assignable-add 0000:08:00.0", the light on it goes off. Then "xl pci-attach vwin7 0000:08:00.0" starts making the noise and showing the controller as detected and ready, but the light on the drive just doesn't come back on. It's like the controller just doesn't get reset.

Do you guys with the GPLPV drivers installed see an UNKONWN PCI DEVICE with the yellow ? or is that just me?


----------



## nbhusain

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> yep, i did that last night. it didn't help WEI due to still being on a spinning HD. Well, I thought I installed it... then I found out you have to edit the bootloader config for it to load them.
> 
> I have successfully passed the VGA and sound card to the win7 domU using xen. This all worked just about the same as KVM, but I can gracefully shut off the guest without bricking the host.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> There's one problem left though, I really can't figure this out. I am unable to pass a PCI usb controller to the guest. Well, it passes ok, the lights on the mouse shutoff so I know it's not attached to dom0 anymore. I've installed the drivers in the guest device manager shows the "ASPEED USB Controller", seems fine, but the mouse and the keyboard just do not come on. Even if I replug them in of course... I have two usb controllers in this board:
> 
> 00:1d:00.0 echi, all kinds of stuff going on with this bus slot - staying in dom0 for now.
> 08:00:00.0 xhci, NOTHING else associated with this one at all. Very isolated and easy to pass over.
> 
> I send 08:00, it clicks off, windows makes noises and shows green icons and nice happy things that it's working fine. But no activity across the controller. Any thoughts? Did anyone run into this?
> 
> It's either
> A) I'm missing something obvious or some implication about passing usb (or usb3). I can't imagine what as the sound and VGA are "harder" to do and worked real easily
> B) A quirk with this particular controller or the way my board uses it -- will try to pick up a cheap PCI card one
> C) A quirk with windows 7 and this controller -- will try a linux domU
> D) A kernel bug in 3.7.10. Win7 did the same thing when I tried this with KVM, usb passthrough never worked despite what I thought I said.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Tried this again with just a jump drive in the controller, after "xl pci-assignable-add 0000:08:00.0", the light on it goes off. Then "xl pci-attach vwin7 0000:08:00.0" starts making the noise and showing the controller as detected and ready, but the light on the drive just doesn't come back on. It's like the controller just doesn't get reset.
> 
> Do you guys with the GPLPV drivers installed see an UNKONWN PCI DEVICE with the yellow ? or is that just me?


Usb2 controllers come on pairs, for example if i want to passthrough a controller i would need to pass the ehci controller and the ohci controller, it might be the case with usb3, im not sure tho.

The unknown pci device after installing gplpv drivers is normal


----------



## lloyd mcclendon

Quote:


> The unknown pci device after installing gplpv drivers is normal


Ok thanks for confirming .. I thought so as I always had one of those with KVM as well...

Well, I setup a linux domU and everything works perfectly. Even the usb controller. I do see a similar unknown pci device in lspci under linux, so that must be expected. But what's the deal with windows and this USB controller? I don't know, I'm no good at windows problems, at least that's all it is.

I will try a clean install I guess.. I installed the windows drivers for it, it picks up the controller, but never turns on the ports. Perhaps an issue with these particular windows drivers of all things


----------



## nbhusain

It seems you've got the same usb3 controller as i do (ASM1042 SuperSpeed USB Host Controller), im passing it though to the windows vm and it works just fine, but in my case the controller appears as 3 different adapters

lspci

Code:



Code:


04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller                                                                                                                                                                                         
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller                                                                                                                                                                                         
06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

maybe its got something to do with that


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> Ok thanks for confirming .. I thought so as I always had one of those with KVM as well...
> 
> Well, I setup a linux domU and everything works perfectly. Even the usb controller. I do see a similar unknown pci device in lspci under linux, so that must be expected. But what's the deal with windows and this USB controller? I don't know, I'm no good at windows problems, at least that's all it is.
> 
> I will try a clean install I guess.. I installed the windows drivers for it, it picks up the controller, but never turns on the ports. Perhaps an issue with these particular windows drivers of all things


You got the same USB controller I got. Here is my lspci output:

Code:



Code:


# lspci -v | grep -i usb
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05) (prog-if 20 [EHCI])
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05) (prog-if 20 [EHCI])
08:00.0 USB controller: Renesas Technology Corp. Device 0015 (rev 02) (prog-if 30 [XHCI])
09:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller (prog-if 30 [XHCI])
0a:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller (prog-if 30 [XHCI])
0b:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller (prog-if 30 [XHCI])

The Renesas Technology USB 3.0 controller is located on an add-on SATA/USB controller board. By the way, if you add a controller board, the PCI IDs will change, so make sure to check.

At first I thought your mouse/keyboard is not working with USB 3 (I think Microsoft discourages the use of USB 3 ports for mouse/keyboard). But that doesn't explain why the drive doesn't work.

I'm passing through 0a:.. and 0b:.. and everything works now. I had some issues when I added the SATA/USB controller card, much the same as your problems. I chose to pass through a different controller and removed one SATA controller I had passed through and eventually it worked. Luckily I didn't need the passed through SATA controller - it was for an eSATA port I never used.

I can't explain why but it seems that sometimes choosing a different controller for pass through works. By the way, I'm still on xm toolstack (Xen hypervisor 4.1.3), not xl.


----------



## lloyd mcclendon

Well, after about 3 days of trial and error headache-ness, i finally got the usb controller to work right. Everything is fine with a linux domU... With a windows domU, I can pass ONE of the USB or VGA, and they'll work by themselves. As soon as I try to bring both in, the USB controller still "works", but it never sends power to the ports.... I also tried an old copy of XP I have, same thing, no power to the ports.

Luckily I stumbled into the fix, which was:

Code:



Code:


$ grep pci /etc/xen/vwin7.cfg
pci=['00:04:00.0', '00:04:00.1', '00:08:00.0,power_mgmt=1']

04 is the VGA & HDMI audio, 08 is the usb controller. The power_mgmt=1 setting allows the guest OS to have full control over the power mgmt of the usb, and now the windows driver actually sends power to the ports.









I haven't checked the performance or really tested the stability yet, but finally all of the hardware is at least working right.







. It did lock up earlier ... and there are some error messages in the qemu log still, perhaps an issue with msitranslate. RTFM







(warning: pdf) http://xen.org/files/Support/XenConfigurationDetails.pdf
Quote:


> pci:Add a PCI device to a domain, using given params (in hex). For example 'pci=c0:02.1'.
> If VSLOT is supplied the device will be inserted into that virtual slot in the guest, else a free
> slot is selected. *If msitranslate is set, MSI-INTx translation is enabled if possible. Guest that
> doesn't support MSI will get IO-APIC type IRQs translated from physical MSI, HVM only.
> Default is 1. The option may be repeated to add more than one pci device. If power_mgmt is
> set, the guest OS will be able to program the power states D0-D3hot of the device, HVM only.
> Default=0.*
> pci (Default=[]; Value=BUSEV.FUNC[@VSLOT][,msitranslate=0|1][,power_mgmt=0|
> 1]
> 
> vscsi: Add a SCSI device to a domain. The physical device is PDEV, which is exported to the
> domain as VDEV(X:X:X:X)
> vscsi (Default= [];Value= 'PDEV,VDEV[,DOM]')
> 
> pci_msitranslate: Global PCI MSI-INTx translation flag (0=disable;1=enable)
> pci_msitranslate (Default=1; Value='TRANSLATE')
> 
> pci_power_mgmt: Global PCI Power Management flag (0=disable; 1=enable)
> pci_power_mgmt (Default=0; Value='POWERMGT')
> 
> xen_platform_pci: Is xen_platform_used?
> xen_platform_pci (Default=1; Value='0|1')


sent from windows ... inside linux .. but you'd never notice!














fantastic. Now to get teh second set of cards to have linux setup primarily and get over here with a bump of the kvm switch.







If only this was a little less sensitive to setup correctly ...


----------



## powerhouse

@lloyd mcclendon: Great that you found a solution. By the way, have you tried the following global config option in your config file?:

Code:



Code:


pci_power_mgmt=1


----------



## lloyd mcclendon

Yes, I did try that and it caused windows to say "code 43" on the ATI card no matter what.









Also it seems I've shot myself in the foot as soon as I posted that, now when I reboot the domU, the passed cards no longer "reset". I swear it was working fine the other day, I think. I must have changed something else.

Code:



Code:


~dom0 # xl create linuxDomU
~dom0 # xl vnc linuxDomU

~linuxDomU $ startx
~linuxDomU # halt

After halting and switching back to dom0 on the KVM switch, then switching back to the dead domU on the switch, the monitors domU were using are still the blank screen but powered on state from the card that should be cut off.







Then when I try to restart domU the screens are forever stuck like that until I reset the host. I'm pretty sure this was working, but maybe not...


----------



## lloyd mcclendon

OK well I fixed that.. I was correct, it was working fine before.









Sometime during that 3 days of trial-and-error-hell I had switched away from the virt-manager / libvirt / xend / xm to just the new xl toolstack. It seems XL has a glitch or I'm doing something wrong with it where PCI reset does not always work the same way as it did with XM. I tried a few other things and barely just remembered I had switched and that was when it was working.









So all happy again.







To summarize:

XEN
ATI
+XM

success

anything else different from the above, good luck.


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> OK well I fixed that.. I was correct, it was working fine before.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Sometime during that 3 days of trial-and-error-hell I had switched away from the virt-manager / libvirt / xend / xm to just the new xl toolstack. It seems XL has a glitch or I'm doing something wrong with it where PCI reset does not always work the same way as it did with XM. I tried a few other things and barely just remembered I had switched and that was when it was working.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> So all happy again.
> 
> 
> 
> 
> 
> 
> 
> To summarize:
> 
> XEN
> ATI
> +XM
> 
> success
> 
> anything else different from the above, good luck.


That's good news! I found this http://osdir.com/ml/general/2013-03/msg34678.html that may shed some light on your problem with the xl toolstack. I for now am running Xen 4.1.3 with xend / xm toolstack and don't find any reason to change that. It's the hypervisor in the Linux Mint 14 / Ubuntu 12.10 repositories and was real easy to install. I've been running this for the past 1/2 year or more and it works very well.


----------



## Virum

Im trying to get XEN to work with my Nvidia 560TI but when I boot into the DOM0 I get a black screen.

As i skim through this thread im starting to feel like Nvidia just is not compatible currently?

Is KVM an alternative for gaming using VM? Im getting mixed feelings on KVM as I browse through here.

Also I have a i7-3770k but my motherboard gives me the option for VT-d still. Am i still able to follow this guide?


----------



## powerhouse

Quote:


> Originally Posted by *Virum*
> 
> Im trying to get XEN to work with my Nvidia 560TI but when I boot into the DOM0 I get a black screen.
> 
> As i skim through this thread im starting to feel like Nvidia just is not compatible currently?
> 
> Is KVM an alternative for gaming using VM? Im getting mixed feelings on KVM as I browse through here.
> 
> Also I have a i7-3770k but my motherboard gives me the option for VT-d still. Am i still able to follow this guide?


Let's start with the painful answer: The i7-3770K won't do. You need one that is listed here http://ark.intel.com/search/advanced/?s=t&FamilyText=3rd%20Generation%20Intel%C2%AE%20Core%E2%84%A2%20i7%20Processors&VTD=true.

KVM won't help here - AFAIK KVM has the same hardware requirements. Even if your CPU and graphics card were supported, it will be more difficult to get things running smooth under KVM. PCI and VGA passthrough were originally developed under Xen, whereas the others (KVM, VMware, etc.) try to get there. I too wanted to use KVM, based on the Phoronix benchmarks and all the hype KVM received/receives. Having gained a better understanding and running Xen for more than half a year I now believe the Phoronix Xen benchmarks are often flawed, to say the least.

With regard to Nvidia graphics cards, the best place to find a solution (if any) is here: http://www.davidgis.fr/documentation/Xen_VGA_PassThrough/doc.html. Instructions for how to build the Xen hypervisor etc. are here: http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through.
David uses the Nvidia GTX 560 Ti in his system, with Windows XP 64 bit as guest.

My advise, get one of the AMD cards mentioned in this thread or listed here: http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters (I try to maintain a list of supported GPUs). Another source of information is http://wiki.xen.org/wiki/Xen_VGA_Passthrough_Tested_Adapters.

Good luck - if you are a dual-booter now, it might well be worth the trouble.


----------



## Virum

Great answer. Im disappointed I did not get the 3770 instead of 3770k (Both are fantastic chips! But I didn't know I'd have this option of VM when i got it). Also there are a lot of con's I'm seeing in those nvidia articles.

Ill have to stick with Wine tweaking. Dual Booting for gaming is just too inconvenient. Every time i do it I find my self just defaulting to Windows despite my distaste of it.


----------



## lloyd mcclendon

KVM currently 'sort-of' works with passthrough - it will assign and may even be picked up and work ok, but the code to 'reset' the card back to normal when the VM shuts off is currently not there. So if you shutdown your guest, your host will crash if you try to start the guest back up. I experienced this myself and there's plenty of other KVM users out there who have documented the same behavior, regardless of hardware selected. The card did actually work fine until I shut the guest off. I'd expect QEMU/KVM to have this code added in a year or two - I may switch back at that point, but lets see how XEN goes for a bit first.

Regarding NVIDIA, supposedly the 'quadro' series cards do support it, but whether or not anyone has gotten it to work is spotty at best. There are even some users out there who have used the 'nvflash' tool to hack in their custom built firmware updates with memory mappings built to support resetting the card. There's a neat pdf out there somewhere where the guy loosely outlines the steps to do this. I would never do this as it's a good way to brick your card ... say you type 0x00007A400000 instead of 0x0007A400000









Also there are some blurbs out there that if you contact nvidia support they will send you patched firmware for the card, but when I called support they said this is misinformation and they have never offered said patches.

"nvidia has been the *single worst company* we've ever dealt with ... so nvidia, :bleep: you." -Linus


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> KVM currently 'sort-of' works with passthrough - it will assign and may even be picked up and work ok, but the code to 'reset' the card back to normal when the VM shuts off is currently not there. So if you shutdown your guest, your host will crash if you try to start the guest back up. I experienced this myself and there's plenty of other KVM users out there who have documented the same behavior, regardless of hardware selected. The card did actually work fine until I shut the guest off. I'd expect QEMU/KVM to have this code added in a year or two - I may switch back at that point, but lets see how XEN goes for a bit first.
> 
> Regarding NVIDIA, supposedly the 'quadro' series cards do support it, but whether or not anyone has gotten it to work is spotty at best. There are even some users out there who have used the 'nvflash' tool to hack in their custom built firmware updates with memory mappings built to support resetting the card. There's a neat pdf out there somewhere where the guy loosely outlines the steps to do this. I would never do this as it's a good way to brick your card ... say you type 0x00007A400000 instead of 0x0007A400000
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Also there are some blurbs out there that if you contact nvidia support they will send you patched firmware for the card, but when I called support they said this is misinformation and they have never offered said patches.
> 
> "nvidia has been the *single worst company* we've ever dealt with ... so nvidia, :bleep: you." -Linus


KVM: I've seen similar reports on the web (but have no hands-on experience).

Nvidia Quadro: I have the Quadro 2000 and it works great with VGA passthrough (as secondary GPU). Really easy! I had the Quadro 600 before and that one DOESN'T work! Nvidia has a line of professional cards that are specified "multi-OS" - just search for that and you'll get a list of all supported cards. They should work with VGA passthrough. That said, you would have to spend a LOT of money to get a Quadro card that's useful for gaming - most Quadro's aren't designed for that and deliver very mediocre benchmark results.

BUT, an hour ago I found this thread/post: http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550.

It turns out you can hardware modify consumer Nvidia cards to identify themselves as pro-cards, thus enabling the pro features as well as VGA passthrough. I really only read a few posts on that forum, but it looks very promising. Certainly this kind of hacking is for the adventurous among us.


----------



## powerhouse

Quote:


> Originally Posted by *Virum*
> 
> Great answer. Im disappointed I did not get the 3770 instead of 3770k (Both are fantastic chips! But I didn't know I'd have this option of VM when i got it). Also there are a lot of con's I'm seeing in those nvidia articles.
> 
> Ill have to stick with Wine tweaking. Dual Booting for gaming is just too inconvenient. Every time i do it I find my self just defaulting to Windows despite my distaste of it.


CPU: Maybe you can exchange it with someone who has a 3770?

Nvidia: I just ran into this thread: http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/.
See also this post: http://www.eevblog.com/forum/projects/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550.
Haven't managed to read much, but this looks real cool.


----------



## nbhusain

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> KVM currently 'sort-of' works with passthrough - it will assign and may even be picked up and work ok, but the code to 'reset' the card back to normal when the VM shuts off is currently not there. So if you shutdown your guest, your host will crash if you try to start the guest back up. I experienced this myself and there's plenty of other KVM users out there who have documented the same behavior, regardless of hardware selected. The card did actually work fine until I shut the guest off. I'd expect QEMU/KVM to have this code added in a year or two - I may switch back at that point, but lets see how XEN goes for a bit first.
> 
> Regarding NVIDIA, supposedly the 'quadro' series cards do support it, but whether or not anyone has gotten it to work is spotty at best. There are even some users out there who have used the 'nvflash' tool to hack in their custom built firmware updates with memory mappings built to support resetting the card. There's a neat pdf out there somewhere where the guy loosely outlines the steps to do this. I would never do this as it's a good way to brick your card ... say you type 0x00007A400000 instead of 0x0007A400000
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Also there are some blurbs out there that if you contact nvidia support they will send you patched firmware for the card, but when I called support they said this is misinformation and they have never offered said patches.
> 
> "nvidia has been the *single worst company* we've ever dealt with ... so nvidia, :bleep: you." -Linus


No not really, like i mentioned in a previous post kvm will crash the host when doing vga passthrough if you dont bind your gpu to pci-stub before the radeon driver loads, i recently switched from arch + xen to opensuse + kvm because of xen poor power management options on my amd machine (suspend doesnt work, cool'n quiet crash my system, etc), and im passing though a radeon 6950 without issues, i can reboot or shutdown the vm as much as i want without a single crash, and it seems most of my performance issues in the past were related to ksm.

Here's how i bind my gpu:

First of all you need to build your kernel with pci-stub built-in or add the pci-stub module to your initrd, then you need to find out the ids of your card.

lspci

Code:



Code:


07:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cayman PRO [Radeon HD 6950]                                                                                
07:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]

lspci -n

Code:



Code:


07:00.0 0300: 1002:6719
07:00.1 0403: 1002:aa80

So now you add this to your grub config file:

Code:



Code:


pci-stub.ids=1002:6719,1002:aa80

It will end up looking something like this:

Code:



Code:


linux   /vmlinuz-3.7.10-1.1-desktop root=/dev/mapper/system-root   resume=/dev/system/swap splash=silent quiet showopts iommu=1  pci-stub.ids=1002:6719,1002:aa80

dmesg | grep pci-stub

Code:



Code:


[    2.136667] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.136674] pci-stub 0000:07:00.0: claimed by stub
[    2.136676] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.136682] pci-stub 0000:07:00.1: claimed by stub

Also there's some interesting development in the kvm + qemu camp using vfio-pci to get primary vga passthrough working on NVIDIA and AMD using secondary devices, here are the patches for qemu:

http://lists.gnu.org/archive/html/qemu-devel/2013-02/msg02758.html
http://lists.gnu.org/archive/html/qemu-devel/2013-02/msg02760.html
http://lists.gnu.org/archive/html/qemu-devel/2013-02/msg02759.html


----------



## powerhouse

Hi guys (and gals). I just couldn't resist posting the Passmark results of my Windows 7 Pro 64 bit *virtual machine* running on the Xen hypervisor:



The only "weak" point is 3D graphics which is due to my graphics card, a Nvidia Quadro 2000. Actually, my 3D Graphics Mark (1,369) inside the VM is even better than the average performance for that card shown on Passmark (1,295)







- see http://www.videocardbenchmark.net/gpu.php?gpu=Quadro+2000.

Xen rocks


----------



## Dh0mp5eur

Hello
I use KVM on Debian Sid with a i7 3770 Asus P8Z77-M and HD7970 for the VM.
It works perfectly.
But I can't restart my VM.

I enable pci-stub in grub

Code:



Code:


GRUB_CMDLINE_LINUX="intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0"

dmesg |grep pci-stub

Code:



Code:


[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0 quiet

kvm   1:1.1.2+dfsg-6
qemu-kvm     1.1.2+dfsg-6


----------



## nbhusain

You need to have the pci-stub module compiled built-in in the kernel or inside your initramfs, you should see something like this:

dmesg | grep pci-stub

Code:



Code:


[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-nb root=/dev/mapper/system-root ro iommu=1 pci-stub.ids=1002:4391,1002:6719,1002:aa80 acpi_sleep=s3_bios,s3_mode quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-nb root=/dev/mapper/system-root ro iommu=1 pci-stub.ids=1002:4391,1002:6719,1002:aa80 acpi_sleep=s3_bios,s3_mode quiet
[    2.087342] pci-stub: add 1002:4391 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.087350] pci-stub 0000:00:11.0: claimed by stub
[    2.087363] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.087371] pci-stub 0000:07:00.0: claimed by stub
[    2.087375] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

pci-stub has to grab the card before the radeon module otherwise your system will crash


----------



## Dh0mp5eur

I loaded the pci-stub in the initramfs.
I blacklisted radeon module.
But I still can't restart my VM.

Code:



Code:


dmesg |grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0 quiet
[    1.031665] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.031671] pci-stub 0000:01:00.0: claimed by stub
[    1.031675] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.031680] pci-stub 0000:01:00.1: claimed by stub


----------



## nbhusain

Does the host crash in your case?


----------



## myweb

Dear all,
I faced with passthrough issue of NVIDIA GT640 on the following configuration:
Intel(R) Core(TM) i5-3470
AsRock Z77 Pro4
Ubuntu Raring (Kubuntu 13.04 Beta 2)
xen-hypervisor-4.2.1

/etc/modprobe.d/xen-pciback.conf

Code:



Code:


blacklist nvidia
options xen-pciback hide=(0000:01:00.0)(0000:01:00.1)

/etc/modules

Code:



Code:


xen-pciback

dmesg | grep pcib

Code:



Code:


[    0.000000] Command line: placeholder root=UUID=cf9f10d5-40d6-41fe-8e10-b7801ef3b87e ro intel_iommu=on xen-pciback.passthrough=1 xen-pciback.hide=(0000:01:00.0)(0000:01:00.1) quiet splash
[    2.165897] Kernel command line: placeholder root=UUID=cf9f10d5-40d6-41fe-8e10-b7801ef3b87e ro intel_iommu=on xen-pciback.passthrough=1 xen-pciback.hide=(0000:01:00.0)(0000:01:00.1) quiet splash
[    5.444046] pciback 0000:01:00.0: seizing device
[    5.444055] pciback 0000:01:00.1: seizing device
[    5.444238] pciback 0000:01:00.0: enabling device (0000 -> 0003)
[    5.450089] xen-pciback: backend is passthrough

Guest OS is Windows 7. Win7 recognized videocard, Nvidia driver was installed, but device manager show Yellow Triangle with Exclamation Mark and Error Code 43.
Could you please clarify are there any chances to get working Nvidia gt640 under xen guest?
Also I am planing to use another Linux guest as movie player (HTPC) since Nvidia hardware acceleration is better than ATI, but right now Ubuntu guest with NVIDIA GT640 hang up during boot.


----------



## nbhusain

You need need patches to get vga passthrough working with NVIDIA cards on xen or kvm, take a look at http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through, u'll find patches to get nvidia cards working under xen, for kvm, there's some experimental patches on the qemu-devel mailing list from april 1.


----------



## myweb

I am quite confused: which revision of xen I should download and which patches I should apply(there are a lot of different version of patches)?


----------



## Dh0mp5eur

Quote:


> Originally Posted by *nbhusain*
> 
> Does the host crash in your case?


Yes my PC crash when rester VM.


----------



## powerhouse

Quote:


> Originally Posted by *Dh0mp5eur*
> 
> I loaded the pci-stub in the initramfs.
> I blacklisted radeon module.
> But I still can't restart my VM.
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> dmesg |grep pci-stub
> [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0 quiet
> [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0 quiet
> [    1.031665] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [    1.031671] pci-stub 0000:01:00.0: claimed by stub
> [    1.031675] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [    1.031680] pci-stub 0000:01:00.1: claimed by stub


How did you blacklist the radeon module?

There are two ways to blacklist:

1. Create or edit blacklist-radeon.conf in /etc/modprobe.d and enter

Code:



Code:


blacklist radeon

2. Edit the /etc/default/grub.cfg file (in Ubuntu, can't remember where Fedora puts it) and add "radeon.blacklist=1" as shown here:

Code:



Code:


GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet splash nomodeset"

Update grub.

Some people report that the first option didn't work for them whereas the second one using the kernel boot option did the trick.


----------



## nbhusain

Quote:


> Originally Posted by *Dh0mp5eur*
> 
> Yes my PC crash when rester VM.


How many cards you've got in your system? are you passing thru your primary card or secondary card?


----------



## Dh0mp5eur

Quote:


> Originally Posted by *nbhusain*
> 
> How many cards you've got in your system? are you passing thru your primary card or secondary card?


I use the Intel IGP for the host and HD7970 on the Windows 7 VM.

Code:



Code:


How did you blacklist the radeon module?

There are two ways to blacklist:

1. Create or edit blacklist-radeon.conf in /etc/modprobe.d and enter
Code:
blacklist radeon

2. Edit the /etc/default/grub.cfg file (in Ubuntu, can't remember where Fedora puts it) and add "radeon.blacklist=1" as shown here:
Code:
GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet splash nomodeset"

Update grub.

Some people report that the first option didn't work for them whereas the second one using the kernel boot option did the trick.

I use the first option.
I will test the second
Thx


----------



## Dh0mp5eur

No, I still can't restart my VM.

Code:



Code:


dmesg |grep pci-stub
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on radeon.blacklist=1 pci-stub.ids=1002:6798,1002:aaa0 radeon.blacklist=1 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on radeon.blacklist=1 pci-stub.ids=1002:6798,1002:aaa0 radeon.blacklist=1 quiet
[    1.022224] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.022231] pci-stub 0000:01:00.0: claimed by stub
[    1.022236] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.022239] pci-stub 0000:01:00.1: claimed by stub

/etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="intel_iommu=on radeon.blacklist=1 pci-stub.ids=1002:6798,1002:aaa0"


----------



## powerhouse

Quote:


> Originally Posted by *myweb*
> 
> I am quite confused: which revision of xen I should download and which patches I should apply(there are a lot of different version of patches)?


Perhaps this link describes it better. It's the same David Techer website, just a different place.


----------



## lloyd mcclendon

Yeah, the 7000 series cards have an issue where they wont reset properly, the 6000 series cards do not have this problem. I believe I read that in this thread somewhere: http://thread.gmane.org/gmane.comp.emulators.kvm.devel/94524 or maybe it was here: http://tavi-tech.blogspot.com/2012/03/vga-passthrough-with-kvm-on-fedora-16.html

I am also back to using KVM, I could make a very long winded post out of this, but XEN just isn't a good fit for me. For one thing the block performance is in the dumps. I followed all the guides and did everything right, and KVM overall out performs it by a good bit. The stability wasn't perfect either, the word "sometimes" can be used a lot ... I think these pictures sum it up for me:







I think all of the performance problems can be attributed to the fact that dom0 is involved like that, plus XEN has to write their own CPU scheduler etc etc etc, none of which can possibly be as good as what's been invested in and hacked at in the main kernel tree for 15 years.

With KVM the only problem I have is that I can't reset the guest. This is the same behavior as the XEN XL toolstack. The XEN XM toolstack did work, which tells me the hardware does support it, it's just a matter of getting the right code in place to make everything happy. I need to get the source for XM and take a good look at what it is doing vs what XL / KVM is not doing. I expect this to be fixed by someone possibly myself in the next 3-6 months.

Also the latest stable amd64 versions of libvirt / qemu in portage had an IRQ issue with this usb controller, the latest testing ~amd64 versions fixed this problem. So upgrade upgrade, latest and greatest is probably what you want if you're on the KVM side.

There also is the VFIO module and qemu patches, which I will be looking at shortly. It seems like a much more well thought out approach to passthrough. The modules are already in the main kernel tree, the qemu patches should be merged soon I think, and then libvirt support to follow.


----------



## powerhouse

Quote:


> Originally Posted by *Dh0mp5eur*
> 
> No, I still can't restart my VM.
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> dmesg |grep pci-stub
> [    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on radeon.blacklist=1 pci-stub.ids=1002:6798,1002:aaa0 radeon.blacklist=1 quiet
> [    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.8-trunk-amd64 root=UUID=4e436466-36c9-45a9-a455-2ad879c7baaf ro intel_iommu=on radeon.blacklist=1 pci-stub.ids=1002:6798,1002:aaa0 radeon.blacklist=1 quiet
> [    1.022224] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [    1.022231] pci-stub 0000:01:00.0: claimed by stub
> [    1.022236] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [    1.022239] pci-stub 0000:01:00.1: claimed by stub
> 
> /etc/default/grub
> GRUB_DEFAULT=0
> GRUB_TIMEOUT=5
> GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
> GRUB_CMDLINE_LINUX_DEFAULT="quiet"
> GRUB_CMDLINE_LINUX="intel_iommu=on radeon.blacklist=1 pci-stub.ids=1002:6798,1002:aaa0"


Just to make sure: After you edited the grub file under /etc/default, did you run update-grub2 (or whatever your Linux distro needs to actually update the grub file)? Without that your changes won't be applied.

By the way, I put my radeon.blacklist here:

Code:



Code:


GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet"


----------



## nbhusain

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> Yeah, the 7000 series cards have an issue where they wont reset properly, the 6000 series cards do not have this problem. I believe I read that in this thread somewhere: http://thread.gmane.org/gmane.comp.emulators.kvm.devel/94524 or maybe it was here: http://tavi-tech.blogspot.com/2012/03/vga-passthrough-with-kvm-on-fedora-16.html
> 
> I am also back to using KVM, I could make a very long winded post out of this, but XEN just isn't a good fit for me. For one thing the block performance is in the dumps. I followed all the guides and did everything right, and KVM overall out performs it by a good bit. The stability wasn't perfect either, the word "sometimes" can be used a lot ... I think these pictures sum it up for me:
> 
> 
> 
> 
> 
> 
> 
> I think all of the performance problems can be attributed to the fact that dom0 is involved like that, plus XEN has to write their own CPU scheduler etc etc etc, none of which can possibly be as good as what's been invested in and hacked at in the main kernel tree for 15 years.
> 
> With KVM the only problem I have is that I can't reset the guest. This is the same behavior as the XEN XL toolstack. The XEN XM toolstack did work, which tells me the hardware does support it, it's just a matter of getting the right code in place to make everything happy. I need to get the source for XM and take a good look at what it is doing vs what XL / KVM is not doing. I expect this to be fixed by someone possibly myself in the next 3-6 months.
> 
> Also the latest stable amd64 versions of libvirt / qemu in portage had an IRQ issue with this usb controller, the latest testing ~amd64 versions fixed this problem. So upgrade upgrade, latest and greatest is probably what you want if you're on the KVM side.
> 
> There also is the VFIO module and qemu patches, which I will be looking at shortly. It seems like a much more well thought out approach to passthrough. The modules are already in the main kernel tree, the qemu patches should be merged soon I think, and then libvirt support to follow.


Im using those patches atm with my radeon 6950 and they seem to work just fine, i can do a primary passthrough of the card (with -vga none), you'll need to use kernel ver > 3.9 (im using 3.9rc5) and you need to enable vfio

Code:



Code:


                                                   --- VFIO Non-Privileged userspace driver framework                                                                       
                                                   <*>   VFIO support for PCI devices                                                                                       
                                                   [*]     VFIO PCI support for VGA devices

To pass it through ex:

Code:



Code:


qemu-system-x86_64 -device vfio-pci,host=07:00.0,x-vga=on

EDIT: you can build qemu from this tree https://github.com/awilliam/qemu-vfio, it contains those patches
EDIT2: have you tried ejecting the gpu before restarting?, i remember i had some performance isssues with xen after i rebooted the guest os, but ejecting it from windows seems to reset the card properly


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> ...I am also back to using KVM, I could make a very long winded post out of this, but XEN just isn't a good fit for me. For one thing the block performance is in the dumps. I followed all the guides and did everything right, and KVM overall out performs it by a good bit. The stability wasn't perfect either, the word "sometimes" can be used a lot ... I think these pictures sum it up for me:


The diagram is kinda comparing apples and eggs (post kernel 3 KVM with pre kernel 3 Xen). IIRC Xen is also integrated in the kernel. Just that Xen uses a hypervisor whereas KVM is part of Linux.

Anyway, KVM is the new kid on the block and it seems that Redhat and others are putting a lot of resources into it.

About block device performance, I would be interested in how KVM performs compared to Xen. I have posted my Passmark results summary before and have a more detailed breakdown here. These Passmark benchmarks were all run in the Windows 7 domU using the latest GPLPV driver and include the following separate test runs:

1. SSD performance (LVM formated SSD volume) - Sandisk Extreme 120GB.
2. Two WD20EARX 2TB HDDs in striped LVM (similar to RAID0).
3. Logical volume consisting of one WD1001FALS and one WD5000AAKS HDD (not striped).

As for Linux performance on Xen, the best is to use PV guests that come with pre-installed Xen PV drivers. Another option is the use of a Linux HVM guest with PV-HVM drivers. I haven't done any performance tests on Linux domUs, but if there was any Xen performance issue of Linux guests, Amazon, Rackspace, and many other Xen users would probably have migrated to a different solution.

If neither a PV nor a HVM guest and drivers work for you, there is still the option of PCI passthrough of the SATA controller.

All that said, I've never tried nor compared KVM with Xen, so I'm curious.


----------



## Dh0mp5eur

Quote:


> Originally Posted by *powerhouse*
> 
> Just to make sure: After you edited the grub file under /etc/default, did you run update-grub2 (or whatever your Linux distro needs to actually update the grub file)? Without that your changes won't be applied.
> 
> By the way, I put my radeon.blacklist here:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet"


Yes i use "update-grub2" and "update-initramfs -u" after changes.

Code:



Code:


cat /etc/modprobe.d/blacklist-radeon.conf 
blacklist radeon

cat /etc/modprobe.d/pci-stub.conf 
options pci-stub ids=1002:6798,1002:aaa0

cat /etc/default/grub 
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet"
GRUB_CMDLINE_LINUX="intel_iommu=on pci-stub.ids=1002:6798,1002:aaa0"

cat /etc/initramfs-tools/modules 
pci_stub ids=1002:6798,1002:aaa0
kvm
kvm_intel


----------



## nbhusain

Could you try ejecting the gpu on windows before restarting?

EDIT:
Did you try to load your card rom from a file? i dont know if its related but i managed to crash my host restarting the vm with vfio-pci, it got fixed after i loaded the bios from a file using "-device vfio-pci,host=07:00.0,x-vga=on,romfile=/usr/share/qemu/vgabios-6950.bin", before that i got a "invalid rom content message" and the host would crash


----------



## lloyd mcclendon

Quote:


> Originally Posted by *nbhusain*
> 
> Could you try ejecting the gpu on windows before restarting?



















this actually just worked....






























thank you x10000







yeah, before shutting off the windows guest, eject the card, go to the vnc window to shutdown from there (or rdesktop or virsh shutdown). works everytime. I think we're good here. The only problem would be when windows update decides to restart on it's own (or anything else), but as long as I can control to eject the card first, it's a pretty good workaround. Otherwise the host would still crash, so the underlying issue is still there.

I wonder what the 'eject' actually does to make it work & if there's a way to force that everytime guest or host side..


----------



## lloyd mcclendon

Quote:


> Originally Posted by *powerhouse*
> 
> if there was any Xen performance issue of Linux guests, Amazon, Rackspace, and many other Xen users would probably have migrated to a different solution.
> 
> All that said, I've never tried nor compared KVM with Xen, so I'm curious.


Yeah, by no means am I trying to bash XEN. It's a respectable piece of software and has it's place. Just for my particular purposes and use habits KVM fits better. If I was running an EC2 type cloud here, a bunch of XEN pvs and an empty dom0 would make sense.

I may have still had something set wrong, but after following all the guides / lots of digging around, the guest performance was not in the ballpark.

emerge chromium took about 6 hours with xen
with kvm the same guest eats it up in about 30 minutes....

to start the windows guest took about a full 2 minutes with xen, very slow at the windows gui boot screen...
only about 30 seconds with kvm

pretty sure i had everything set right, tried everything...


----------



## nbhusain

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> this actually just worked....
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> thank you x10000
> 
> 
> 
> 
> 
> 
> 
> yeah, before shutting off the windows guest, eject the card, go to the vnc window to shutdown from there (or rdesktop or virsh shutdown). works everytime. I think we're good here. The only problem would be when windows update decides to restart on it's own (or anything else), but as long as I can control to eject the card first, it's a pretty good workaround. Otherwise the host would still crash, so the underlying issue is still there.
> 
> I wonder what the 'eject' actually does to make it work & if there's a way to force that everytime guest or host side..


Nice!, take a look at this page http://lists.xen.org/archives/html/xen-users/2012-02/msg00443.html, there's seems to be a tool called "DevEject" to automate the process.

Did you try to load your gpu option-rom from a file?, i was having the same problem using vfio-pci, rebooting the guest would crash the host, but after i passed the romfile= parameter i got it working, also using pci-stub i remember i sometimes the host would crash when i opened gpu-z after a reboot, but after passing the romfile it didnt happen again.


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> Yeah, by no means am I trying to bash XEN. It's a respectable piece of software and has it's place. Just for my particular purposes and use habits KVM fits better. If I was running an EC2 type cloud here, a bunch of XEN pvs and an empty dom0 would make sense.
> 
> I may have still had something set wrong, but after following all the guides / lots of digging around, the guest performance was not in the ballpark.
> 
> emerge chromium took about 6 hours with xen
> with kvm the same guest eats it up in about 30 minutes....
> 
> to start the windows guest took about a full 2 minutes with xen, very slow at the windows gui boot screen...
> only about 30 seconds with kvm
> 
> pretty sure i had everything set right, tried everything...


I'm sure you didn't mean to bash Xen. It's just the huge performance difference you encountered that doesn't really make sense. But what gives - if you can make KVM work the way you want with little effort, yet Xen gives you a hard time, the choice is obvious.

Here is a benchmark by Ian Campbell / Xen in response to a Phoronix article and benchmark: http://blog.xen.org/index.php/2011/11/29/baremetal-vs-xen-vs-kvm-redux/. As you can easily see from the article, even benchmark sites like Phoronix get things wrong, or put differently, don't delve enough into the matter to resolve obvious performance issues.

I would have gone the KVM way but at the time I started there simply wasn't much information / documentation available on KVM and VGA passthrough. I'm glad to see that things have improved and I might even jump boats if I could see a justification. Since I'm using Xen and my Windows domU on a production system, I'm less inclined to just try out some new stuff, or install a totally different system.

By the way, were you able to use your existing Windows VM in both Xen and KVM? Or did you have to do a new installation? To me reinstalling Windows and all the apps would be a major pain in the neck.

By the way: Booting Windows (login prompt) on my Xen box takes 46 seconds. Not particularly fast, but I can live with it.


----------



## richardstallman

I've taken on one of these hypervisor "gaming" machines as a project, but I'm trying a different approach. It seems like openSUSE has the framework to really handle a lot of the dirty work for me (Xen 4.2.1 and the latest kernel come with SUSE right out of the box, and it has an essentially idiot-proof YaST module to handle the setup of KVM if I so desire). However, I am confused about a few things, and I'd appreciate some help:


As of today, should I be aiming for KVM or Xen? Which is "better" for a casual such as myself? (Ease of use is great! The simpler the setup, the better, for me.)
If I go with Xen, should I use the xl or the xm toolstack?
Can either Xen or KVM support block-level, raw drive mounting for a host? In other words, I want to unmount an entire drive from dom0, mount it within a domU, and have it behave exactly as a "regular" drive would--i..e. I can unmount it, remount it in dom0, and see all of the files as if a normal host system wrote to the drive. I did look into this with Xen, but the only results I could find involved mucking around with LVM. Is that the only way to accomplish "direct" drive access from one of these VMs, or am I missing something?
Thanks guys


----------



## nbhusain

Quote:


> Originally Posted by *richardstallman*
> 
> I've taken on one of these hypervisor "gaming" machines as a project, but I'm trying a different approach. It seems like openSUSE has the framework to really handle a lot of the dirty work for me (Xen 4.2.1 and the latest kernel come with SUSE right out of the box, and it has an essentially idiot-proof YaST module to handle the setup of KVM if I so desire). However, I am confused about a few things, and I'd appreciate some help:
> 
> 
> As of today, should I be aiming for KVM or Xen? Which is "better" for a casual such as myself? (Ease of use is great! The simpler the setup, the better, for me.)
> If I go with Xen, should I use the xl or the xm toolstack?
> Can either Xen or KVM support block-level, raw drive mounting for a host? In other words, I want to unmount an entire drive from dom0, mount it within a domU, and have it behave exactly as a "regular" drive would--i..e. I can unmount it, remount it in dom0, and see all of the files as if a normal host system wrote to the drive. I did look into this with Xen, but the only results I could find involved mucking around with LVM. Is that the only way to accomplish "direct" drive access from one of these VMs, or am I missing something?
> Thanks guys



Both have pros and cons, im using kvm myself atm but ive used xen in the past, if you want to get a setup running as fast as possible kvm is the way to go all you need is the linux kernel and qemu.
Well xm is deprecated, its not under development anymore, only bug fixes, but i had better results using it since some radeon cards are not correctly reseted by xl.
Yes, on xen i think it was something like this: disk = [phy:/dev/sda1,xvda,w],using seabios. you can also pass an entire sata controller on ahci or ide more and it will boot from a drive attached to the controller.


----------



## richardstallman

Quote:


> Originally Posted by *nbhusain*
> 
> 
> Both have pros and cons, im using kvm myself atm but ive used xen in the past, if you want to get a setup running as fast as possible kvm is the way to go all you need is the linux kernel and qemu.
> Well xm is deprecated, its not under development anymore, only bug fixes, but i had better results using it since some radeon cards are not correctly reseted by xl.
> Yes, on xen i think it was something like this: disk = [phy:/dev/sda1,xvda,w],using seabios. you can also pass an entire sata controller on ahci or ide more and it will boot from a drive attached to the controller.


Thanks for the help. Does KVM have the same issue as xl where the card isn't reset?


----------



## Scorpion667

"it's benefits, usage and limitations. "

Where are the limitations? is there more input lag?


----------



## nbhusain

Quote:


> Originally Posted by *richardstallman*
> 
> Thanks for the help. Does KVM have the same issue as xl where the card isn't reset?


Not as secondary passthrough, atm im using experimental patches for qemu and seabios that allow primary vga passthrough and it does show the problem, the card doesnt reset correctly which cause a 90% performance drop on 3d applications :S


----------



## richardstallman

Quote:


> Originally Posted by *nbhusain*
> 
> Not as secondary passthrough, atm im using experimental patches for qemu and seabios that allow primary vga passthrough and it does show the problem, the card doesnt reset correctly which cause a 90% performance drop on 3d applications :S


Dropped back in to say that I actually got everything working great with KVM. However, I'm having the issue wherein I can not reset the Windows guest without it locking up upon the second boot. If I kill the guest during this frozen second boot and then start it again, it freezes the host. I've tried what you posted here and my dmesg output looked similar to yours (seemingly confirming that the card was bound to pci-stub) but the problem persists regardless. I've tried ejecting the card from Windows, and although I think that it worked the first time around, I don't seem to be able to get it to work again (the VM still bugs out and the whole system freezes when the guest is reset, even if I eject the GPU). Any idea what I may be doing wrong? User "Dh0mp5eur" seems to be having the same problem.

Also a bit dubious as far as audio goes--can't seem to figure out how to get that to work.

Here's my output for _dmesg | grep pci-stub_
[ 0.546138] pci-stub: add 1002:6819 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.546142] pci-stub 0000:01:00.0: claimed by stub
[ 0.546143] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.546146] pci-stub 0000:01:00.1: claimed by stub

Thanks for your help

*Edit:*
Some updated findings:
With the above configuration (pci-stub configured as mentioned), if I run the command "modprobe -r radeon" after the first crash, start the VM again, wait for it to freeze, force shut down from virt-manager, and then start it again, it boots as it should without crashing the host. However, every time I do this, an "IRQ 52" line pops up in my dmesg | grep pci-stub output. This either hints to me that the Radeon driver isn't properly blacklisted, or it means that something else is going on and I just don't know enough to trouble-shoot it. Here's that output after killing and (successfully) restarting the VM four times in a row:

[ 0.547088] pci-stub: add 1002:6819 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.547092] pci-stub 0000:01:00.0: claimed by stub
[ 0.547093] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.547096] pci-stub 0000:01:00.1: claimed by stub
[ 244.854433] pci-stub 0000:01:00.0: enabling device (0140 -> 0143)
[ 357.362600] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X
[ 537.729194] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X
[ 626.116084] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X
[ 706.919188] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X


----------



## nbhusain

Quote:


> Originally Posted by *richardstallman*
> 
> Dropped back in to say that I actually got everything working great with KVM. However, I'm having the issue wherein I can not reset the Windows guest without it locking up upon the second boot. If I kill the guest during this frozen second boot and then start it again, it freezes the host. I've tried what you posted here and my dmesg output looked similar to yours (seemingly confirming that the card was bound to pci-stub) but the problem persists regardless. I've tried ejecting the card from Windows, and although I think that it worked the first time around, I don't seem to be able to get it to work again (the VM still bugs out and the whole system freezes when the guest is reset, even if I eject the GPU). Any idea what I may be doing wrong? User "Dh0mp5eur" seems to be having the same problem.
> 
> Also a bit dubious as far as audio goes--can't seem to figure out how to get that to work.
> 
> Here's my output for _dmesg | grep pci-stub_
> [ 0.546138] pci-stub: add 1002:6819 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [ 0.546142] pci-stub 0000:01:00.0: claimed by stub
> [ 0.546143] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [ 0.546146] pci-stub 0000:01:00.1: claimed by stub
> 
> Thanks for your help
> 
> *Edit:*
> Some updated findings:
> With the above configuration (pci-stub configured as mentioned), if I run the command "modprobe -r radeon" after the first crash, start the VM again, wait for it to freeze, force shut down from virt-manager, and then start it again, it boots as it should without crashing the host. However, every time I do this, an "IRQ 52" line pops up in my dmesg | grep pci-stub output. This either hints to me that the Radeon driver isn't properly blacklisted, or it means that something else is going on and I just don't know enough to trouble-shoot it. Here's that output after killing and (successfully) restarting the VM four times in a row:
> 
> [ 0.547088] pci-stub: add 1002:6819 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [ 0.547092] pci-stub 0000:01:00.0: claimed by stub
> [ 0.547093] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
> [ 0.547096] pci-stub 0000:01:00.1: claimed by stub
> [ 244.854433] pci-stub 0000:01:00.0: enabling device (0140 -> 0143)
> [ 357.362600] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X
> [ 537.729194] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X
> [ 626.116084] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X
> [ 706.919188] pci-stub 0000:01:00.0: irq 52 for MSI/MSI-X


Take a look at http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/580#post_19703549

Make sure the radeon module isnt on your initramfs.

Another thing you could try is loading your gpu rom by file, i remember to have similar problems but it got fixed by doing that


----------



## richardstallman

Quote:


> Originally Posted by *nbhusain*
> 
> Take a look at http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/580#post_19703549
> 
> Make sure the radeon module isnt on your initramfs.
> 
> Another thing you could try is loading your gpu rom by file, i remember to have similar problems but it got fixed by doing that


If I'm looking at the right post, this is actually the problem that I'm having--ejecting the card does not work for me most of the time. Also, assuming I do get even this working, is ejecting always necessary? I hear that with Xen's xm tools, it's not necessary to eject the card.
Quote:


> Originally Posted by *lloyd mcclendon*
> 
> thank you x10000
> 
> 
> 
> 
> 
> 
> 
> yeah, before shutting off the windows guest, eject the card, go to the vnc window to shutdown from there (or rdesktop or virsh shutdown). works everytime./quote]


----------



## nbhusain

Yeah this problem is present in kvm, and xen using the xl toolstack, using xm the card does reset correctly, you could try that, the only reason im not using xen right now is it doesnt support s3 resume and cool'n quiet on my cpu, also you could use these patches http://lists.xen.org/archives/html/xen-devel/2012-12/msg00009.html, to get primary passthrough working, and it does work great, at least on my box.

EDIT:
Another thing i found is, after shutting down your vm, and puting the host to sleep, seems to reset my card, and the performance issuses are gone.


----------



## myweb

Dear All,

Could you please help me to make working configuration:
Core i5 i5-3470
Asrock z77 pro4
ASUS GT640-1GD3-L
ASUS HD7750-DCSL-1GD5

Xen 4.2.1
Kubuntu 13.04 (Beta)
3.8.0-17-generic #27-Ubuntu SMP Sun Apr 7 19:39:35 UTC 2013 x86_64 x86_64 x86_64 GNU/Linu

The both videocards are working without Xen.
The both videocards are not working in DomU.
I tried DomU: Wondows 7, Windows 8, Kubuntu 13.04 (Beta)

I get BSOD during boot of ASUS HD7750-DCSL-1GD5 under Windows 7, 8 : atikmpag.sys
I tried to install only drivers (that is impossible to do with drivers from Ati website: windows could not find atikmpag.sys, I installed drivers from asus site)
Windows 8 has embedded driver.

I use xorg edgers ppa under ubuntu - exactly the same that works without Xen.

ATI_Kernel.txt 52k .txt file


ATI_Syslog.txt 69k .txt file


Ati_Xorg.log.txt 7k .txt file


Ati_Xorg.txt 4k .txt file


Nvidia_Kernel.log.txt 8k .txt file


Nvidia_Syslog.log.txt 8k .txt file


Nvidia_Xorg.conf.txt 8k .txt file


Nvidia_Xorg.log.txt 8k .txt file


NvidiaAti_ubuntu.cfg.txt 1k .txt file


NvidiaWin.cfg.txt 1k .txt file


----------



## lloyd mcclendon

Quote:


> Originally Posted by *myweb*
> 
> I get BSOD during boot of ASUS HD7750-DCSL-1GD5 under Windows 7, 8 : atikmpag.sys
> I tried to install only drivers (that is impossible to do with drivers from Ati website: windows could not find atikmpag.sys, I installed drivers from asus site)


Yeah, this happened to me at one point with the ASUS 7870. It seemed (?) to be from overclocking the card. I did that which worked well, but then after a reboot I was stuck with the same bluescreen as you. What was really strange was that even after I reinstalled windows (twice) it still bluescreened the same - which leads me to believe it was due to the bits on the card itself..

The fix in my case was to (reinstall I guess, may not be 100% necessary) but install the latest drivers from AMD's site - forget ASUS site as they were way back on v9, while AMD had 13 beta which has worked much better overall. Before rebooting with the drivers installed I did go into the OC section of CCC and set defaults.

The question was asked earlier "should I use KVM or XEN" - I would say if you're serious enough about this, you pretty much should try both out for a bit and see which works / fits you best.

With KVM and ejecting the card, the reboot issue is fixed (thank you again nbhusain







), but I'm now having a weird issue where about 80% of the time I cold boot the host, upon starting the windows VM just doesn't seem to get the card properly to turn the monitor on, and I have to destroy it and reboot the host ... then it works great :rollseyes:. If I could ever get this board to suspend properly it may be a non issue. That was kind of the point.

The XEN guide said it is not possible to suspend/save guests with PCI devices attached, I can imagine why, I'd guess qemu needs a ton of code to support that, but wow will it be sweet if that ever works.


----------



## nbhusain

Im able to suspend win7 x64 under kvm with -M q35, lastest vfio-vga + seabios patches (april 1) , gpu behind pcie root port, but it doesnt seem to solve my problem (after reboot the gpu clock gets down-clocked to 250mhz :S and doesnt wanna go up), BUT like i mentioned before suspending the host DOES seem to work, it looks like it gets reseted properly, maybe you guys could try that.

To get my host to suspend , i enabled s3 repost in my bios, and added acpi_sleep=s3_bios,s3_mode to my grub cfg file


----------



## myweb

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> Yeah, this happened to me at one point with the ASUS 7870. It seemed (?) to be from overclocking the card. I did that which worked well, but then after a reboot I was stuck with the same bluescreen as you. What was really strange was that even after I reinstalled windows (twice) it still bluescreened the same - which leads me to believe it was due to the bits on the card itself..
> 
> The fix in my case was to (reinstall I guess, may not be 100% necessary) but install the latest drivers from AMD's site - forget ASUS site as they were way back on v9, while AMD had 13 beta which has worked much better overall. Before rebooting with the drivers installed I did go into the OC section of CCC and set defaults.


It looks strange for me, because I did not overclock the card: I just bought new card and never get it working under Windows (guest OS)


----------



## nbhusain

This is how i got my VM working perfectly using kvm with qemu git + seabios patches using the new vfio-pci with vga support, kernel 3.9, doing a primary vga passthrough (no cirrus emulated card) on Archlinux, with a RADEON 6950.

EDIT: This post is outdated see my archlinux guide here: https://bbs.archlinux.org/viewtopic.php?id=162768

[/code]



Bonus: Vmware using svm, runing under kvm (with -cpu host,-hypervisor)


----------



## powerhouse

@nbhusain: Thanks for the kvm tutorial! I hope to be able to make use of it sometime soon.

Question: Can you use an existing Windows VM from Xen? (I think I already asked this before.) I would hate to re-install Windows.


----------



## nbhusain

Im pretty sure you can, just uninstall gplpv drivers and make sure you use a utility called glplv-uninstall.bat as admin( RUNNING THIS IS VERY IMPORTANT OTHERWISE U'LL GET A BSOD), also, u might need to setup the registry to load the ide,achi or scsi drivers take a look at http://www.ocztechnologyforum.com/forum/showthread.php?57789-How-to-enable-Ide-Ahci-n-m-raid-mode-without-reinstalling&s=c04b19e98dbbb716ef3192bb40b60f15


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> Im pretty sure you can, just uninstall gplpv drivers and make sure you use a utility called glplv-uninstall.bat as admin( RUNNING THIS IS VERY IMPORTANT OTHERWISE U'LL GET A BSOD), also, u might need to setup the registry to load the ide,achi or scsi drivers take a look at http://www.ocztechnologyforum.com/forum/showthread.php?57789-How-to-enable-Ide-Ahci-n-m-raid-mode-without-reinstalling&s=c04b19e98dbbb716ef3192bb40b60f15


Thanks, great info. I totally forgot about the GPLPV drivers.


----------



## nbhusain

Good news! i just got a Geforce 470 GTX from my brother and it works using the guide i posted, reboot/shutdown works fine, the only game i tested was bioshock infinite and it was working great @ max settings.

This is how i started qemu:

Code:



Code:


qemu-system-x86_64 -m 8192 -M q35 -mem-path /dev/hugepages/ -enable-kvm 
-cpu host -name hvm -vga none -nographic 
-bios /usr/local/share/qemu/seabios/bios.bin 
-smp 6,sockets=1,cores=6,threads=1 
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=pcie.1
-device vfio-pci,host=00:11.0,bus=pcie.0,addr=1f.4
-device vfio-pci,host=07:00.0,bus=pcie.1,multifunction=on,addr=00.0,x-vga=on
-device vfio-pci,host=07:00.1,bus=pcie.1,addr=00.1
-device vfio-pci,host=04:00.0,bus=pcie.0
-device vfio-pci,host=05:00.0,bus=pcie.0
-device vfio-pci,host=06:00.0,bus=pcie.0
-netdev bridge,br=br0,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0

Results:


----------



## nbhusain

outdated


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> ... EDIT2: Archlinux guide https://bbs.archlinux.org/viewtopic.php?id=162768


Thanks for all the information and the link to your Archlinux guide. This is awesome!

Since you are familiar with both Xen and kvm, do you see advantages in moving to kvm? So far Xen has been good to me, except the latest update (Xen hypervisor 4.1.3-3ubuntu1.5) which prevented my Windows guest from loading. I had to roll back.

Anyway, I'm interested in your thoughts.


----------



## nbhusain

First of all i want to say that performance with xen was never an issue, it works great for me, it feels almost native, but so does kvm.
I moved away from xen initially because, like i mentioned in previous posts, it doesnt support ( as of today ) s3 resume ( on my machine at least ), and using
cool'n quiet crash my host, i cant even get to X before it crashes, i never had any issue using kvm (KVM IS LINUX!)
Second, kvm/qemu is moving way faster than xen/qemu, xen is still using qemu 0.12 i believe ( xen 4.3 supports upstream qemu but its buggy as hell), which is like 4 years old, so its missing a lot of features from upstream (GL USING AUDIO PASSTHROUGH ON WIN7 X64), and xen supposed "vga-passthrough support" involves tons of hacks that have never been merged in like 3-4 years.

Intitially what i wanted to achieve was to "partition" my machine ( each one independent as possible from the other ) using 2 cards, so for example
if wanted to say play on my windows vm and build a kernel on my dom0 using make -j128 (lol) the host wouldnt slow down the vm, or the other way arround, so i though hell why dont i passthrough my sata controller to the vm, dedicate 6 of my 8 cores to it ( and leave 2 for the host + 4GB of ram ). Well unfortunately i could never get rombios from xen to boot from it, i also tried to force it to load the optiomrom for the controller ( from a file ), and it did load but the boot table got corrupted and it wouldnt boot anything, then i found out i could use a edk2 duet image to recognize my controller on ahci mode, and boot windows 7 x64 uefi, and yeah that worked, but power management and audio passthrough still didnt work.

So then i found kvm, and it could do eveything everything xen did + more ( seabios does boot my sata controller, audio passthrough works w00t ) but PRIMARY vga passthrough didnt work, until a month ago, when i found a message on qemu-devel implementing this upstream ( without 3'rd party unmantained hacks ) so im happy with it and its working great.

EDIT: i'd like to add that gplpv drivers are horrible! HORRRIBLEEEEEEEEEE OMG SRSLY THEY MAKE ME CRY, never had a problem with virtio drivers though.


----------



## nbhusain

I found another blog doing using the same method, apparently this guy managed to passthrough a geforce 660 but didnt succeed with fermi cards, which is weird cause it did work on my 470 gtx, anyway here's the link http://www.frankdtech.org/?p=6


----------



## powerhouse

@nbhusain: Thanks for your detailed answer. Yes, the lack of S3 support and power management when using AMD CPUs looks like a good enough reason to switch to kvm.

It's a real pity that a year ago the kvm documentation was so much behind (I hope for the sake of kvm that it improved), else I would have definitely started with kvm instead of Xen. But once I have a little more time I will give kvm a try. You definitely raised my curiosity!

It looks like kvm has better support for Nvidia cards?

About the link you posted (re Nvidia passthrough with kvm): It seems that Frank has some issues with the video performance under kvm, at least with certain games and benchmarks. Is this performance inconsistency something one has to expect with kvm, or just an exception to the rule?

One thing you may have missed about kvm: Nvidia doesn't support Xen (at least not that I'm aware of), so if you want to use a Nvidia card for dom0 you'd have to use the nouveau (or nv) driver. I don't think this is an issue with kvm, which gives kvm a clear advantage with regard to graphics adapters support.

Edit: What's holding me back from trying kvm is that I couldn't use a standard, off the shelf Linux distro like Ubuntu or Linux Mint, but that I would have to compile my kernel etc. I'm using Xen on a production machine so it needs to be solid as a rock. kvm seems to be still a little bleeding edge.


----------



## nbhusain

Well, i havent had any performance issues using my radeon card, the only test i ran on the geforce 470 was bioshock infinite and it was working fine @ max settings remember this is still WIP so i expect most issues being ironed out since its actually being developed and mantained.

Thats why i switched from gentoo-->opensuse-->arch, you can still have that that out of the box experience but at the same time you can easly create/use bleeding edge packages and try some of the lastest features if you want. Using a distro like ubuntu you could always take a source deb, modify it, put a couple of patches or whatever, you could even create your own local repository if you wanted to, anyway remember most of this stuff is already upstream, kernel 3.9 has been released, and the work on qemu should be ready by v1.5 http://wiki.qemu.org/ChangeLog/1.5, so it should ship on ubuntu some day xD.


----------



## Rbby258

If i set this up on my pc as the host then i can use my laptop to play games right?
Will this work as long as my host pc is on, could i take my laptop to my friends and play games on the host?
Last of all do the games install on the laptop hard drive or the host hard drive?

Also what are the minimum specs you guys recommend for the guest computer.


----------



## nbhusain

Nope, what we're doing here is passing through a physical gpu to a virtual machine, to play games on that virtual machine locally, if you wanna play games on your laptop, while rendering on your host, you need another solution, like RemoteFX.


----------



## Rbby258

Quote:


> Originally Posted by *nbhusain*
> 
> Nope, what we're doing here is passing through a physical gpu to a virtual machine, to play games on that virtual machine locally.


Whats the point in that? Or am i missing something. Im a VM noob.


----------



## nbhusain

Well the advantage for us linux users is that we dont have to dual boot anymore, we can fire up windows or mac(with some patches) and play games while doing whatever we wanna do on our linux host,like also gaming or watcha movie, also if you have like 4 gpus you could in theory create 4 virtual machines each one with its monitor, keyboard, mouse, gpu, etc and play games on those 4 vm's at the same time xD


----------



## Rbby258

Quote:


> Originally Posted by *nbhusain*
> 
> Well the advantage for us linux users is that we dont have to dual boot anymore, we can fire up windows or mac(with some patches) and play games while doing whatever we wanna do on our linux host,like also gaming or watcha movie, also if you have like 4 gpus you could in theory create 4 virtual machines each one with its monitor, keyboard, mouse, gpu, etc and play games on those 4 vm's at the same time xD


i see thanks for the info and letting me know about remotefx im now going that route


----------



## Pip Boy

Quote:


> Originally Posted by *Rbby258*
> 
> Whats the point in that? Or am i missing something. Im a VM noob.


The point is, a lot of people are almost 100% linux households. Routers, firewalls, laptops, pc's the last thing my particular Linux machine cant do is play dx10/11 games like Skyrim. I already managed to get DVDfab working with a VM on win and a virtual DVD drive and mp3tag was no problem for wine, so litterally just PC games for me that use the Direct X -API

I hope one day that it becomes easier for the noob to get an out of box VM solution (like virtualbox) where pass through is nice and simple to play win7 / 8 machines and games with a nice step by step wizard









going back to win now just feels like stepping down in productivity and capability, it also looks so hideous


----------



## tritron

Does anyone have patches for xen 4.2 website with patches is down and out


----------



## richardstallman

So I got this working well with vanilla KVM using the latest (at the time) kernel which was 3.9.1, but my performance is so-so. In Heaven Bench 3.0, I get almost perfect performance similar to native (within 99% similar), but with certain games, e.g. Killing Floor, I get massive FPS dips that shouldn't occur with such an old game. (Natively, my GPU runs Killing Floor at anywhere from 30 to 700 FPS, probably around 200 average, whereas in my Windows virtual machine I'm seeing it drop to 30 quite often and it only goes up to the mid 100's or so.)

I don't need 100% native performance, but being unable to run a game based on an engine which is half a decade old by now is really too much for me to look passed for this kind of setup. Has anyone else had similar issues, and/or does anyone else know how I might troubleshoot what would cause this (if this is indeed fixable)? I assume that it is indeed something that I can correct, since Heaven 3.0 is so similar to my native result.

Running Debian Wheezy and kernel 3.9.1, vanilla Windows 8 with catalyst 13.4 loaded from a virtual disk image. GPU is a Radeon 7850.

Thanks guys


----------



## DEWE

Would it be possible to virtualize windows 2012 and run a Windows virtualization server from inside?


----------



## KeltecRFB

Quote:


> Originally Posted by *DEWE*
> 
> Would it be possible to virtualize windows 2012 and run a Windows virtualization server from inside?


Why would you want to? It sounds convoluted and a waste of resources. Just virtualize each of the OSes through XEN.


----------



## kcslefty

I got another question and ill hope you can answer it.
When you passthrough a pci device (in this example the vga card) it is exclusiv for one vm.
But if you are not running this vm what happens to the vga card?
For what i understand you cannot use it in your host system. Does it mean it is completly unused by the computer and in terms of power?
So for example host systems running with onboard xeon igpu with (just an example) 60 watt and when you start the vm with vga passthrough and playing a game with about 200 watt? So you got your host system and the other vm´s like a power saving mode cause vga is not used?

Thx to all


----------



## lloyd mcclendon

pretty much correct

If you pass the extra card to pci_stub on boot, the fglrx or nvidia module won't load to use it and it won't pull any [much] power from the wall.

If you have a different primary GPU, you're also free to use the second card in your host system, then stop using it, unload the module, reset it [dicey, depends on the card], pass it to pci_stub and send it to your starting guest VM.







rinse repeat


----------



## myweb

Dear All,
Could you please help me to solve host hang during VM starting uinder KVM/Qemu 1.5.
I use:
Core i5 i5-3470
Asrock z77 pro4
ASUS GT640-1GD3-L
ASUS HD7750-DCSL-1GD5
Kubuntu 13.10 (Kernel 3.9.5)
Primary videocard is ASUS HD7750-DCSL-1GD5 and I am trying to pass it to VM

Host system boots to KDE, then I shut down lightdm (X server), unbind ASUS HD7750-DCSL-1GD5 and bind it to vfio. When I start VM host system hangs.
Quote:


> Originally Posted by *lloyd mcclendon*
> 
> stop using it, *unload the module, reset it* [dicey, depends on the card], pass it to pci_stub and send it to your starting guest VM


Could you please clarify what do you mean saying unload the module? (is it correct: echo $dev > /sys/bus/pci/devices/$dev/driver/unbind ?)
and how to reset the videocard (for example ATI HD 7750)?

Thank you!


----------



## eXXon

Sub'd for later use & + rep for the great guide.


----------



## nbhusain

Quote:


> Originally Posted by *myweb*
> 
> Dear All,
> Could you please help me to solve host hang during VM starting uinder KVM/Qemu 1.5.
> I use:
> Core i5 i5-3470
> Asrock z77 pro4
> ASUS GT640-1GD3-L
> ASUS HD7750-DCSL-1GD5
> Kubuntu 13.10 (Kernel 3.9.5)
> Primary videocard is ASUS HD7750-DCSL-1GD5 and I am trying to pass it to VM
> 
> Host system boots to KDE, then I shut down lightdm (X server), unbind ASUS HD7750-DCSL-1GD5 and bind it to vfio. When I start VM host system hangs.
> Could you please clarify what do you mean saying unload the module? (is it correct: echo $dev > /sys/bus/pci/devices/$dev/driver/unbind ?)
> and how to reset the videocard (for example ATI HD 7750)?
> 
> Thank you!


You'll need to blacklist the radeon module, or bind it at boot to pci-stub, also, passing through a radeon card in kvm can result in the host crashing after a reboot/shutdown, fortunately theres a couple of solutions you can use, the best one is patching qemu and the kernel, the other one is ejecting the card before rebooting/shutting down, you can find the patches, the eject methodand some tips here: https://bbs.archlinux.org/viewtopic.php?id=162768


----------



## lloyd mcclendon

Quote:


> Originally Posted by *myweb*
> 
> Could you please clarify what do you mean saying unload the module? (is it correct: echo $dev > /sys/bus/pci/devices/$dev/driver/unbind ?)
> and how to reset the videocard (for example ATI HD 7750)?


If you startx, (and your X setup is correct with GL working using the fglrx or radeon module), just by existing X, it does not unload the module and free the card. You have to exit X, and then rmmod [fglrx or radeon] and then you have to unbind the card, and bind it to pci_stub. If you go back a number of pages and read all of nbusains posts, he has already listed out how to accomplish this. Everything he's said is correct and should be read.









The commands to bind/unbind are also on the kvm wiki for vga / pci device assignment.

Recommended setup IMO is to use an nvidia card for your linux host, keep X rolling, and send your second ATI card to pci_stub on boot and just use it for the guest. KVM switch, dual monitors and synergyc







It would be a lot easier and better to use, you can pick up a gtx295 or something for cheap.

If only I could ever get my host to resume from sleep at all with this board







otherwise everything is good, the guest behaves correctly. The guest VGA drivers seem to have a lot to do with it, 9.0 was terrible, 13.0 was better, 13.4 seems pretty happy.


----------



## myweb

I am still not succeed with VGA passthrough (ATI HD 7750). Please find details bellow:

SeaBIOS 1.7.2- 20130528_111335-allspice
Linux desktop 3.9.0-6-generic #13-Ubuntu SMP Fri Jun 14 15:47:09 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.9.0-6-generic root=UUID=ce664220-6cd8-4d42-98aa-9460387f2904 ro intel_iommu=on radeon.blacklist=1 quiet splash vt.handoff=7

1. iommu_group shows a lot of devices - is it correct?

Code:



Code:


ls /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices
0000:00:01.0  0000:01:00.0  0000:01:00.1

Code:



Code:


ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices
0000:00:1c.0  0000:00:1c.4  0000:00:1c.5  0000:00:1c.7  0000:02:00.0  0000:02:00.1  0000:03:00.0  0000:04:01.0  0000:05:00.0  0000:06:00.0

Code:



Code:


lspci -nn
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
00:02.0 Display controller [0380]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0152] (rev 09)
00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
00:1c.4 PCI bridge [0604]: Intel Corporation 82801 PCI Bridge [8086:244e] (rev c4)
00:1c.5 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 [8086:1e1a] (rev c4)
00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750] [1002:683f]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK107GL [GRID K1] [10de:0ff2] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GK107 HDMI Audio Controller [10de:0e1b] (rev a1)
03:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 03)
04:01.0 Multimedia controller [0480]: Philips Semiconductors SAA7146 [1131:7146] (rev 01)
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller [10ec:8168] (rev 06)
06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)

2. "-vga none" does not work for me

3. "-vga std"

*Primary videocard is onboard (core i5) Intel:*
Windows VM does not work with VGA passthrough
Ubuntu Live CD shows dialog with option "Try" and "Install" *via VGA passthrough*, but if choose any options - screen goes to black
Installed Ubuntu - black screen

*Primary videocard is ATI HD 7750*
I get the following warning when VM starts:

Code:



Code:


-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported

And a lot of the following messages when VM started:

Code:



Code:


...
system-x86_64: vfio_bar_write(,0x13ff50, 0x36703067,4) failed: Device or resource busy
system-x86_64: vfio_bar_write(,0x13ff54, 0x0,4) failed: Device or resource busy
...

and black screen

Code:



Code:


sudo qemu-system-x86_64 --enable-kvm -machine q35 -m 2048 -cpu host \
-smp 2,sockets=2,cores=1,threads=1  \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/var/lib/libvirt/images/Win7.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/media/CopiedData/Soft/saucy-desktop-amd64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd -boot c \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-usb \
-device usb-host,hostbus=1,hostaddr=6 \
-vga std

cat /var/log/syslog | grep -e DMAR -e IOMMU

Code:



Code:


Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.000000] ACPI: DMAR 00000000a99e90d0 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.000000] Intel-IOMMU: enabled
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.045465] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.045471] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.045542] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.487718] DMAR: No ATSR found
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.487755] IOMMU 0 0xfed90000: using Queued invalidation
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.487756] IOMMU 1 0xfed91000: using Queued invalidation
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.487758] IOMMU: Setting RMRR:
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.487766] IOMMU: Setting identity map for device 0000:00:02.0 [0xab800000 - 0xaf9fffff]
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.488064] IOMMU: Setting identity map for device 0000:00:1d.0 [0xa95e6000 - 0xa9617fff]
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.488084] IOMMU: Setting identity map for device 0000:00:1a.0 [0xa95e6000 - 0xa9617fff]
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.488101] IOMMU: Setting identity map for device 0000:00:14.0 [0xa95e6000 - 0xa9617fff]
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.488113] IOMMU: Prepare 0-16MiB unity mapping for LPC
Jun 15 22:34:35 abrilevskiy-desktop kernel: [    0.488119] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

cat /boot/config-3.9.0-6-generic | grep -i vfio

Code:



Code:


CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y


----------



## nbhusain

Quote:


> Originally Posted by *myweb*
> 
> 2. "-vga none" does not work for me


Stock seabios wont work you'll need a patched one:

Code:



Code:


git clone git://git.seabios.org/seabios.git -b 1.7.2-stable
cd seabios
unzip path/to/pciinit_enable default_vga_device.patch.zip
patch -p1 <pciinit_enable\ default_vga_device.patch 
make
cp out/bios.bin /path/to/somewhere/bios.bin

Quote:


> Originally Posted by *myweb*
> 
> 1. iommu_group shows a lot of devices - is it correct?
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> ls /sys/bus/pci/devices/0000\:01\:00.0/iommu_group/devices
> 0000:00:01.0  0000:01:00.0  0000:01:00.1
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices
> 0000:00:1c.0  0000:00:1c.4  0000:00:1c.5  0000:00:1c.7  0000:02:00.0  0000:02:00.1  0000:03:00.0  0000:04:01.0  0000:05:00.0  0000:06:00.0


No that doesnt look ok, this might be the solution: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

It seems there are some problems using the IGD as the primary vga boot device, see: http://qemu.11.n7.nabble.com/VFIO-VGA-test-branches-td201483.html#a202782

Also it seems nvidia drivers have a bug, that prevents vga passthrough from working even if the device you want to passthrough is a radeon, theres a patch for it, its at the end of this post: http://lists.gnu.org/archive/html/qemu-devel/2013-05/msg04300.html

pciinit_enable default_vga_device.patch.zip 1k .zip file


----------



## myweb

Quote:


> Originally Posted by *nbhusain*
> 
> Stock seabios wont work you'll need a patched one:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> git clone git://git.seabios.org/seabios.git -b 1.7.2-stable
> cd seabios
> unzip path/to/pciinit_enable default_vga_device.patch.zip
> patch -p1 <pciinit_enable\ default_vga_device.patch
> make
> cp out/bios.bin /path/to/somewhere/bios.bin


I have tried, but it does not help. Also I tried 3.9.6 and 3.9.10 kernel - the same situation.
In order to avoid NVIDIA driver issue I have removed the NVIDIA driver.
After all I can not get GUI Ubuntu Live dialog (Try and install button) on passthrough videocard - I get text output saying kernel fail: radeon drm module.
I am using QEMU from ubuntu repo: QEMU emulator version 1.5.0

Could you please help me to chose a right direction:
1. Stay on Ubuntu
1.1 patch 3.9.0-6 Kernel in order to solve IGD issue, iommu_group
1.2 patch mainline 3.9.6 kernel
1.3 patch 3.9.10-rc6 kernel
1.11 1.1 + compile Qemu from GIT
1.12 1.2 + compile Qemu from GIT
1.13 1.3 + compile Qemu from GIT
2. Move to Arch Linux

Thank you!

UPDATE: I get Ubuntu GUI dialog on passthrough videocard if do NOT bind to vfio the following device (device added automatically from iommu_group):
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)


----------



## nbhusain

Quote:


> Originally Posted by *myweb*
> 
> I have tried, but it does not help. Also I tried 3.9.6 and 3.9.10 kernel - the same situation.
> In order to avoid NVIDIA driver issue I have removed the NVIDIA driver.
> After all I can not get GUI Ubuntu Live dialog (Try and install button) on passthrough videocard - I get text output saying kernel fail: radeon drm module.
> I am using QEMU from ubuntu repo: QEMU emulator version 1.5.0
> 
> Could you please help me to chose a right direction:
> 1. Stay on Ubuntu
> 1.1 patch 3.9.0-6 Kernel in order to solve IGD issue, iommu_group
> 1.2 patch mainline 3.9.6 kernel
> 1.3 patch 3.9.10-rc6 kernel
> 1.11 1.1 + compile Qemu from GIT
> 1.12 1.2 + compile Qemu from GIT
> 1.13 1.3 + compile Qemu from GIT
> 2. Move to Arch Linux
> 
> Thank you!
> 
> UPDATE: I get Ubuntu GUI dialog on passthrough videocard if do NOT bind to vfio the following device (device added automatically from iommu_group):
> 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)


Yes you're not supposed to bind the root port.

Moving to arch would be the best bet, you dont have to worry about breaking your system from custom built packages, and custom packages on arch are super easy to maintain


----------



## myweb

Quote:


> Originally Posted by *nbhusain*
> 
> custom packages on arch are super easy to maintain


Could you please help me with some kind of guide: custom packages on arch?

I have compiled Qemu from git, but I have not installed just run qemu-system-x86_64 from x86_64-softmmu folder. But result is the same: GUI dialog from Live Ubuntu: try or install; and then black screen.


----------



## myweb

I have installed Arch linux, but when I run qemu I get the followin error:

Code:



Code:


No protocol specified

** (qemu-system-x86_64:2451): WARNING **: Could not open X display                                                                                                                                                                                                      
No protocol specified

(qemu-system-x86_64:2451): Gtk-WARNING **: cannot open display: :0

Could you pease help to solve the issue?


----------



## nbhusain

Quote:


> Originally Posted by *myweb*
> 
> Could you please help me with some kind of guide: custom packages on arch?
> 
> I have compiled Qemu from git, but I have not installed just run qemu-system-x86_64 from x86_64-softmmu folder. But result is the same: GUI dialog from Live Ubuntu: try or install; and then black screen.


When building from source, in my opinion, its better to create or modify your own custom packages, so you can handle dependencies better.

Have a look at this:
https://wiki.archlinux.org/index.php/Makepkg
https://wiki.archlinux.org/index.php/Arch_Build_System
https://wiki.archlinux.org/index.php/Arch_User_Repository

Most packages contain a file called PKGBUILD and some extra files/patches, PKGBUILD contains the instruction to build the package from source, the packages i use are from ABS and AUR but modified to include some patches

here they are:

linux-mainline.zip 129k .zip file


qemu.zip 10k .zip file


seabios.zip 2k .zip file


For example to build qemu and install the needed dependencies:

Code:



Code:


unzip qemu.zip
cd qemu
makepkg -s
sudo pacman -U qemu-1.5.0-1-x86_64.pkg.tar.xz

Quote:


> Originally Posted by *myweb*
> 
> I have installed Arch linux, but when I run qemu I get the followin error:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> No protocol specified
> 
> ** (qemu-system-x86_64:2451): WARNING **: Could not open X display
> No protocol specified
> 
> (qemu-system-x86_64:2451): Gtk-WARNING **: cannot open display: :0
> 
> Could you pease help to solve the issue?


You'll need to give permission to access the server you can do that as a regular user like this:

Code:



Code:


xhost +

you can also modify /etc/sudoers to grant permissions see: https://wiki.archlinux.org/index.php/Sudo#Run_X11_apps_using_sudo

then launch qemu like this:

Code:



Code:


sudo qemu-system-x86_64 blah blah blah


----------



## lloyd mcclendon

Quote:


> Originally Posted by *myweb*
> 
> (qemu-system-x86_64:2451): Gtk-WARNING **: cannot open display: :0


Whatever you're bringing up in qemu requires X to be running and you haven't started it.

OR are you using SDL? When I used to do that I'd see that, it's just X permissions. Your qemu process is starting as a user (likely root) that can not access your account's X session. The preferred way is to have qemu start under a system account (useradd -r) and grant access to ~/.Xauthority with setfacl.

http://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#SDL_Graphics

English is not your primary language I take it?


----------



## myweb

Quote:


> Originally Posted by *nbhusain*
> 
> When building from source, in my opinion, its better to create or modify your own custom packages, so you can handle dependencies better.
> 
> Have a look at this:
> https://wiki.archlinux.org/index.php/Makepkg
> https://wiki.archlinux.org/index.php/Arch_Build_System
> https://wiki.archlinux.org/index.php/Arch_User_Repository
> 
> Most packages contain a file called PKGBUILD and some extra files/patches, PKGBUILD contains the instruction to build the package from source, the packages i use are from ABS and AUR but modified to include some patches
> 
> here they are:
> 
> linux-mainline.zip 129k .zip file
> 
> 
> qemu.zip 10k .zip file
> 
> 
> seabios.zip 2k .zip file
> 
> 
> For example to build qemu and install the needed dependencies:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> unzip qemu.zip
> cd qemu
> makepkg -s
> sudo pacman -U qemu-1.5.0-1-x86_64.pkg.tar.xz
> 
> You'll need to give permission to access the server you can do that as a regular user like this:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> xhost +
> 
> you can also modify /etc/sudoers to grant permissions see: https://wiki.archlinux.org/index.php/Sudo#Run_X11_apps_using_sudo
> 
> then launch qemu like this:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> sudo qemu-system-x86_64 blah blah blah


Thank you very much for detailed explanation. I have successfully compiled qemu, linux kernel, seabios and also fix "Gtk-WARNING **: cannot open display: :0" (xhost+ and sudoers)
Sad news - I am still can't get Seabios messages on passthrough videocard.

Also, I have faced with little issues: there is "kernel-vfio-vga-reset.patch" but "vfio-vga-reset.patch" is specified in PKGBUILD and also a lot of errors like: "include/fdt.h error: unknown type name 'fdt64_t'"/ "include/fdt.h error: unknown type name 'fdt320_t'", but I found solution for qemu 1.4/
Quote:


> Originally Posted by *lloyd mcclendon*
> 
> English is not your primary language I take it?


Yes, You are right I am not English native/primary speaker. How did you recognize that?







I am sorry for bad English


----------



## nbhusain

Quote:


> Originally Posted by *myweb*
> 
> Thank you very much for detailed explanation. I have successfully compiled qemu, linux kernel, seabios and also fix "Gtk-WARNING **: cannot open display: :0" (xhost+ and sudoers)
> Sad news - I am still can't get Seabios messages on passthrough videocard.
> 
> Also, I have faced with little issues: there is "kernel-vfio-vga-reset.patch" but "vfio-vga-reset.patch" is specified in PKGBUILD and also a lot of errors like: "include/fdt.h error: unknown type name 'fdt64_t'"/ "include/fdt.h error: unknown type name 'fdt320_t'", but I found solution for qemu 1.4/


Yes i think i might have renamed the patch by mistake sorry about that.

Which card is your primary card and which one are you trying to passthrough, remember you'll need to passthrough your secondary card. You can also try this kernel patch: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

If you cant get it to work with vfio-pci, you might want to use pci-assign instead take a look at: https://bbs.archlinux.org/viewtopic.php?pid=1274453#p1274453

That will passthrough the card as a secondary device assuming you're passing through a radeon card, for nvidia you'll need to use vfio.
Also you might experience host crashes using pci-assign, you can solve this by using the eject method i detailed in my arch guide.


----------



## Phobia




----------



## myweb

Quote:


> Originally Posted by *nbhusain*
> 
> Yes i think i might have renamed the patch by mistake sorry about that.
> 
> Which card is your primary card and which one are you trying to passthrough, remember you'll need to passthrough your secondary card. You can also try this kernel patch: http://marc.info/?l=linux-kernel&m=136993923507507&w=2
> 
> If you cant get it to work with vfio-pci, you might want to use pci-assign instead take a look at: https://bbs.archlinux.org/viewtopic.php?pid=1274453#p1274453
> 
> That will passthrough the card as a secondary device assuming you're passing through a radeon card, for nvidia you'll need to use vfio.
> Also you might experience host crashes using pci-assign, you can solve this by using the eject method i detailed in my arch guide.


My primary card is onboard intel (CPU integrated)
PCI-E One: ATI HD 7750
PCI-E Two: Nvidia 640 modded to Grid K1

I would like to passthrough both cards (Nvidia to Linux VM and ATI to Windows or Vice versa), but I started with ATI HD 7750 passthrough.
Also, I compiled Qemu, SeaBios, Linux kernel in parallel being booted to stock kernel, is it ok? Or I should compile Linux kernel at first, install its headers, boot and then compile Qemu and Seabios?


----------



## nbhusain

Right now using the IGP as the primary device doesnt seem to work using vfio from the reports i've seen, you should disable it.
Once you build the kernel you'll see 2 files, the kernel and the headers to install both do this:

Code:



Code:


sudo pacman -U linux-mainline-*

Then you'll need to upgrade grub:

Code:



Code:


sudo grub-mkconfig -o /boot/grub/grub.cfg

Attached here is the kernel package, updated to 3.9.6 with this patch applied: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

linux-mainline.zip 131k .zip file


You should read the patch documentation, and update your /etc/default/grub accordingly (using the pcie_acs_override= boot option), and also update grub.cfg with the above command, after you built and installed everything you must reboot into the new kernel.

If none of the above work you might want to use pci-assign like i mentioned before


----------



## Masta Squidge

So, what is stopping someone from taking a computer with say... a 3770K and two titans from hooking up two sets of KVM and running a virtual lan... All you need is two copies of the same game for the sake of authenticating the installations correct?

Voila, two person gaming rig with knockout performance. Not many games are going to take advantage of more than a few logical cores anyways correct? I realize a HT core isn't the same as hardware, but can you assign half the cpu to one, half to the other, and only have to deal with a slight performance hit from the primary OS to both VMs, or do you need to specifically leave more available?

Or even better, run two titans, a third cheap video card for the primary OS (and a third monitor for keeping an eye on it), and nobody has to give up any video performance right?


----------



## aznpersuazn

Quote:


> Originally Posted by *Masta Squidge*
> 
> So, what is stopping someone from taking a computer with say... a 3770K and two titans from hooking up two sets of KVM and running a virtual lan... All you need is two copies of the same game for the sake of authenticating the installations correct?
> 
> Voila, two person gaming rig with knockout performance. Not many games are going to take advantage of more than a few logical cores anyways correct? I realize a HT core isn't the same as hardware, but can you assign half the cpu to one, half to the other, and only have to deal with a slight performance hit from the primary OS to both VMs, or do you need to specifically leave more available?
> 
> Or even better, run two titans, a third cheap video card for the primary OS (and a third monitor for keeping an eye on it), and nobody has to give up any video performance right?


I have always wondered about VM use such as this. Anyone have more information regarding this setup?


----------



## Masta Squidge

Or, you know, run 780s or quad 7970s or whatever gets you wet. I wonder if you can run two different sli/xfire setups like this... Probably not, which is why I suggest going for the best single card (can be a 690 or 7990 I suppose) solution in this case.

Since I know someone will complain about my Titan statement. What you use for monitors is your call.


----------



## nbhusain

Yes i believe its possible, right now i "reserve" 2 cores, 4gb of ram to the host and a cheap radeon 5450, and my windows 8 guest runs on 6 cores, 8gb of ram and a radeon 6950, if i had another GPU i could easly create 2 windows gaming VM's, once i get some money i plan to get another GPU and more ram to do this.


----------



## Masta Squidge

Well, with intel right now it would have to be maybe 1 core plus its HT brother, but that doesn't leave an even amount for two VMs. A 3770k might have 8 logical threads, but only 4 cores. Or perhaps assign the two HT threads to the OS, and two primary cores + 1 HT thread to each vm?

Is it possible to assign half the cpu to each VM while the OS just "takes what it needs"?


----------



## nbhusain

Quote:


> Originally Posted by *Masta Squidge*
> 
> Well, with intel right now it would have to be maybe 1 core plus its HT brother, but that doesn't leave an even amount for two VMs. A 3770k ight have 8 logical threads, but only 4 cores. Or perhaps assign the two HT threads to the OS, and two primary cores + 1 HT thread to each vm?
> 
> Is it possible to assign half the cpu to each VM while the OS just "takes what it needs"?


Yeah, using linux cgroups (if you're using kvm) you can assign a share or percentage of the cpu resources to an application, or you can just give more priority to the VM thread


----------



## Masta Squidge

Quote:


> Originally Posted by *nbhusain*
> 
> Yeah, using linux cgroups (if you're using kvm) you can assign a share or percentage of the cpu resources to an application


That would be pretty win.

Ivy Bridge dual core + one extra HT, plus a Titan and say.... 8GB of ram per VM. Would make for a pretty monster pair of gaming rigs. Wouldn't be doing very much video recording on either one, but I think that is enough power to run pretty much every game there is.

Would likely have to give each one its own HDD though. In the case of games that require login through a server however, you only need one installation. Examples being everything from Blizzard.

This is sweet, because you can plop it into a ramdisk. I seriously doubt any game would have any trouble trying to share read/write access between two clients from a ramdisk. In my case, I would give each OS 4gb of ram, reserve maybe 2gb to the host os, and the remaining 22GB should be enough for almost every game aside from wow. In fact, I think even wow can fit in there with some creative methods.

If not playing wow, which I wouldn't be, I would give each OS 6 gigs, plus the 2 to the host, leaving 18GB for whatever else, which afaik right now includes every game I own - again, assuming you can run from a single installation.

Or you know, use an SSD. But I don't have one, and I will be getting another 16GB of ram for sure either way.

Eve online, all blizzard games, rift, gw, gw2, basically all MMOs. All these require authentication from the server, not from the installation, meaning you only need one install.


----------



## nbhusain

Quote:


> Originally Posted by *Masta Squidge*
> 
> That would be pretty win.
> 
> Ivy Bridge dual core + one extra HT, plus a Titan and say.... 8GB of ram per VM. Would make for a pretty monster pair of gaming rigs. Wouldn't be doing very much video recording on either one, but I think that is enough power to run pretty much every game there is.
> 
> Would likely have to give each one its own HDD though. In the case of games that require login through a server however, you only need one installation. Examples being everything from Blizzard.
> 
> This is sweet, because you can plop it into a ramdisk. I seriously doubt any game would have any trouble trying to share read/write access between two clients from a ramdisk. In my case, I would give each OS 4gb of ram, reserve maybe 2gb to the host os, and the remaining 22GB should be enough for almost every game aside from wow. In fact, I think even wow can fit in there with some creative methods.
> 
> If not playing wow, which I wouldn't be, I would give each OS 6 gigs, plus the 2 to the host, leaving 18GB for whatever else, which afaik right now includes every game I own - again, assuming you can run from a single installation.
> 
> Or you know, use an SSD. But I don't have one, and I will be getting another 16GB of ram for sure either way.
> 
> Eve online, all blizzard games, rift, gw, gw2, basically all MMOs. All these require authentication from the server, not from the installation, meaning you only need one install.


It might need only one installation, but i think you'll need an account for each vm, since each machine will be its own pc


----------



## Masta Squidge

Well yeah, I went over that already. You can't log in to the same account twice.


----------



## fusioner

Hi.First of all i want to thank you for this great thread.

I am using Ubuntu 13.04 x64 with xen 4.2.1 and virt-manager 0.9.4-2. i am getting the following error when trying to start Windows 8 x64 VM with PCI 0000:01.00.0 and 0000:01.00.0 added as new hardware in virt-manager :

Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device

command: xm pci-list-assignable-devices
returns:
Error: pciback/pci-stub not loaded?

command: lspci | grep 01:00 returns :
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde XT [Radeon HD 7770 GHz Edition]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

i used in /etc/default/grub the following lines
GRUB_CMDLINE_LINUX="apparmor=0 xen-pciback.hide=(01:00.0)(01:00.1)"
GRUB_CMDLINE_XEN="iommu=1"

lsmod | grep pci returns nothing so i think none of the xen-pcback , pciback , pci-stub modules are loaded

I tried the tutorial from http://wiki.xen.org/wiki/Xen_PCI_Passthrough but i think i am missing something

Anyone can help me with the hiding/assigning of the pci devices - a short step-by-step guide?


----------



## powerhouse

Quote:


> Originally Posted by *Masta Squidge*
> 
> Well, with intel right now it would have to be maybe 1 core plus its HT brother, but that doesn't leave an even amount for two VMs. A 3770k might have 8 logical threads, but only 4 cores. Or perhaps assign the two HT threads to the OS, and two primary cores + 1 HT thread to each vm?
> 
> Is it possible to assign half the cpu to each VM while the OS just "takes what it needs"?


I don't think a Intel 3770K will work, as you need VT-d support (for Intel CPUs) - see http://ark.intel.com/search/advanced/?s=t&VTD=true.


----------



## powerhouse

Quote:


> Originally Posted by *fusioner*
> 
> Hi.First of all i want to thank you for this great thread.
> 
> I am using Ubuntu 13.04 x64 with xen 4.2.1 and virt-manager 0.9.4-2. i am getting the following error when trying to start Windows 8 x64 VM with PCI 0000:01.00.0 and 0000:01.00.0 added as new hardware in virt-manager :
> 
> Error starting domain: POST operation failed: xend_post: error from xen daemon: (xend.err "pci: PCI Backend and pci-stub don't own device
> 
> command: xm pci-list-assignable-devices
> returns:
> Error: pciback/pci-stub not loaded?
> 
> command: lspci | grep 01:00 returns :
> 01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Cape Verde XT [Radeon HD 7770 GHz Edition]
> 01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
> 
> i used in /etc/default/grub the following lines
> GRUB_CMDLINE_LINUX="apparmor=0 xen-pciback.hide=(01:00.0)(01:00.1)"
> GRUB_CMDLINE_XEN="iommu=1"
> 
> lsmod | grep pci returns nothing so i think none of the xen-pcback , pciback , pci-stub modules are loaded
> 
> I tried the tutorial from http://wiki.xen.org/wiki/Xen_PCI_Passthrough but i think i am missing something
> 
> Anyone can help me with the hiding/assigning of the pci devices - a short step-by-step guide?


You can't put GRUB_CMDLINE_LINUX="xen-pciback.hide=(01:00.0)(01:00.1)" in your grub file unless you have pciback compiled into the kernel (which isn't the case with Ubuntu).

Check this how-to: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013, particularly steps 9 to 11. Hope it helps.


----------



## Masta Squidge

Quote:


> Originally Posted by *powerhouse*
> 
> I don't think a Intel 3770K will work, as you need VT-d support (for Intel CPUs) - see http://ark.intel.com/search/advanced/?s=t&VTD=true.


Wait, so the 3770 can, but the K can't?

Dumbest thing I have seen today.


----------



## brandontaz2k2

So I'm confused, I need 2 video cards in order to game in a virtual machine? And would I be able to use my soundcard in this virtual machine?


----------



## powerhouse

Quote:


> Originally Posted by *Masta Squidge*
> 
> Wait, so the 3770 can, but the K can't?
> 
> Dumbest thing I have seen today.


Don't know if it's dumb, but many of the processors are "crippled" and don't support VT-d. Among these crippled ones are almost all K versions (except the 3930K that I use).

I'm not 100% sure whether or not KVM requires VT-d for PCI passthrough (Xen and VMware do), but most likely it is needed. Likewise AMD processors must have AMD-V support.

And to your question: Yes, the 3770 can do, the 3770K won't work.


----------



## powerhouse

Quote:


> Originally Posted by *brandontaz2k2*
> 
> So I'm confused, I need 2 video cards in order to game in a virtual machine? And would I be able to use my soundcard in this virtual machine?


Yes. Look at it this way: Using the methods described here you get two independently working operating systems running on one PC, each one using its own graphics card. If your CPU has a built-in GPU, like many modern Intel or AMD processors (e.g. Intel 3770 or AMD A10), you can use the onboard GPU for Linux and the graphics card for Windows, for example.

Sound can be a bit tricky. You can passthrough your sound card as a PCI device to your Windows guest, but then your Linux host will have no sound. I've solved it by attaching a USB sound card to a passed through USB port and connected the output of the sound card to the input of the onboard sound card. This way I have sound under Windows and under Linux.

It would be much nicer to have a pulseaudio solution, but the Windows driver is ancient (from around 2002) and most likely won't work with Windows 7 or 8.


----------



## onopono

This thread has really sparked my interest in the virtualization scene as I use Linux as a primary OS and would like to play Starcraft 2 without having to dual boot. How does this build look for Linux host and Windows VM?

i5 4670
16gb 1600 mhz RAM
ASUS Gryphon Z87
2 x Nvidia GTX650 Ti - one for each host and guest
240gb ssd
Antec EA-650

Also is there an updated guide on which OS and kernel patches best support this setup? Skimming the thread seems to suggest Arch as a good bleeding edge OS with the requisite software.


----------



## lloyd mcclendon

if that board supports VT-D you should be all set I believe ... however getting the nvidia card to passthrough may be one of: just works/dicey/adventurous/impossible. Typically ATI cards support reset better, you should read the entire thread, it will save you time even at its current length.

You have to decide if you want to go the route of XEN or KVM. The pros and cons of each are many, you kind of have to read around and decide - probably if you're serious enough, try both and see what setup meets your ideals better.

That said, Arch or any distro that's actively maintained should have the right current software available. It's hard for me to recommend debian or ubuntu for anything, they typically they lag behind or have too many chefs in the kitchen patching the software in negative ways. Arch would be a fine choice, many here are using that successfully.

You can always git clone and build the bleeding edge of the packages yourself if the distro isn't working out. (Though I never like to do that). Gentoo amd64 with ~amd64 qemu/libvirt has been working superbly.


----------



## powerhouse

@onopono: In addition to what lloyd mcclendon wrote above, consider the following:

1. It won't be easy to get Xen working with 2 Nvidia cards. First, the Nvidia proprietary drivers won't work under Xen (they should work OK under KVM) - at least I haven't found any workaround for current kernel releases.

2. As mentioned before, passing through a Nvidia card under Xen may or may not work. Chances are it won't. Many/most modern AMD cards are known to work, but check for compatibility to make sure!

3. The Xen development team introduced some PCI passthrough bugs in recent updates, which supposedly have been fixed with the latest updates. However, depending on the distro you are using, they won't be included yet (e.g. Ubuntu / Linux Mint).

4. Given suitable hardware, Xen can be installed from packages under Ubuntu/Linux Mint and other distros (Fedora, Arch, etc.) without patching kernels or Xen. Except for the last two Xen updates, all updates I've run worked smooth. Right now I've frozen (force version) the Xen release to not update to newer versions to avoid the passthrough problems (see thread above).

5. KVM is newer, and VGA passthrough often requires the latest kernels and/or patches. It seems to have better Nvidia support, including support of Nvidia proprietary drivers under Linux KVM (remember point 1, Xen is problematic).

Here some links that may be useful:

For KVM, see here: https://bbs.archlinux.org/viewtopic.php?id=162768.

For Xen on Linux Mint or Ubuntu: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013.


----------



## benben84

I was just thinking; man it would be great to be able to setup a Windows 8 Pro Hyper-V VM setup for gaming...

I would love to have it so I could just checkpoint it and revert it whenever I wanted to. I just don't see any way around it though, not going to happen.


----------



## Rezz

Quote:


> Originally Posted by *brandontaz2k2*
> 
> And would I be able to use my soundcard in this virtual machine?


As Powerhouse has said, you can pass it through as a PCI device, and if you have sound on your mainboard, keep mainboard sound for the host.

I just use the HDMI sound (on the graphics card I've passed through) in the virtual machine.......and use the mainboard sound (which could also be a sound card) on the host (Linux).

This works fine and allows me to have sound on both the host and virtual machine.


----------



## Eatfoodnow

First off, this is a really awesome thread, I've been getting involved with VMs as I get into software development and it's totally the technology of the future.

Anyways. I have a 2011 MBP 15" with an i7-2720QM and 6750M 1GB. I've read that the hardware supports VT-d, but I don't know if Apple has enabled it or not. Does anybody know if it is or not? Or, does anybody know of a way that I can check?

I'm thinking, if it's possible, to run a VM of Windows from OS X, using the integrated graphics for OS X (since I don't really need to do anything graphically intensive), while using the discrete card for Windows.


----------



## dolio

For reference, I think I'm pretty close to having this working with kvm using nbhusain's guide. I've successfully booted a fedora live-cd with my 7870 passed through to the VM.

I'm using an i7-4770 and an Asrock z87-extreme6, so Asrock's Haswell motherboards seem to continue successfully supporting VT-d.

Edit: Turns out I couldn't get the vfio passthrough working _with_ properly accelerated drivers, either on Linux or Windows guests. However, I found that Windows 7 was able to use the card passed through as a secondary fine (it hadn't been working for my old copy of Windows XP). So Vt-d is working.


----------



## fergalom

Possibly someone here can help me, have followed this guide and successfully running

Win7 VM with AMD card passed throug
Fedora 18 VM with Nvidia Quadro 2000 card passed through

Trying to run a custom weather forecast which uses the Cuda capability of the card on the F18 VM.

Latest Nvidia drivers 319.32
Cuda toolkit 5.5
Most Cuda toolkit samples will compile and run
However the output from the forecast programme is as follows:

Code:



Code:


NCL VERSION 6
USING NCL wrf_user_getvar()
Module ncl_jack_fortran.so compiled at Tue Apr 24 14:30:48 2012, expires 2017-12-06 
Drjack Cuda routines Init: Module ncl_jack.so compiled for Cuda computelevel 2.1 
                with opts " --generate-code arch=compute_20,code=sm_21 "
Found 1 cuda devices
 using Device 262144: "Quadro 2000"
Cuda ERROR: invalid device ordinalcudaSetDevice failed Drjacks wrf_user init...Complete
Output files in ./NCL

The key item is the ":invalid device ordinal" with the device number as Device "262144" which should be 0 typically I'm told.

Could xen be causing this?
Are there further settings on gpu passthrough?

Thanks
F


----------



## kizer

Very nice thread!
Have gotten it to work with my AMD video card, but it seems I made a mistake when I bought a micro-ATX motherboard. ASRock Z87M Pro4. It seems it only has 1 USB controller, which makes it kind of hard to get USB passthrough working...
Could anyone confirm this? I can't seem to find ANY info on the net about wether any motherboard has 1 or more USB controllers.

lspci | grep USB gives:
00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)
00:1a.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #2 (rev 04)
00:1d.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #1 (rev 04)

No matter which USB connector I use, everything is connected to 00:14.0. Also, if I set USB legacy support to enabled instead of smart auto (default) in the BIOS (UEFI), 00:1a.0 and 00:1d.0 dissapear, and lspci | grep USB only gives:
00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)

This leads me to believe that my motherboard only has one USB controller.
Any thoughts?


----------



## Chetyre

Hey.

It's been a long time since I posted, but that was because I was saving money to buy an AMD card. I now have a HD 7970, and would just like to report a successful secundary passthrough using xen 4.3 on an Archlinux Dom0! This thread has been most helpful all this time. Thank you!


----------



## Flakes

Hi all. I need some help with latest Nvidia GTX 770 (or 680) and Xen/KVM/whatever or actually I'm looking for facts that someone succeeded to do it.

So far I tried only Xen 4.2 and Ubuntu 12.04.2. My hardware supports VT-d, because I could passthrough few PCI devices and they worked very well.

But it's not the case for my GTX 770. In Windows the card is recognized properly, I could install the latest Nvidia driver. But I get error code 43 which is a very common problem. I had error 43 even before installing Nvidia driver. I think GTX 770 is very similar to 680 but I didn't see any success stories for either of the models. I feel like I'm the only one with that card trying to passthrough to a virtual machine.


----------



## powerhouse

Quote:


> Originally Posted by *kizer*
> 
> Very nice thread!
> Have gotten it to work with my AMD video card, but it seems I made a mistake when I bought a micro-ATX motherboard. ASRock Z87M Pro4. It seems it only has 1 USB controller, which makes it kind of hard to get USB passthrough working...
> Could anyone confirm this? I can't seem to find ANY info on the net about wether any motherboard has 1 or more USB controllers.
> 
> lspci | grep USB gives:
> 00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)
> 00:1a.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #2 (rev 04)
> 00:1d.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #1 (rev 04)
> 
> No matter which USB connector I use, everything is connected to 00:14.0. Also, if I set USB legacy support to enabled instead of smart auto (default) in the BIOS (UEFI), 00:1a.0 and 00:1d.0 dissapear, and lspci | grep USB only gives:
> 00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)
> 
> This leads me to believe that my motherboard only has one USB controller.
> Any thoughts?


You do have 3 USB controllers, but often only one USB controller has connectors on the board. Check your motherboard manual, you will probably see some USB headers on the board for the other USB controllers. Sometimes the front USB ports are connected to a different controller.

If you have USB headers on the motherboard, and a spare slot on the back, just buy a cheap USB PC expansion slot cover that connects to a header on the board - see here:


Good luck!


----------



## powerhouse

Quote:


> Originally Posted by *Flakes*
> 
> Hi all. I need some help with latest Nvidia GTX 770 (or 680) and Xen/KVM/whatever or actually I'm looking for facts that someone succeeded to do it.
> 
> So far I tried only Xen 4.2 and Ubuntu 12.04.2. My hardware supports VT-d, because I could passthrough few PCI devices and they worked very well.
> 
> But it's not the case for my GTX 770. In Windows the card is recognized properly, I could install the latest Nvidia driver. But I get error code 43 which is a very common problem. I had error 43 even before installing Nvidia driver. I think GTX 770 is very similar to 680 but I didn't see any success stories for either of the models. I feel like I'm the only one with that card trying to passthrough to a virtual machine.


Most Nvidia graphics are problematic to pass through, except some of the expensive Quadro, Tesla and other professional cards are suitable (those designated "Multi-OS"). There are some patches for other Nvidia cards to make them work. See http://www.davidgis.fr/blog/index.php?2011/12/07/860-xen-42unstable-patches-for-vga-pass-through and https://bbs.archlinux.org/viewtopic.php?id=162768.

Another way is to turn your 680 or other non-pro Nvidia card into a professional card by soldering some resistors - this of course voids your warranty and bears the risk of bricking your Nvidia card - see here for more: http://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/, specifically http://www.eevblog.com/forum/chat/hacking-nvidia-cards-into-their-professional-counterparts/msg207550/#msg207550 (it may be worth scanning the entire thread for more info on your card). Again, this is a very risky method.


----------



## vanquestly

you are right, I go and I am hoping to revise the guide and even make this about 4 steps less if I can.thanks


----------



## Dh0mp5eur

Hi all.

I would like to test the vfio VGA passthrough on Debian Sid with default qemu 1.6 and seabios 1.7.3.

Config:
I7 3770 (no K)
Asus P8Z77-M
First VGA: Intel IGP
Passthrough VGA: ATI/AMD HD7970

I recompiled the kernel 3.10.11 with

Code:



Code:


CONFIG_KVM=y
CONFIG_KVM_INTEL=y
CONFIG_PCI_STUB=y
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
CONFIG_HOTPLUG_PCI=y
CONFIG_HOTPLUG_PCI_PCIE=y

/etc/default/grub

Code:



Code:


GRUB_CMDLINE_LINUX="intel_iommu=on,pt,igfx_off kvm.ignore_msrs=1 kvm_intel.emulate_invalid_guest_state=0 vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:6798,1002:aaa0"

/etc/modprobe.d/kvm.conf

Code:



Code:


options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1
options kvm_intel emulate_invalid_guest_state=0
options vfio_iommu_type1 allow_unsafe_interrupts=1
blacklist radeon




Spoiler: Run Script



#!/bin/bash

CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l)

QEMU=/usr/bin/qemu-system-x86_64

VCPUS="6" #must be > 0 < cpu core number
CONFIG="-M q35 -enable-kvm"
MEM="8192"
MEM_PATH=""
CPU="host"
BIOS="/usr/share/qemu/bios.bin"
SND="-device hda-duplex,cad=0"
SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128"
USB_DEVICES="tablet host:04f2:0833 host:1ea7:0011"
GPU="01:00.0"
GPU_SND="01:00.1"
VBIOS=""
HDD="windows.img"
CDROM=""
EXTRA_ARGS="-boot menu=on -monitor stdio"

if [ -n "$GPU" ]; then
CONFIG+=" -vga none -nographic"
DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1"
if [ -n "$VBIOS" ]; then
DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1"
else
DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1"
fi
if [ -n "$GPU_SND" ]; then
DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1"
fi
fi

if [ -n "$PCI_DEVICES" ]; then
for d in $PCI_DEVICES; do
DEV+=" -device vfio-pci,host=$d"
done
fi

if [ -n "$HDD" ] || [ -n "$CDROM" ]; then
DEV+=" -device ahci,bus=pcie.0,id=ahci"
if [ -n "$HDD" ]; then
STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk"
fi
if [ -n "$CDROM" ]; then
STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd"
fi
fi

if [ -n "$SND" ]; then
SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 $SND"
export $SND_DRIVER_OPTS
fi

if [ -n "$CPU" ]; then
CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1"
else
CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1"
fi

if [ -n "$BIOS" ]; then
BIOS=" -bios $BIOS"
fi

if [ -n "$MEM" ]; then
MEM="-m $MEM"
if [ -n "$MEM_PATH" ]; then
MEM+=" -mem-path $MEM_PATH"
fi
else
if [ -n "$MEM_PATH" ]; then
MEM+=" -mem-path $MEM_PATH"
fi
fi

if [ -n "$USB_DEVICES" ]; then
for u in $USB_DEVICES; do
USB_DEV+=" -usbdevice $u"
done
fi

$QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $STORAGE $EXTRA_ARGS



VGA passthrough works fine.
The card is recognized as a standard vga.
No problem to restart the VM.
But when I install the Catalyst driver.
My VM crash after the splash screen.
It is possible that is just a driver problem?


----------



## SpacePirate

Hi, Im currently trying to get a setup like this running. So far i failed with KVM and Xen.
The problem is i have no errors on the console dmesg/xl dmesg - I just get no output on the passed GPU.

I tried to pass it to a minimal Guest (no network, usb and drives). Could that be the problem? that i need to boot an image first to get output?


----------



## norcoen

Hello,
is the initial post outdated?
I can't get this to run with fedora 19 - guess it just does not run the xen kernel.
It looks like it does in GRUB, but in the system: no xen if I'm not mistaken

Code:



Code:


> uname -r
3.11.1-200.fc19.x86_64

also getting the same error as someone else on this step:

Code:



Code:


> xm new /etc/xen/win7a.sxp
Using config file "/etc/xen/win7a.sxp".
Error: [Errno 2] No such file or directory

But I don't understand his solution. What .xml files did he delete?


----------



## SpacePirate

Quote:


> Originally Posted by *norcoen*
> 
> Hello,
> is the initial post outdated?
> I can't get this to run with fedora 19 - guess it just does not run the xen kernel.
> It looks like it does in GRUB, but in the system: no xen if I'm not mistaken
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> > uname -r
> 3.11.1-200.fc19.x86_64


Thats correct, even if you have booted xen, because xen boots your Fedora as dom0 so you run on a fedora system.
Try "xl info" or "xm info". It should print info about your system if xen is running
Quote:


> also getting the same error as someone else on this step:
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> > xm new /etc/xen/win7a.sxp
> Using config file "/etc/xen/win7a.sxp".
> Error: [Errno 2] No such file or directory
> 
> But I don't understand his solution. What .xml files did he delete?


He didnt delete anything. What its saying that it cant find a file which you specified. propably the config file (/etc/xen/win7a.sxp).


----------



## nbhusain

Quote:


> Originally Posted by *Dh0mp5eur*
> 
> Hi all.
> 
> I would like to test the vfio VGA passthrough on Debian Sid with default qemu 1.6 and seabios 1.7.3.
> 
> Config:
> I7 3770 (no K)
> Asus P8Z77-M
> First VGA: Intel IGP
> Passthrough VGA: ATI/AMD HD7970
> 
> I recompiled the kernel 3.10.11 with
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> CONFIG_KVM=y
> CONFIG_KVM_INTEL=y
> CONFIG_PCI_STUB=y
> CONFIG_VFIO_IOMMU_TYPE1=y
> CONFIG_VFIO=y
> CONFIG_VFIO_PCI=y
> CONFIG_VFIO_PCI_VGA=y
> CONFIG_HOTPLUG_PCI=y
> CONFIG_HOTPLUG_PCI_PCIE=y
> 
> /etc/default/grub
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> GRUB_CMDLINE_LINUX="intel_iommu=on,pt,igfx_off kvm.ignore_msrs=1 kvm_intel.emulate_invalid_guest_state=0 vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=1002:6798,1002:aaa0"
> 
> /etc/modprobe.d/kvm.conf
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> options kvm ignore_msrs=1
> options kvm allow_unsafe_assigned_interrupts=1
> options kvm_intel emulate_invalid_guest_state=0
> options vfio_iommu_type1 allow_unsafe_interrupts=1
> blacklist radeon
> 
> 
> 
> 
> Spoiler: Run Script
> 
> 
> 
> #!/bin/bash
> 
> CPU_CORE_NR=$(cat /proc/cpuinfo | egrep "core id|physical id" | tr -d "\n" | sed s/physical/\\nphysical/g | grep -v ^$ | sort | uniq | wc -l)
> 
> QEMU=/usr/bin/qemu-system-x86_64
> 
> VCPUS="6" #must be > 0 < cpu core number
> CONFIG="-M q35 -enable-kvm"
> MEM="8192"
> MEM_PATH=""
> CPU="host"
> BIOS="/usr/share/qemu/bios.bin"
> SND="-device hda-duplex,cad=0"
> SND_DRIVER_OPTS="QEMU_AUDIO_DRV=pa QEMU_PA_SAMPLES=128"
> USB_DEVICES="tablet host:04f2:0833 host:1ea7:0011"
> GPU="01:00.0"
> GPU_SND="01:00.1"
> VBIOS=""
> HDD="windows.img"
> CDROM=""
> EXTRA_ARGS="-boot menu=on -monitor stdio"
> 
> if [ -n "$GPU" ]; then
> CONFIG+=" -vga none -nographic"
> DEV+=" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1"
> if [ -n "$VBIOS" ]; then
> DEV+=" -device vfio-pci,host=$GPU,x-vga=on,rombar=0,romfile=$VBIOS,addr=0.0,multifunction=on,bus=root.1"
> else
> DEV+=" -device vfio-pci,host=$GPU,x-vga=on,addr=0.0,multifunction=on,bus=root.1"
> fi
> if [ -n "$GPU_SND" ]; then
> DEV+=" -device vfio-pci,host=$GPU_SND,addr=0.1,bus=root.1"
> fi
> fi
> 
> if [ -n "$PCI_DEVICES" ]; then
> for d in $PCI_DEVICES; do
> DEV+=" -device vfio-pci,host=$d"
> done
> fi
> 
> if [ -n "$HDD" ] || [ -n "$CDROM" ]; then
> DEV+=" -device ahci,bus=pcie.0,id=ahci"
> if [ -n "$HDD" ]; then
> STORAGE+=" -drive file=$HDD,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk"
> fi
> if [ -n "$CDROM" ]; then
> STORAGE+=" -drive file=$CDROM,id=isocd -device ide-cd,bus=ahci.1,drive=isocd"
> fi
> fi
> 
> if [ -n "$SND" ]; then
> SND="-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 $SND"
> export $SND_DRIVER_OPTS
> fi
> 
> if [ -n "$CPU" ]; then
> CPU="-cpu $CPU -smp $VCPUS,sockets=1,cores=$VCPUS,threads=1"
> else
> CPU="-smp $VCPUS,sockets=1,cores=$VCPUS,threads=1"
> fi
> 
> if [ -n "$BIOS" ]; then
> BIOS=" -bios $BIOS"
> fi
> 
> if [ -n "$MEM" ]; then
> MEM="-m $MEM"
> if [ -n "$MEM_PATH" ]; then
> MEM+=" -mem-path $MEM_PATH"
> fi
> else
> if [ -n "$MEM_PATH" ]; then
> MEM+=" -mem-path $MEM_PATH"
> fi
> fi
> 
> if [ -n "$USB_DEVICES" ]; then
> for u in $USB_DEVICES; do
> USB_DEV+=" -usbdevice $u"
> done
> fi
> 
> $QEMU $CONFIG $BIOS $CPU $MEM $DEV $USB_DEV $SND $STORAGE $EXTRA_ARGS
> 
> 
> 
> VGA passthrough works fine.
> The card is recognized as a standard vga.
> No problem to restart the VM.
> But when I install the Catalyst driver.
> My VM crash after the splash screen.
> It is possible that is just a driver problem?


You might need to patch qemu and kernel to get it working, see: https://bbs.archlinux.org/viewtopic.php?id=162768, as of kernel 3.12 the patches wont be required anymore see: https://bbs.archlinux.org/viewtopic.php?pid=1333216#p1333216


----------



## antarell

Quote:


> Originally Posted by *kizer*
> 
> Very nice thread!
> Have gotten it to work with my AMD video card, but it seems I made a mistake when I bought a micro-ATX motherboard. ASRock Z87M Pro4. It seems it only has 1 USB controller, which makes it kind of hard to get USB passthrough working...
> Could anyone confirm this? I can't seem to find ANY info on the net about wether any motherboard has 1 or more USB controllers.
> 
> lspci | grep USB gives:
> 00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)
> 00:1a.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #2 (rev 04)
> 00:1d.0 USB controller: Intel Corporation Lynx Point USB Enhanced Host Controller #1 (rev 04)
> 
> No matter which USB connector I use, everything is connected to 00:14.0. Also, if I set USB legacy support to enabled instead of smart auto (default) in the BIOS (UEFI), 00:1a.0 and 00:1d.0 dissapear, and lspci | grep USB only gives:
> 00:14.0 USB controller: Intel Corporation Lynx Point USB xHCI Host Controller (rev 04)
> 
> This leads me to believe that my motherboard only has one USB controller.
> Any thoughts?


I have the same board and am using USB pass through:

Code:



Code:


-usb 
-usbdevice host:046d:c315 \
-usbdevice host:045e:00cb

Replace the USB device id's with the ones from your devices and it will capture them from the host.

What host kernel/qemu versions and options are you using? I am having no end of dramas getting Windows to boot with my AMD 7850 card, it will install but then gets stuck in a reboot loop on the first boot. It works fine if I use emulated graphics?!

Ant.


----------



## shadman

Shame to see I'm posting again exactly 13 months later and Nvidia cards are still not easy to use. I guess the hardware is the limiting factor here. I was considering the new AMD cards anyway, then I can finally do this!


----------



## nbhusain

Quote:


> Originally Posted by *shadman*
> 
> Shame to see I'm posting again exactly 13 months later and Nvidia cards are still not easy to use. I guess the hardware is the limiting factor here. I was considering the new AMD cards anyway, then I can finally do this!


Qemu-kvm with nvidia cards works using vfio-pci, there has been success reports of people using geforce 660gtx


----------



## nodriver

Hi, I am having a little trouble with this. I followed most of this guide, when starting my Windows guest I had the following error:

Error: pci: PCI Backend and pci-stub don't own device 0000:02:00.0

after doing some searching I learned that I need to unbind the PCI devices and rebind them to the pciback interface. I found the following instructions:

echo 0000:08:00.0 > /sys/bus/pci/devices/0000:08:00.0/driver/unbind
echo 0000:08:00.0 > /sys/bus/pci/drivers/pciback/new_slot
echo 0000:08:00.0 > /sys/bus/pci/drivers/pciback/bind

The first thing I do this for is my PCI USB card. It works perfectly, and I am able to pass the device to my VM. Then I try to do it for my video card, which is a Radeon 7970. The relevant output of lspci:

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/R9 280X]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

When I sudo -i to root and issue the command: echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind

I have the following result: -bash: /sys/bus/pci/devices/0000:02:00.0/driver/unbind: No such file or directory

when I issue the command for the audio portion of the device it just hangs. I investigated the issue a little by going to the relevant directories and looking around:

pwd:

/sys/bus/pci/devices/0000:02:00.0

ls -al:

boot_vga consistent_dma_mask_bits drm local_cpus power resource resource4 subsystem_vendor
broken_parity_status d3cold_allowed enable modalias remove resource0 rom uevent
class device irq msi_bus rescan resource0_wc subsystem vendor
config dma_mask_bits local_cpulist numa_node reset resource2 subsystem_device

so I think the reason it doesn't unbind is because there is no driver directory and nothing called unbind. When I go to the audio area:

pwd: /sys/bus/pci/devices/0000:02:00.1

broken_parity_status d3cold_allowed enable modalias power resource subsystem_device
class device irq msi_bus remove resource0 subsystem_vendor
config dma_mask_bits local_cpulist msi_irqs rescan sound uevent
consistent_dma_mask_bits driver local_cpus numa_node reset subsystem vendor

So there actually is a driver directory and it contains a file called unbind:

pwd: /sys/bus/pci/devices/0000:02:00.1/driver
ls: 0000:00:03.0 0000:00:1b.0 0000:02:00.1 bind module new_id remove_id uevent unbind

I don't know how to continue from here. I wish that I could unbind my discrete video card components as easily as I did my PCI USB card. Please help







.

I should also add that if I attempt the following:

echo 0000:02:00.0 > /sys/bus/pci/drivers/pciback/new_slot
echo 0000:02:00.0 > /sys/bus/pci/drivers/pciback/bind

the first command works fine but the second gives me the error: -bash: echo: write error: No such device

I should also add that my dom0 is kubuntu


----------



## CptDanko

Gaming on Linux in a virtual machine? Bleh, and on top of that with Fedora? Which is geared towards en business environment?

I say nay, don't get me wrong I love Linux but I wouldn't game with it, but if you insist check out Ultimate gamers edition. Its based on Ubuntu, and comes out of the box with everything you need.


----------



## nodriver

It doesn't really matter what the host is because the guest is Windows. I don't know what the ultimate gaming distro is, but it probably is using something like Wine. I want to be able to play all Windows games and use all the mods for the games. Wine supports a lot of games but is has no support for many graphics mods. Using a Windows guest with VT-d to pass a graphics card to it is the best option.

I hope I can get it to work, I am only a small step away. I hope somebody can help me out.

edit: I just tried: echo -n 0000:02:00.0 > /sys/bus/pci/drivers/radeon/unbind , this results in it hanging

edit: lspci -k

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/R9 280X]
Subsystem: Hightech Information System Ltd. Tahiti XT2 [Radeon HD 7970 GHz Edition]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
Subsystem: Hightech Information System Ltd. Device aaa0
Kernel driver in use: snd_hda_intel

So it doesn't look like the video controller of the 7970 is bound to any driver, which explains why unbinding it doesn't work. But it doesn't explain why my terminal hangs when I try to unbind the audio device, or why I can't bind the video controller to pciback.

sudo echo 0000:02:00.0 > /sys/bus/pci/drivers/pciback/bind
echo: write error: No such device

Why doesn't it think this is a real device? It is in the output of lspci







.


----------



## nodriver

Hi once again, I managed to get everything working! I had a very hard time to do this so I will share the steps that I followed and hope that it helps someone. This might help people even if they have different hardware than I do, but I think it will be the most help to people who use Ubuntu and who have Radeon graphics cards.

1. Install the dom0.

When installing dom0 you should use LVM. If you plan to use only virtual machines without using dom0 for anything else, you only need to make a partition for boot and root. Make the boot partition the first one, put root in the LVM. Your root should be about 15 gigabytes. Don't do anything with the rest of the space, you will use it for making a virtual drive for your domu. If you plan to use dom0 as your regular OS and only plan to make a Windows domu to play games then you should configure your drive however you want, but you should still leave some free space in the LVM for your Windows domu. How much space you leave for Windows is up to you, I think 100 gigabytes is enough. Using LVM like this will give you better performance than using a file.

2. Install xen.

The way to do this will probably depend on your dom0 operating system. https://help.ubuntu.com/community/Xen has some steps that you can follow if you use Ubuntu.

sudo apt-get install xen-hypervisor-amd64
sudo sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Ubuntu GNU/Linux, with Xen hypervisor"/' /etc/default/grub
sudo update-grub
sudo sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen

The guide for Ubuntu is missing some steps.

https://bugs.launchpad.net/ubuntu/+source/virt-manager/+bug/915954

sudo nano /etc/xen/xend-config.sxp
change (xend-unix-server no) to (xen-unix-server yes), uncomment (xen-unix-server yes) and (xend-unix-path /var/lib/xend/xend-socket)
ctrl-o saves ctrl-x exits

http://www.howtoforge.com/paravirtualization-with-xen-4.0-on-debian-squeeze-amd64

mv /etc/grub.d/10_linux /etc/grub.d/50_linux
update-grub2
update-grub
sudo reboot

3. install virt-manager, configure networking

virt-manager is very buggy and I almost don't like it, but I still used it for many steps.

sudo apt-get install virt-manager

Don't forget that your dom0 is a virtual machine, the virtual network interfaces are used by dom0 to connect to the internet. I don't know how you want to configure your networking but I will point you in the right direction. You can run virt-manager with the command virt-manager and then go to edit and then go to connection details and then go to network interfaces. You should read more about networking with Xen and figure out what you want, I had a lot of trouble with figuring out how to configure things. You should know that the Ubuntu network manager doesn't really do anything on Xen even though it looks like it does, I removed it from my system tray. You probably want to make a new network interface by clicking on the green plus sign and telling it to make a bridge that you add an eth interface to. If your internet stops working try restarting, this is very buggy.

4. make sure that you have VT-d enabled, blacklist radeon drivers.

You need to restart and go to your BIOS. It should have an option for VT-d if your CPU and motherboard support it. AMD has something like VT-d called IOMMU. After you enable VT-d in your BIOS boot back into your dom0. Now you need to tell Xen to use VT-d.

nano /etc/default/grub

There are a few changes you need to make in this file. Here is what I have in mine

GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M radeon.modeset=0"

iommu=1 tells Xen to use VT-d, dom0_mem tells Xen how much memory to let dom0 use, radeon.modeset=0 tells Xen not to load the radeon driver. If you've got a radeon graphics card you probably want to add radeon.modeset=0 but if more than your graphics card uses that driver it will stop whatever else uses it from working. You can see all of your PCI devices that use the radeon driver with lspci -k, if only your graphics card is using it then you want to add that line. If you have an Nvidia graphics card I don't know what you need to do. I also put the same radeon.modeset=0 in my GRUB_CMDLINE_LINUX, I don't know if this is what stopped the driver from loading or if it was from putting it in the xen line. After doing this make sure to sudo update-grub, I also sudo update-grub2 but I don't know if I needed to.

I also did nano /etc/modprobe.d/blacklist.conf and added "blacklist radeon" to the bottom of the file. I don't think this is what finally managed to blacklist the radeon driver, but I never changed it back.

sudo reboot

5. Configure pci passthrough

lspci -k | grep radeon

there shouldn't be any output from this command, because hopefully your radeon driver didn't load. You need to start xen-pciback now, for me this was a module but I don't know why and I don't know how to make it part of my kernel.

modprobe xen-pciback

lspci -k

you should get a big list of all of your PCI devices, mine looks like this

02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/R9 280X]
Subsystem: Hightech Information System Ltd. Tahiti XT2 [Radeon HD 7970 GHz Edition]
Kernel driver in use: pciback
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
Subsystem: Hightech Information System Ltd. Device aaa0
Kernel driver in use: pciback

pay special attention to the numbers in the xx:xx.x format. You need to find the devices that you want to passthrough to Windows. Probably you want to pass your VGA compatible controller and your Audio Device. These might have different names on your system. Both are part of your video card. I read that if the first numbers are not the same that you can't get this to work, for example 02:00.0 and 02:00.1 is ok but 03:00.0 and 02:00.0 isn't going to work. I don't know if this is true or not. If your video card has a VGA controller and an audio controller you need to passthrough both of them.

Now you need to give the following command for each of your devices, replace the numbers with the ones you got from lspci

echo 0000:02:00.1 > /sys/bus/pci/devices/0000:02:00.1/driver/unbind
echo 0000:02:00.1 > /sys/bus/pci/drivers/pciback/new_slot
echo 0000:02:00.1 > /sys/bus/pci/drivers/pciback/bind

I noticed that after I blacklisted my radeon driver my radeon VGA compatible controller would automatically bind to xen-pciback, but I still had to unbind and bind all of my other devices. I'm still stuck on getting my USB controller to passthrough, now it gives me the same 'device not found' error that I had when I was first trying to passthrough my graphics card. At first I couldn't passthrough my graphics card but could passthrough my USB controller, now I can passthrough my graphics card but I can't passthrough my USB controller. Gordan Bobic suggested that I use xl instead of xm, this was the first thing I did that let me passthrough my graphics card but xl is very buggy compared to xm and I switched back. I think blacklisting the radeon driver is what let me use the echo commands but I don't know why my USB controller can't passthrough now.

6. Install the Windows HVM.

I did this in virt-manager but I don't know if you should do this because it will break after you install the drivers. I will say how I did it and how I fixed it.

First you need to make a logical volume to install Windows to. In virt-manager go to edit and then go to connection details and then go to storage and then click on the name of your logical volume group and then click the New Volume button. You can name the volume whatever you want to and give it as much space as you want, I allocated all of the space right away. In virt-manager click on the create a new virtual machine button and go through the steps for making a windows HVM. I gave my Windows HVM 8 gigabytes of memory and 5 CPUs. On step 4 select managed or other existing storage and click the browse button. Select the logical volume that you just made. On the last step make sure that you select the network interface you want, this is probably a bridge. Select to customize configuration before install and then click finish. A window will appear, click on the add hardware button. Now click on PCI host device and select the PCI device you want to passthrough. You need to do this for each of the PCI devices you want to passthrough. Make sure that you've already bound all of these devices to xen-pciback. Now finish and start the Windows installation.

7. Configure Windows

After you have Windows installed you need to install the drivers for your video card. Some people report trouble when using Catalyst Control Center, I only selected to install the video and audio drivers to try to avoid problems. It works for some people though.
After your drivers are installed you will be asked to reboot. Go ahead and reboot, you will probably boot into a BSOD though. At this point I tried various things to make this work. I repeatedly rebooted dom0 because virt-manager wasn't able to power the Windows HVM on again after the BSOD. Rebooting dom0 fixed this. Every time you reboot dom0 you need to do step 5 again. You can make a script to do this if you want. No matter how many times I rebooted and tried to make it work I always ended up with a BSOD. At this point you need to stop using virt-manager.

cd /etc/xen
mkdir windows
cd windows
nano windows.conf

my windows.conf file looks like this, I copied almost all of it from a post I saw on a forum.

builder='hvm'
memory = 8192
vcpus=5
name = "win7"
vif = [ 'type=ioemu, bridge=xenbr0' ]
disk = [ 'phy:/dev/lvg/windows,xvda,w' ]
acpi = 1
boot="c"
sdl=0
serial='pty'
vnc=1
pci=[ '02:00.0', '02:00.1' ]

for memory and vcpus you should put what you want to give to your windows HVM. Vif and disk should be modified for your configuration. You also need to put the numbers of your PCI devices in pci, remember that you can find these numbers with lspci. You need to use a different name than you used when you made your Windows HVM.

ctrl-o to save and ctrl-x to quit

xm create windows.conf

now Windows starts and you can go and look at it in virt-manager just like your first Windows HVM. Instead of BSOD you should now have your screen freeze during the loading Windows part of booting. Now you can switch your monitors input to whatever your Radeon card uses and you should go to full screen Windows! If you can't move your mouse you need to switch back to your regular video input and click into the vnc window. If you manage to passthrough a USB controller you can probably use a USB switch.


----------



## lloyd mcclendon

nbhusain or anyone using VFIO successfully --

I gave it a try a couple weeks ago, but wasn't able to get anywhere. I went back to the legacy pci-assign, and well, it pretty much sucks. When it works, it's great, but it hardly works day-to-day to continue to be useful.

Well, I tried VFIO again last night, and was able to get something going. I think I am pretty close to a great success. The guest gets the card OK, I can install various things, but eventually I get stuck in a "launch startup repair" loop. I've yet to determine exactly what is the culprit here, it could even be the disk controller of all things (or guest VGA drivers, or failure to reset). I believe my motherboard supports interrupt remapping as I don't have any error messages anywhere ... well, except for this one, sometimes, which I've seen before:

Code:



Code:


kvm [6867]: vcpu0 unhandled wrmsr: 0x680 data 0

unhandled worm senior.. interesting.








Any ideas what that is and if it's a problem how to fix?

I was also wondering *if we currently need any patches still?* I'm reading posts that are several months old, talking about patches, but what's the current state? I am using:

* kernel 3.12 stable - directly from git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/ Are _all_ of AWilliam's changes merged into there by now?

* ~app-emulation/qemu-1.6.1 (the ~ is the latest unstable from gentoo's portage). This supports VFIO just fine as far as I can tell, but it may be missing some one-off patch

* ~ sys-firmware/seabios-1.7.3 -- In all your posts you talk about a seabios 1.7.2 with a patch. I compared the contents of the patch to the 1.7.3 tree has the changes in there, but I still see recommendations to use the patched 1.7.2. still? I never see any text from seabios, just a black screen and then the animated starting windows logo. Somehow.









Not that it matters much but I am also using the latest unstable libvirt/virt-manager/virt-inst tools to manage this guy, I did have to add the custom QEMU commands to get x-vga in there as they don't support that argument yet.

Code:



Code:


<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
....
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=86:00.0,id=hostdev7,bus=pci.2,multifunction=on,x-vga=on,addr=0x6'/>
  </qemu:commandline>
</domain>

I am currently trying a linux guest with the card instead, but just getting a black screen. However everything is alive and well with the guest, I am ssh'd to it, and lspci shows the card in there and dmesg looks happy. I'm leaning towards a seabios issue as I never see grub or any text long before the kernel config would matter?

Thanks in advance for any help!


----------



## nbhusain

Yeah there's a few patches still needed, specially on intel, i see you're using an nvidia card on your host, so if you're using the nvidia drivers you'll probably need to patch it too, and also you'll need the lastest qemu-git (or qemu 1.7 when it gets released) that support the new pcie reset interface on kernel 3.12, as for seabios the lastest release 1.7.3.2 doesnt require any additional patches, also most of my posts here are outdated you should probably check out my up-to-date guide on the arch forums https://bbs.archlinux.org/viewtopic.php?id=162768, the kernel dev (alex williamson) is also posting there so if you have any issues he'll probably help you out.

EDIT: the wrmsr are probably harmless you can tell kvm to ignore it:

Code:



Code:


echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf


----------



## Peon

Now that Citrix has made XenServer (and XenCenter) completely free, has anyone gotten this working in XenServer? I'd imagine that clicking the GPU passthrough button in XenCenter is a much more widely accessible solution than entering commands and modifying config files in Fedora - assuming that Citrix hasn't locked the functionality to select professional-grade GPUs, that is...


----------



## wonsnot

A bit of a noob here, but I was wondering what kind of hardware you would recommend for this software setup.
eg.
1. Proc
2. Video Card (I know you mention a high end newer Radeon) I was wondering if you would suggest a workstation card or a standard gaming card.
3. Any idea on what kind of network would be needed for this?

I know that this probably isn't what you intended, but I want to do a setup like this for a dumb terminal to make it easier to game in different rooms of my house. It would be epic to be able to play BF4 in my living room without having to my rig to the living room.

Thanks for your awesome guide and input.


----------



## azazippixy

good,Too bad there's no way to "sym link" it to the servers forum. I bet a lot of people in there would love this,thank you


----------



## lloyd mcclendon

Quote:


> Originally Posted by *wonsnot*
> 
> A bit of a noob here, but I was wondering what kind of hardware you would recommend for this software setup.
> eg.
> 1. Proc
> 2. Video Card (I know you mention a high end newer Radeon) I was wondering if you would suggest a workstation card or a standard gaming card.
> 3. Any idea on what kind of network would be needed for this?
> 
> I know that this probably isn't what you intended, but I want to do a setup like this for a dumb terminal to make it easier to game in different rooms of my house. It would be epic to be able to play BF4 in my living room without having to my rig to the living room.
> 
> Thanks for your awesome guide and input.


That's not really what this for at all. You'd still have to run cables all over your house, and there's no reason you couldn't accomplish that just running windows on bare metal.

You could possibly avoid all the cables by using certain software to stream full video feeds to distant rooms across the network, though I'm not familiar with them (yet). I would think gaming performance (both video and input response) would be in the tank if your motherboard is located in a different room and only connected by 1Gb/s Ethernet. Again, running a VM with a live graphics card has nothing to do with gaming in different rooms - you need the motherboard in the room or very long HDMI / USB cables everywhere.

All that the concept discussed in this thread really enables is for us Linux types to rightfully run Linux on the bare metal, but still have access to full native 3D acceleration (and other pci devices) for use in certain software that is only designed for certain other operating systems. It officially obsoletes a multi-boot setup forever. Obviously it is best to run Linux on the bare metal for all of its power and capabilities, while treating any other OS as a unknowing child - just a relocatable LV with snapshot capability. As contrasted with running the other OS natively to have the 3D accel there and maybe running some Linux VMs inside some poor-performing hypervisor over there - no good, do not want.

In other news, KVM with VFIO is the only way to go. It is rock solid. 15 days of uptime and counting since I got it working. I have rebooted this guest maybe 50 times. Perfect. The days of XEN pci-back and kvm pci-assign were ABYSMAL compared to this stack. Everyone should make an effort to switch to it.









Thanks again to nbhusian for putting the recipe together, it really does work.


----------



## wonsnot

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> That's not really what this for at all. You'd still have to run cables all over your house, and there's no reason you couldn't accomplish that just running windows on bare metal,
> 
> You could possibly avoid all the cables by using certain software to stream full video feeds to distant rooms across the network, though I'm not familiar with them (yet). I would think gaming performance (both video and input response) would be in the tank if your motherboard is located in a different room and only connected by 1GBt/s Ethernet. Again, running a VM with a live graphics card has nothing to do with gaming in different rooms - you need the motherboard in the room or very long HDMI / USB cables everywhere.
> 
> In other news, KVM with VFIO is the only way to go. It is rock solid. 15 days of uptime and counting since I got it working. I have rebooted this guest maybe 50 times. Perfect. The days of XEN pci-back and kvm pci-assign were ABYSMAL compared to this stack. Everyone should make an effort to switch to it.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks again to nbhusian for putting the recipe together, it really does work.


Thanks, That is exactly the feedback I needed.


----------



## powerhouse

My comments below refer to a Xen installation on Ubuntu or Ubuntu derivatives. For more info see my how-to for Linux Mint here: HOW-TO make dual-boot obsolete using XEN VGA passthrough, which can also be used for Ubuntu.
Quote:


> Originally Posted by *nodriver*
> 
> ...
> 
> 2. Install xen.
> 
> The way to do this will probably depend on your dom0 operating system. https://help.ubuntu.com/community/Xen has some steps that you can follow if you use Ubuntu.
> 
> sudo apt-get install xen-hypervisor-amd64
> sudo sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Ubuntu GNU/Linux, with Xen hypervisor"/' /etc/default/grub
> sudo update-grub
> sudo sed -i 's/TOOLSTACK=.*\+/TOOLSTACK="xm"/' /etc/default/xen
> 
> The guide for Ubuntu is missing some steps.
> 
> https://bugs.launchpad.net/ubuntu/+source/virt-manager/+bug/915954
> 
> sudo nano /etc/xen/xend-config.sxp
> change (xend-unix-server no) to (xen-unix-server yes), uncomment (xen-unix-server yes) and (xend-unix-path /var/lib/xend/xend-socket)
> ctrl-o saves ctrl-x exits
> 
> http://www.howtoforge.com/paravirtualization-with-xen-4.0-on-debian-squeeze-amd64
> 
> mv /etc/grub.d/10_linux /etc/grub.d/50_linux
> update-grub2
> update-grub
> sudo reboot


a) Editing xend-config.sxp is only necessary when you run virt-manager - see my comment below.

b)

Code:



Code:


mv /etc/grub.d/10_linux /etc/grub.d/50_linux

is one way of booting into Xen,

Code:



Code:


sudo sed -i 's/GRUB_DEFAULT=.*\+/GRUB_DEFAULT="Ubuntu GNU/Linux, with Xen hypervisor"/' /etc/default/grub

is another way.

Both should be followed by:

Code:



Code:


update-grub

No need to run update-grub2 in modern Ubuntu distros.

Quote:


> Originally Posted by *nodriver*
> 
> 3. install virt-manager, configure networking
> 
> virt-manager is very buggy and I almost don't like it, but I still used it for many steps.
> 
> sudo apt-get install virt-manager


As you wrote, virt-manager is very buggy. I did use it before but today I wouldn't use it with Xen, particularly not with recent distro releases. Instead install a vncviewer and use the

Code:



Code:


sudo xm vncviewer <domain>

command from a terminal window to view the Windows installation. In that case you need to prepare a Windows configuration file (see below).

Quote:


> Originally Posted by *nodriver*
> 
> 4. make sure that you have VT-d enabled, blacklist radeon drivers.
> 
> You need to restart and go to your BIOS. It should have an option for VT-d if your CPU and motherboard support it. AMD has something like VT-d called IOMMU. After you enable VT-d in your BIOS boot back into your dom0. Now you need to tell Xen to use VT-d.
> 
> nano /etc/default/grub
> 
> There are a few changes you need to make in this file. Here is what I have in mine
> 
> GRUB_CMDLINE_XEN="iommu=1 dom0_mem=2048M radeon.modeset=0"
> 
> iommu=1 tells Xen to use VT-d, dom0_mem tells Xen how much memory to let dom0 use, radeon.modeset=0 tells Xen not to load the radeon driver. If you've got a radeon graphics card you probably want to add radeon.modeset=0 but if more than your graphics card uses that driver it will stop whatever else uses it from working. You can see all of your PCI devices that use the radeon driver with lspci -k, if only your graphics card is using it then you want to add that line. If you have an Nvidia graphics card I don't know what you need to do. I also put the same radeon.modeset=0 in my GRUB_CMDLINE_LINUX, I don't know if this is what stopped the driver from loading or if it was from putting it in the xen line. After doing this make sure to sudo update-grub, I also sudo update-grub2 but I don't know if I needed to.
> 
> I also did nano /etc/modprobe.d/blacklist.conf and added "blacklist radeon" to the bottom of the file. I don't think this is what finally managed to blacklist the radeon driver, but I never changed it back.
> 
> sudo reboot


The GRUB_CMDLINE_XEN="... radeon.modeset=0" entry does *NOT* tell Xen to not load the radeon driver. This option helps prevent a "black screen" issue when the graphics card driver uses a screen setting incompatible with your monitor screen. Most often the option "nomodeset" will do a similar job.

The blacklisting is done either by:

a) Add "blacklist radeon" (for AMD Radeon cards) to the /etc/modprobe.d/blacklist.conf file, as you wrote.

b) Changing the /etc/default/grub to include the following line:

Code:



Code:


GRUB_CMDLINE_LINUX_DEFAULT="radeon.blacklist=1 quiet splash nomodeset"

The important part is "radeon.blacklist=1". If you want to pass through a Nvidia card (only select Nvidia card models will work as of now !!!!!!!!!!), you need to change "radeon.blacklist=1" to "nouveau.blacklist=1".

Option b) may require xen-pciback to be compiled into the kernel (and not loaded as a module as is being done in the other options).

A yet better way to make sure that the graphics card driver is not bound to the graphics card you wish to pass through to Windows is doing the following:

c) Determine the PCI IDs of the VGA card you wish to pass through (in the example below the PCI IDs of my second graphics card are 02:00.0 and 02:00.1), then as root enter:

Code:



Code:


echo "xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)" >> /etc/initramfs-tools/modules

Code:



Code:


echo "xen-pciback passthrough=1" >> /etc/modules

Code:



Code:


update-initramfs -k all -c

You must change the PCI IDs to the ones you determined!
The above commands tell Linux to bind the graphics card to pciback when the initramfs is processed. This is during the early phase of the boot process and usually prevents the graphics driver to attach itself to the graphics card. In a regular terminal window (non-root !!!), run:

Code:



Code:


lspci -v | grep -A 10 VGA

and you will see which driver is attached to the VGA card. After you performed the above steps and rebooted into Xen, you should get:

Code:



Code:


Kernel driver in use: pciback

for the graphics card you want to pass through. You can also run:

Code:



Code:


sudo xm pci-list-assignable-devices

and get the PCI IDs of the graphics card that are bound to pciback.

Quote:


> Originally Posted by *nodriver*
> 
> 5. Configure pci passthrough
> 
> lspci -k | grep radeon
> 
> there shouldn't be any output from this command, because hopefully your radeon driver didn't load. You need to start xen-pciback now, for me this was a module but I don't know why and I don't know how to make it part of my kernel.
> 
> modprobe xen-pciback
> 
> lspci -k
> 
> you should get a big list of all of your PCI devices, mine looks like this
> 
> 02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/R9 280X]
> Subsystem: Hightech Information System Ltd. Tahiti XT2 [Radeon HD 7970 GHz Edition]
> Kernel driver in use: pciback
> 02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
> Subsystem: Hightech Information System Ltd. Device aaa0
> Kernel driver in use: pciback
> 
> pay special attention to the numbers in the xx:xx.x format. You need to find the devices that you want to passthrough to Windows. Probably you want to pass your VGA compatible controller and your Audio Device. These might have different names on your system. Both are part of your video card. I read that if the first numbers are not the same that you can't get this to work, for example 02:00.0 and 02:00.1 is ok but 03:00.0 and 02:00.0 isn't going to work. I don't know if this is true or not. If your video card has a VGA controller and an audio controller you need to passthrough both of them.
> 
> Now you need to give the following command for each of your devices, replace the numbers with the ones you got from lspci
> 
> echo 0000:02:00.1 > /sys/bus/pci/devices/0000:02:00.1/driver/unbind
> echo 0000:02:00.1 > /sys/bus/pci/drivers/pciback/new_slot
> echo 0000:02:00.1 > /sys/bus/pci/drivers/pciback/bind
> 
> I noticed that after I blacklisted my radeon driver my radeon VGA compatible controller would automatically bind to xen-pciback, but I still had to unbind and bind all of my other devices. I'm still stuck on getting my USB controller to passthrough, now it gives me the same 'device not found' error that I had when I was first trying to passthrough my graphics card. At first I couldn't passthrough my graphics card but could passthrough my USB controller, now I can passthrough my graphics card but I can't passthrough my USB controller. Gordan Bobic suggested that I use xl instead of xm, this was the first thing I did that let me passthrough my graphics card but xl is very buggy compared to xm and I switched back. I think blacklisting the radeon driver is what let me use the echo commands but I don't know why my USB controller can't passthrough now.


See my comments on the previous step. If you use option 5 c), that is

echo "xen-pciback passthrough=1 hide=(02:00.0)(02:00.1)" >> /etc/initramfs-tools/modules

then you only need to do the above steps for the USB and other PCI devices you wish to pass through, that is not for the graphics card.

Quote:


> Originally Posted by *nodriver*
> 
> 6. Install the Windows HVM.
> 
> I did this in virt-manager but I don't know if you should do this because it will break after you install the drivers. I will say how I did it and how I fixed it.


It's easier to prepare a Windows config file (see next step) and use

Code:



Code:


sudo xm vncviewer <domain>

after running xm create ...

Quote:


> Originally Posted by *nodriver*
> 
> 7. Configure Windows
> 
> After you have Windows installed you need to install the drivers for your video card. Some people report trouble when using Catalyst Control Center, I only selected to install the video and audio drivers to try to avoid problems. It works for some people though.
> After your drivers are installed you will be asked to reboot. Go ahead and reboot, you will probably boot into a BSOD though. At this point I tried various things to make this work. I repeatedly rebooted dom0 because virt-manager wasn't able to power the Windows HVM on again after the BSOD. Rebooting dom0 fixed this. Every time you reboot dom0 you need to do step 5 again. You can make a script to do this if you want. No matter how many times I rebooted and tried to make it work I always ended up with a BSOD. At this point you need to stop using virt-manager.
> 
> cd /etc/xen
> mkdir windows
> cd windows
> nano windows.conf
> 
> my windows.conf file looks like this, I copied almost all of it from a post I saw on a forum.
> 
> builder='hvm'
> memory = 8192
> vcpus=5
> name = "win7"
> vif = [ 'type=ioemu, bridge=xenbr0' ]
> disk = [ 'phy:/dev/lvg/windows,xvda,w' ]
> acpi = 1
> boot="c"
> sdl=0
> serial='pty'
> vnc=1
> pci=[ '02:00.0', '02:00.1' ]
> 
> for memory and vcpus you should put what you want to give to your windows HVM. Vif and disk should be modified for your configuration. You also need to put the numbers of your PCI devices in pci, remember that you can find these numbers with lspci. You need to use a different name than you used when you made your Windows HVM.
> 
> ctrl-o to save and ctrl-x to quit
> 
> xm create windows.conf
> 
> now Windows starts and you can go and look at it in virt-manager just like your first Windows HVM. Instead of BSOD you should now have your screen freeze during the loading Windows part of booting. Now you can switch your monitors input to whatever your Radeon card uses and you should go to full screen Windows! If you can't move your mouse you need to switch back to your regular video input and click into the vnc window. If you manage to passthrough a USB controller you can probably use a USB switch.


Use xm vncviewer after you started the Windows guest (xm create ...) to see your Windows installation in a VNC window. Once Windows is installed and rebooted, install the graphics driver for the graphics card under Windows. After another reboot switch to your second graphics card and if all went well you should be able to enjoy VGA passthrough.


----------



## powerhouse

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> In other news, KVM with VFIO is the only way to go. It is rock solid. 15 days of uptime and counting since I got it working. I have rebooted this guest maybe 50 times. Perfect. The days of XEN pci-back and kvm pci-assign were ABYSMAL compared to this stack. Everyone should make an effort to switch to it.
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Thanks again to nbhusian for putting the recipe together, it really does work.


Thanks for your feedback on KVM with VFIO.

I'm running my Windows VM on Xen for more than a year, using the xm toolstack with Xen 4.1 and now an "experimental" install of Xen 4.3 (again xm toolstack). Both performance and stability is top notch. I'm using an off-the-shelf Linux Mint distribution and an unpatched kernel. The only real limitations I'm aware of are Nvidia graphics cards, most of which aren't supported under Xen.

Of course I can reboot my Windows guest as often as I want.

So I'm interested to hear your experience with Xen, and why you find KVM better? I would like to give KVM a shot, but can't justify the effort at the moment.


----------



## lloyd mcclendon

I've already detailed my thoughts and experience with XEN further back in this thread. If you search for my posts you can see them in detail. In summary:

1) Nvidia cards are a disaster for DOM0 or DOMU and I happen to have one of those cards.
2) The XL toolstack did not work right for me, at all.
3) The XM toolstack did work but was not 100%. Not even close to the stability of the VFIO stack. For me at the time at least. With a newer version and some more patience, YMMV. It did work, but even the KVM pci-assign stack worked slightly better.
4) The whole architecture of XEN doesn't fit well with my brain. The concept of DOM0 just doesn't fit. There are two diagrams I posted earlier that really articulated this point. Here: http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/560_40#post_19699969
5) Half baked kernel code in XEN when compared to all of the work put in to linux stable. Different CPU schedulers etc. No thanks.
6) GPLPV drivers - ick

KVM+VFIO wins for me by a long shot. For me it is the perfect setup. And going forward it has all the momentum. I can't see XEN being the way of the future.


----------



## lloyd mcclendon

1000 posts


----------



## nbhusain

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> I've already detailed my thoughts and experience with XEN further back in this thread. If you search for my posts you can see them in detail. In summary:
> 
> 1) Nvidia cards are a disaster for DOM0 or DOMU and I happen to have one of those cards.
> 2) The XL toolstack did not work right for me, at all.
> 3) The XM toolstack did work but was not 100%. Not even close to the stability of the VFIO stack. For me at the time at least. With a newer version and some more patience, YMMV. It did work, but even the KVM pci-assign stack worked slightly better.
> 4) The whole architecture of XEN doesn't fit well with my brain. The concept of DOM0 just doesn't fit. There are two diagrams I posted earlier that really articulated this point. Here: http://www.overclock.net/t/1205216/guide-create-a-gaming-virtual-machine/560_40#post_19699969
> 5) Half baked kernel code in XEN when compared to all of the work put in to linux stable. Different CPU schedulers etc. No thanks.
> 6) GPLPV drivers - ick
> 
> KVM+VFIO wins for me by a long shot. For me it is the perfect setup. And going forward it has all the momentum. I can't see XEN being the way of the future.


7) The fact that XEN doesnt support resetting pcie devices after a shutdown/reset, which means that for most radeon cards you have to restart the host after the first run or use some hacks (ejecting the card, etc)


----------



## powerhouse

Quote:


> Originally Posted by *nbhusain*
> 
> 7) The fact that XEN doesnt support resetting pcie devices after a shutdown/reset, which means that for most radeon cards you have to restart the host after the first run or use some hacks (ejecting the card, etc)


Thanks lloyd mcclendon for posting your thoughts once again - please forgive my bad memory. I totally agree on lack of Nvidia support under Xen dom0, and sketchy support under domU.

I'm running Xen 4.1.3 and 4.3 using Linux Mint 14 and LM 16 respectively and have no problem with either version using the xm toolstack (aside from having to pin (force version) Xen 4.1.3 under LM14). I haven't had to compile a single line, and use only the packages available in the repository. My machine has been running Xen and VGA passthrough for about 15-16 months.

However, I can see the advantages of running KVM, particularly in conjunction with Nvidia VGA cards. What held me back from trying it out was having to get the right patches and compiling the kernel, and the documentation.

Yesterday I've prepared an external USB drive to give KVM a try.

@nbhusain: I think the PCIe reset you mention refers to the xl toolstack - I never heard about this issue with the xm toolstack. It's a real pity that the xm toolstack has been deprecated whereas the xl toolstack shows some bugs, particularly with VGA passthrough.

I guess I will be following your KVM how-to soon.


----------



## dizzy4

Hey guys. Sorry I am not able to update and be here to help more, but I do have news to report!

I got a new motherboard due to a failure and have my system up and running almost flawlessly.

The new setup:
dom0: CentOS 6.4 with a custom kernel (3.11) and Xen 4.3 compiled from source
domU: Windows 8.1 with integrated HD2000 passed through (!Must have bios=qemu-xen-traditional) -- more on this later
domU: Lubuntu 13.04 with HD7750 passed through (report on issues later)
domU: CentOS 6.4 Multi-purpose file/other server (headless)

dom0 is 90% stable which makes me very happy. I figured out how to properly use GRUB2 with efi and have hidden what I am passing through. This consists of USB controllers, VGA controllers and an Audio device. The only time I see a crash or lockup is after I have shutdown and reboot whichever VM controls the HD7750. The lockup only occurs when trying to reassign the VGA device to a new VM (this includes reboots). I think there is an easy fix for this as a kernel module or option. I built all of the proper IOMMU and VT-d options into the kernel rather than loading them as modules. This seems to be a more favorable option than loading the module.

Intel integrated passthrough only works with bios=qemu-xen-traditional option set in the configuration file for the vm. This is actually a fault of QEMU upstream not having pci-passthrough enabled upstream. It is an issue that QEMU devs seem to know about, but have done little to fix. This is the only way that the video bios can be loaded and might even be an option for nVidia users to try. I am waiting with baited breath for this to be updated.

The Intel integrated passthrough will occasionally cause a BSOD to my windows domU when I try to play a different game without a restart. It seems to be very minor and will work fine after the restart. I think this could be solved by provisioning extra shadow memory or possibly in a memory option somewhere. I can report decent quality with DOTA2 (my main game) and in Source mods.

Slowly but surely this technology is becoming more stable. I will try to be more helpful in the future and would welcome any input on a new guide.

PS: It looks like the xl toolstak not properly resetting passed through cards only applies to AMD VGAs. My integrated chip has no issues with being reset, nor do any other PCI devices.


----------



## powerhouse

Hello dizzy4, good to hear from you!

About xl versus xm - I'm still using the xm tool stack because of the domU restart issue, although I've switched to Xen 4.3. So you are saying this restart issue only occurs with AMD GPUs being passed through?

dizzy4, can you share your motherboard model and BIOS version? It's for the VT-d compatible motherboard thread.

I've noticed that some vendors have fixed or are in the process of fixing BIOS bugs to get VT-d or AMD-V working, but there is still a lot of uncertainty and lack of documentation, let alone commitment from vendors. See for example here regarding VT-d and Ivybridge CPU support: http://rog.asus.com/forum/showthread.php?40309-No-VT-d-support-for-IVB-E-on-R4E-boards.


----------



## dizzy4

I am using a Gigabyte H77N-WIFI

It's a mini itx socket 1155 board with 2 NICs and wireless. My previous board was an as rock mini-itx board that also worked properly with vt-d.

I really like xl over xm because of the ease of use, but it doesn't appear to be as mature at this point. I can report that the integrated video crashes more often when loading new apps or games in a singular session. I am not sure if it is an issue with my configuration or seething else, but I do know that it resets without issue and after a crash the VM restarts.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> I am using a Gigabyte H77N-WIFI
> 
> It's a mini itx socket 1155 board with 2 NICs and wireless. My previous board was an as rock mini-itx board that also worked properly with vt-d.
> 
> I really like xl over xm because of the ease of use, but it doesn't appear to be as mature at this point. I can report that the integrated video crashes more often when loading new apps or games in a singular session. I am not sure if it is an issue with my configuration or seething else, but I do know that it resets without issue and after a crash the VM restarts.


Thanks for the hardware info!

I had to iron out some stuff too (some missing configs), but nothing that affected stability. I must say so far I've been real lucky - I can't remember seeing Linux or the Windows VM crash. My Xen setup has been rock stable for the past 1 1/2 years, except for once when I had to downgrade Xen. Now I'm running a stock Linux Mint 16 Mate with stock xen package. Performance is also great.

Reading your remark on xl vs. xm I believe I'll wait a while before trying xl.


----------



## dizzy4

The only instability I have had with either board (since I started using xen) is with the resetting of the AMD card. My first few iterations using xm also had reset issues. In fact, apart from that everything has worked prorperly since early 2012.


----------



## Xaero252

Here's a question I've been wondering about for a while with this... My laptop has nVidia Optimus (GT650m) and I prefer running Linux on it, since I get better battery life overall, and the majority of my games work in Linux natively, or via wine. However very rarely I run into a game I would like to play that doesn't work well on Linux. Has anybody had any success with an optimus laptop and XEN VGA Passthru?

I figure I may try to save myself some time, but if nobody has tried it - I may as well be the first.


----------



## powerhouse

It depends on your hardware and motherboard BIOS. Actually they are rather slim given a laptop and Nvidia card.

The laptop (and BIOS) must support VT-d (not to be confused with VT-x) for Intel or AMD-V for AMD. Without that it will never work.

Next obstacle is the graphics card - I doubt that the Nvidia 650m will work, but there might be patches or ways to get it to work. However, you first need to make sure that the laptop supports VT-d (or AMD-V).


----------



## Mshenay

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> "why someone would want to do this as opposed to running Windows natively"
> 
> Well, obviously _if_ it works just as well there's no need to boot into windows. I never cared for a dual boot, huge waste of time. This way everything is kept in linux, and you have some throw away windows VM for gaming. If the performance is as good as the OP is claming, this is 1000 times better than wine, and 100000 times better than a dual boot. So if it works, i don't know why you wouldn't want to do this. Really this is HUGE news - how many linux users crawl back to windows entirely or dual boot just for gaming. And wine .. it hurts to say this, but it's practically impossible for that project to keep up with the crazyness that the MS developers create.
> 
> Apparently KVM does offer the VT-D PCI pass through ... but i'm a little fuzzy on whether or not it actually works right yet. I'll be trying to get one of my gentoo vms with xorg on it to use this... and if that actually works I should be able to get my XP VM showing good FPS as well.
> 
> I guess maybe I don't understand this - if I give a windows guest pass through access to my graphics card, and install the windows nvidia drivers package etc - what happens to the host (and the 12 other VMs) using the same graphics card?


GAWD YES, this is something I want to do









hoping to build a sexy new rig in the Fall to do just this,

How ever I might need to get moRam for it anyways, let's talk purely the build at this point

http://pcpartpicker.com/p/2B6XJ

what would you suggest I upgrade CPU wise


----------



## powerhouse

Quote:


> Originally Posted by *Mshenay*
> 
> GAWD YES, this is something I want to do
> 
> 
> 
> 
> 
> 
> 
> 
> 
> hoping to build a sexy new rig in the Fall to do just this,
> 
> How ever I might need to get moRam for it anyways, let's talk purely the build at this point
> 
> http://pcpartpicker.com/p/2B6XJ
> 
> what would you suggest I upgrade CPU wise


About your hardware list:

CPU: Definitely NOT the 4770K !!! In most cases the "K" stands for "krippled", that is the CPU doesn't support VT-d. The 4770 is fine, though. See here for a list of compatible Intel CPUs.

Motherboard: Have a look at http://www.overclock.net/t/1338063/vt-d-compatible-motherboards. The list is by far NOT complete, but it should give you a starting point. *You should make absolutely sure that the motherboard, the BIOS and the CPU are supporting VT-d!* For example, Asus X79 boards support VT-d, but not (yet) for the newer Ivybridge-E CPUs (except the RIVE).

Graphics cards (for the Windows guest): Most modern AMD cards are fine. See http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters. Again, the list is NOT complete.

Both compatibility lists are based on user reports who had success with VGA passthrough. If you succeed with hardware that's not listed, please post in the corresponding thread.


----------



## Mshenay

Quote:


> Originally Posted by *powerhouse*
> 
> About your hardware list:
> 
> CPU: Definitely NOT the 4770K !!! In most cases the "K" stands for "krippled", that is the CPU doesn't support VT-d. The 4770 is fine, though. See here for a list of compatible Intel CPUs.
> 
> Motherboard: Have a look at http://www.overclock.net/t/1338063/vt-d-compatible-motherboards. The list is by far NOT complete, but it should give you a starting point. *You should make absolutely sure that the motherboard, the BIOS and the CPU are supporting VT-d!* For example, Asus X79 boards support VT-d, but not (yet) for the newer Ivybridge-E CPUs (except the RIVE).
> 
> Graphics cards (for the Windows guest): Most modern AMD cards are fine. See http://www.overclock.net/t/1307834/xen-vga-passthrough-compatible-graphics-adapters. Again, the list is NOT complete.
> 
> Both compatibility lists are based on user reports who had success with VGA passthrough. If you succeed with hardware that's not listed, please post in the corresponding thread.


thanks, although sadly the K Cpus where the ones unlocked for OCn to my understanding :/

either way my build is a little over 8 months away, I can revise it along the way :3 thanks for the help!


----------



## F3ERS 2 ASH3S

Quote:


> Originally Posted by *Mshenay*
> 
> thanks, although sadly the K Cpus where the ones unlocked for OCn to my understanding :/
> 
> either way my build is a little over 8 months away, I can revise it along the way :3 thanks for the help!


http://en.wikipedia.org/wiki/X86_virtualization
VT-d
An input/output memory management unit (IOMMU) enables guest virtual machines to directly use peripheral devices

I do see that the 4770k supports VT-x not sure what the exact difference is


----------



## powerhouse

Quote:


> Originally Posted by *F3ERS 2 ASH3S*
> 
> http://en.wikipedia.org/wiki/X86_virtualization
> VT-d
> An input/output memory management unit (IOMMU) enables guest virtual machines to directly use peripheral devices
> 
> I do see that the 4770k supports VT-x not sure what the exact difference is


VT-x is *NOT* the same as VT-d. To get the IOMMU feature, the CPU and motherboard/BIOS must support VT-d. Without that it won't work! The 4770K is therefor not an option.

I don't know why Intel "cripples" their "K" line, except for the LGA 2011 CPUs.


----------



## F3ERS 2 ASH3S

Quote:


> Originally Posted by *powerhouse*
> 
> VT-x is *NOT* the same as VT-d. To get the IOMMU feature, the CPU and motherboard/BIOS must support VT-d. Without that it won't work! The 4770K is therefor not an option.
> 
> I don't know why Intel "cripples" their "K" line, except for the LGA 2011 CPUs.


I got that.. so x does not have the option to use iommu and that is the only difference?


----------



## powerhouse

Quote:


> Originally Posted by *F3ERS 2 ASH3S*
> 
> I got that.. so x does not have the option to use iommu and that is the only difference?


VT-x and VT-d are just two different things. Most modern Intel CPUs will have VT-x support, but VT-d support is less common. VT-d is mostly used in servers, in order to pass through PCI devices to achieve better performance. A common use for it is to pass through network cards or storage controllers to guests on a server. The guest will then have direct access to the hardware, without emulation or going through the host OS.

AMD also has a range of CPUs that support IOMMU, they just call it AMD-Vi (or AMD-V ?, not sure).


----------



## F3ERS 2 ASH3S

Quote:


> Originally Posted by *powerhouse*
> 
> VT-x and VT-d are just two different things. Most modern Intel CPUs will have VT-x support, but VT-d support is less common. VT-d is mostly used in servers, in order to pass through PCI devices to achieve better performance. A common use for it is to pass through network cards or storage controllers to guests on a server. The guest will then have direct access to the hardware, without emulation or going through the host OS.
> 
> AMD also has a range of CPUs that support IOMMU, they just call it AMD-Vi (or AMD-V ?, not sure).


Quote:


> Originally Posted by *powerhouse*
> 
> VT-x is *NOT* the same as VT-d. To get the IOMMU feature, the CPU and motherboard/BIOS must support VT-d. Without that it won't work! The 4770K is therefor not an option.
> 
> I don't know why Intel "cripples" their "K" line, except for the LGA 2011 CPUs.


I got that.. so x does not have the option to use iommu and that is the only difference?

It AMD-V Thank you for the info. I know aboutAMDs just not intel


----------



## Mshenay

Quote:


> Originally Posted by *powerhouse*
> 
> VT-x and VT-d are just two different things. Most modern Intel CPUs will have VT-x support, but VT-d support is less common. VT-d is mostly used in servers, in order to pass through PCI devices to achieve better performance. A common use for it is to pass through network cards or storage controllers to guests on a server. The guest will then have direct access to the hardware, without emulation or going through the host OS.
> 
> AMD also has a range of CPUs that support IOMMU, they just call it AMD-Vi (or AMD-V ?, not sure).


That is good to know :O,

Also I'm finally starting to enjoy my Linux class and I'm one step closer to getting prep for my Linux Gaming Build! I'll modify the CPU I've got set later this week most likely, depends on how much I can knock out!

Still, getting excited to build a monster machine for this :3 my inner geek is getting pumped!


----------



## dizzy4

Quote:


> Originally Posted by *Mshenay*
> 
> That is good to know :O,
> 
> Also I'm finally starting to enjoy my Linux class and I'm one step closer to getting prep for my Linux Gaming Build! I'll modify the CPU I've got set later this week most likely, depends on how much I can knock out!
> 
> Still, getting excited to build a monster machine for this :3 my inner geek is getting pumped!


That's great that you are considering this! I would say that your hardware is already compatible, except that you have one of the few socket 1155 gigabyte boards that do NOT support VT-d. The normal D3H does support it. It might have something to do with the extra USB 3.0 and marvel controllers.

On another note, I am testing out using fedora 20 and having an easier time with the install. The xen package is again available in the main fedora repository and is up-to-date. I created some kernel .rpm files that will also work. I am still using the xl toolstack and have to restart, but there has been a lot of effort put into xen-unstable to fix the iommu/vt-d issues.

New guide coming soon! (hopefully for real!)

Domain-0 setup:
1. Install F20
2. Install kernel
3. Install Xen
4. Small tweaks for networking

That's right! No more compiling needed in the next guide!


----------



## Mshenay

Quote:


> Originally Posted by *dizzy4*
> 
> That's great that you are considering this! I would say that your hardware is already compatible, except that you have one of the few socket 1155 gigabyte boards that do NOT support VT-d. The normal D3H does support it. It might have something to do with the extra USB 3.0 and marvel controllers.
> 
> On another note, I am testing out using fedora 20 and having an easier time with the install. The xen package is again available in the main fedora repository and is up-to-date. I created some kernel .rpm files that will also work. I am still using the xl toolstack and have to restart, but there has been a lot of effort put into xen-unstable to fix the iommu/vt-d issues.
> 
> New guide coming soon! (hopefully for real!)
> 
> Domain-0 setup:
> 1. Install F20
> 2. Install kernel
> 3. Install Xen
> 4. Small tweaks for networking
> 
> That's right! No more compiling needed in the next guide!


Hurray! Maybe by Septermber this will be even easier. looking forward to this build ^^


----------



## dizzy4

Hey all! I finally got it updated!! The version number for these beta versions will follow the fedora release ex 20.1 is beta1 for fedora 20

I also encourage you to try Powerhouse's guide. It may be more complex, but possibly more flexible. It's another solution which does some things better than this guide, but uses older software. Hopefully the main issue of radeon graphics not resetting under XL will be fixed.

http://forums.linuxmint.com/viewtopic.php?f=42&t=112013


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> Hey all! I finally got it updated!! The version number for these beta versions will follow the fedora release ex 20.1 is beta1 for fedora 20
> 
> I also encourage you to try Powerhouse's guide. It may be more complex, but possibly more flexible. It's another solution which does some things better than this guide, but uses older software. Hopefully the main issue of radeon graphics not resetting under XL will be fixed.
> 
> http://forums.linuxmint.com/viewtopic.php?f=42&t=112013


Wonderful news! I have to try it now that I got an external HDD just for experimenting.

Actually, I wish I could have made my how-to as simple as yours! The reason I wrote it was that I feel more comfortable with Linux Mint (or Debian/Ubuntu based distros) than with Redhat/Fedora etc. I think people should choose the distro they like and then pick the corresponding guide.


----------



## dizzy4

Quote:


> Originally Posted by *powerhouse*
> 
> Wonderful news! I have to try it now that I got an external HDD just for experimenting.
> 
> Actually, I wish I could have made my how-to as simple as yours! The reason I wrote it was that I feel more comfortable with Linux Mint (or Debian/Ubuntu based distros) than with Redhat/Fedora etc. I think people should choose the distro they like and then pick the corresponding guide.


I agree. If you want to make your guide a little simpler I would suggest making your scripts more user friendly. That is my next update. For instance, I want to have the user run a script, get the network info and give input as to which device to use. That way there will be no file editing involved.

The second thing you could do is compile a custom kernel. I suggest using the standard kernel config and just building in the Xen pci backend, front end and IOMMU instead of as modules. I think there are only about 5 or so options that will be changed in the kernel config. When I tried your guide i could not get the modules to load properly via your scripts.


----------



## powerhouse

Quote:


> Originally Posted by *dizzy4*
> 
> I agree. If you want to make your guide a little simpler I would suggest making your scripts more user friendly. That is my next update. For instance, I want to have the user run a script, get the network info and give input as to which device to use. That way there will be no file editing involved.
> 
> The second thing you could do is compile a custom kernel. I suggest using the standard kernel config and just building in the Xen pci backend, front end and IOMMU instead of as modules. I think there are only about 5 or so options that will be changed in the kernel config. When I tried your guide i could not get the modules to load properly via your scripts.


Thanks for the feedback - very helpful! I've made some changes recently, and put the pciback option in the initramfs, which - at least for me - works nicely. But then I use secondary passthrough ! Have you tried the Linux Mint 16 version?

The reason I didn't want to touch the kernel is that in my how-to a user can update/upgrade from the repos without having to worry. In the worst case, if something does go wrong, he can roll back / force version. I'm not sure how a custom kernel holds over time when packages are updated, and I wouldn't want to have to compile the kernel every time there is a new kernel. Then again, if there was a script to automate it, that would OK too. As long as a user doesn't run into dependency issues.

Regarding scripts: This is a weak spot of mine, which is why there aren't many scripts in my how-to (actually only one, IIRC). But you are absolutely correct - I could simplify the whole installation by just writing one or two short installation scripts. I'll think about it.

From what I gather from your guide, Xen solved the reset issue with Radeon cards? Is that Xen 4.3.1, or would 4.3 be OK too?


----------



## dizzy4

I tried the LM 16 edition. I will have to give it another shot and see what I can do. The reset issue should be fixed soon -- I was unclear. It looks like specific fixes are going to be included in 4.4 that should allow xl to do a proper job. The ability for users to upgrade without getting a fresh kernel is also very nice. It's not a huge issue for me since I just keep the Domain-0 a minimalistic install and virtualize everything. It's suggested by xen to keep the Domain-0 that way and not use it as a desktop. However, for the sake of being easy for the every-day user it makes more sense to just treat the Domain-0 as a working environment.

My setup:
Dom-0: Fedora 20 -- headless
Dom-U: Windows 8.1 -- Integrated Intel HD2000 -- Light Gaming
Dom-U: Lubuntu 13.04 -- Radeon HD7750 -- Media VM -- TV
Dom-U: CentOS 6.4 -- headless -- Home VPS -- Fileserver

I can assign the HD7750 to either VM, but a hard restart is still needed









I actually really liked linux mint when I tried your guide and am considering switching it to my media VM in stead of Lubuntu

Getting the xen-pciback module to load might be a good idea for me to work on too. I don't upgrade my Dom-0 much, but I bet other users will.


----------



## Natriumatom

Thanks a lot for this tutorial!

Your thread is the first I found that shows that it can be done what I want. I just bought a new system consisting of the following:

Intel Core i5 - 4440
AsRock B85M Pro4
8GB Ram
120GB SSD
Bluray Writer

and some additional stuff like storage drives and a Intel PRO/1000 CT PCI-E network card.

My current network consists of just a bunch of Apple MacBooks etc. so I don't want to run a Win Server. This new machine should be a replacement for my current NAS, a Acer H340 Homeserver with nas4free and my media player, a raspberry pi with OpenELEC Xbmc running.

The plan is to run a headless Xen, KVM or vSphere Server as a host, passthrough one of the Gbit NICs + internal HD4600 VGA + internal sound + Bluray Drive to a Win8.1 Host on which I plan so install XBMC with the assorted Bluray Player etc., on another headless VM I would like to install OMV or Xpenology for storage and downloading tasks etc. and then another headless VM with a LAMP stack for web development.

What would you say are my chances of getting it running with your script?

My current research shows me that the headless Xen with passthrough should work through your or the powerhouse tutorial. KVM could also be an option as explained here: http://blog.ktz.me/?p=403 and last but not least I haven't found any vSphere configuration where a headless server passedthrough a VGA card.

Thanks a lot for your work


----------



## dizzy4

Quote:


> Originally Posted by *Natriumatom*
> 
> Thanks a lot for this tutorial!
> 
> Your thread is the first I found that shows that it can be done what I want. I just bought a new system consisting of the following:
> 
> Intel Core i5 - 4440
> AsRock B85M Pro4
> 8GB Ram
> 120GB SSD
> Bluray Writer
> 
> and some additional stuff like storage drives and a Intel PRO/1000 CT PCI-E network card.
> 
> My current network consists of just a bunch of Apple MacBooks etc. so I don't want to run a Win Server. This new machine should be a replacement for my current NAS, a Acer H340 Homeserver with nas4free and my media player, a raspberry pi with OpenELEC Xbmc running.
> 
> The plan is to run a headless Xen, KVM or vSphere Server as a host, passthrough one of the Gbit NICs + internal HD4600 VGA + internal sound + Bluray Drive to a Win8.1 Host on which I plan so install XBMC with the assorted Bluray Player etc., on another headless VM I would like to install OMV or Xpenology for storage and downloading tasks etc. and then another headless VM with a LAMP stack for web development.
> 
> What would you say are my chances of getting it running with your script?
> 
> My current research shows me that the headless Xen with passthrough should work through your or the powerhouse tutorial. KVM could also be an option as explained here: http://blog.ktz.me/?p=403 and last but not least I haven't found any vSphere configuration where a headless server passedthrough a VGA card.
> 
> Thanks a lot for your work


Your hardware looks good and so does your plan. I run a headless server with a windows VM for games, Linux VM for media and a VPS that has all my media and can host whatever I need.

Make sure that you have your device model set for qemu-xen-traditional on your home theater VM. Windows 8.1 handles the integrated chip better than linux when passed through. something to note is that I still get some crashes with the integrated chip when switching games, but never when doing media viewing or streaming.

Network setup for advanced users can be tricky. Remember that you get near native performance between virtual machines that share a bridge. I advise using a bridge and passing through the extra adapter. You don't need extra data going to the switch and back again.

Be sure to use RAID for the fileserver and not ZFS. You need a lot more than 8GB for those filesystems. I actually suggest more than 8, but it might be enough for you. here is an example:

1GB Domain-0
2GB fileserver
4GB windows 8.1
1GB iGPU memory

you can always add 8GB more and another Graphics card to have 2 fully functional computers running at the same time










I would say your chances are very high. Good luck and ask if you need help.


----------



## Natriumatom

I will give it a try tomorrow when all the hardware arrives.

Did you ever give XenServer a try together with Xen Orchestra for remote administration of the headless system?


----------



## dizzy4

I don't use XenServer because it is very expensive and the free version does not have very many features. I suppose you could use XCP with XAPI and then use Xen Orchestra, but the hypervisor in the main Xen releases is much newer.

I have not tried XCP lately, but might try it again.


----------



## Natriumatom

All the stuff arrived and I put it together. The Fedora 20 install went smooth and all but when the time comes to start the VM I get the following error:

sudo xl create win8.sxp

-> libxl: error: libxl.c:87:libxl_ctx_alloc: Is xenstore daemon running?
failed to stat /var/run/xenstored.pid: No such file or directory
cannot init xl context

I tried reinstalling Xen but it tells me the newest version is already installed, and it boots your Xen kernel just fine at the start.

There also seems something amiss with the network config as my em1 connection just vanishes from time to time and I have to reboot.


----------



## dizzy4

hmm. It looks like you are very close. I have had that issue in the past. It has to do with the xenstore permissions or not being started at boot. I forget how I fixed it, but others online have had the same problem. Also be sure that you are booting xen then booting linux. Don't just boot the new kernel and expect xen to load. There should be an entry that says xen on linux or something.

What do your networking scripts look like? Have you set up the bridge yet? Post what your ifcfg-xenbr0 and ifcfg-em1 scripts look like


----------



## Natriumatom

I'm now away for the weekend but I will get back at it on sunday.

I overcame the first problem with selecting Xen Hypervisor in the Boot Menu together with your Kernel, but then ran into a problem with cant find the cfg file etc.

I will give it a clean start on sunday with a fresh install. For the cfg files I copied your stuff 1 to 1 but edited the device name to em1 which is the built in intel ethernet adapter and changed the MAC address. But after I restarted the network service my connection was gone and the ethernet connection didnt even show up in Settings -> Network anymore. I had to empty out the bridge cfg file for it to reappear.


----------



## dizzy4

ah yes. It shouldn't show up since the configuration file tells NetworkManager not to control it. if you do:

Code:



Code:


$ ifconfig

it should show the devices. NetworkManager is the service that makes it show up in the settings screen and other GUI programs, but has a few issues with xen networking still. It should still be present and functioning.


----------



## Natriumatom

Got the VM working now with network, will test passthrough soon.


----------



## saivajr

Hey Natriumatom,

How excacty did you overcome the " libxl: error: libxl.c:87:libxl_ctx_alloc: Is xenstore daemon running?
failed to stat /var/run/xens" error?
Btw great tutorial dizzy


----------



## Natriumatom

Did a complete new install and made sure I started the Xen as Hypervisor from the boot menu and from there it went. There are some quirks in the posted code like a mishap with "virtual_disks" and then "virtual_drives" used etc.

I couldnt get the passthrough to work. I tried it but after a reboot everything went to hell and nothing worked anymore. I will give the MINT tutorial now a try after I failed with XenServer 6.2 as well as it seems that cant be run headless and wont give up the IGP.


----------



## dizzy4

Quote:


> Originally Posted by *Natriumatom*
> 
> Did a complete new install and made sure I started the Xen as Hypervisor from the boot menu and from there it went. There are some quirks in the posted code like a mishap with "virtual_disks" and then "virtual_drives" used etc.
> 
> I couldnt get the passthrough to work. I tried it but after a reboot everything went to hell and nothing worked anymore. I will give the MINT tutorial now a try after I failed with XenServer 6.2 as well as it seems that cant be run headless and wont give up the IGP.


That seems odd. I don't understand how things are getting so messed up. It seems like an unrelated issue. Are you hiding the devices at boot? You can also try running the command

Code:



Code:


$ sudo xl pci-assignable-add xx:xx.x

Perhaps I should change the wording? What I am suggesting is creating an empty file of XX size to use as the virtual drive. I am still not sure which issues you are having. You shouldn't need to do a fresh install each time either. I think there is another issue that you are having. Make sure that VT-d is enabled in the bios and that all the other features are enabled.

When you say it went to hell, what exactly happened?


----------



## Natriumatom

I was in the last step of the install.

Enabled SSH and tested it - check
Hide PCI devices on boot - check
Enable IOMMU on boot - check
Define Passthrough Devices - check

Then I did a reboot and it started normally with the Xen Hypervisor, but there should be no VGA output anymore. When I logged in I saw that there was no more ethernet again and I got some error when trying to start the VM.

VT-d is enabled in bios and the devices I tried to passthrough were HD4600 VGA, on chip HD Sound and one USB controller.


----------



## Natriumatom

I'm giving up. It just wont work it seems with just the internal HD4600 and no other vga cards installed.

The Mint Tutorial just ended in a reboot loop after the passthrough was activated...

I think I will either buy a radeon card now and give XenServer 6.2 another try or just dumb a Ubuntu install on it and wait some time till this is more matured...


----------



## Xaero252

Quote:


> Originally Posted by *lloyd mcclendon*
> 
> Yeah, the 7000 series cards have an issue where they wont reset properly, the 6000 series cards do not have this problem. I believe I read that in this thread somewhere: http://thread.gmane.org/gmane.comp.emulators.kvm.devel/94524 or maybe it was here: http://tavi-tech.blogspot.com/2012/03/vga-passthrough-with-kvm-on-fedora-16.html
> 
> I am also back to using KVM, I could make a very long winded post out of this, but XEN just isn't a good fit for me. For one thing the block performance is in the dumps. I followed all the guides and did everything right, and KVM overall out performs it by a good bit. The stability wasn't perfect either, the word "sometimes" can be used a lot ... I think these pictures sum it up for me:
> 
> 
> 
> 
> 
> 
> 
> I think all of the performance problems can be attributed to the fact that dom0 is involved like that, plus XEN has to write their own CPU scheduler etc etc etc, none of which can possibly be as good as what's been invested in and hacked at in the main kernel tree for 15 years.
> 
> With KVM the only problem I have is that I can't reset the guest. This is the same behavior as the XEN XL toolstack. The XEN XM toolstack did work, which tells me the hardware does support it, it's just a matter of getting the right code in place to make everything happy. I need to get the source for XM and take a good look at what it is doing vs what XL / KVM is not doing. I expect this to be fixed by someone possibly myself in the next 3-6 months.
> 
> Also the latest stable amd64 versions of libvirt / qemu in portage had an IRQ issue with this usb controller, the latest testing ~amd64 versions fixed this problem. So upgrade upgrade, latest and greatest is probably what you want if you're on the KVM side.
> 
> There also is the VFIO module and qemu patches, which I will be looking at shortly. It seems like a much more well thought out approach to passthrough. The modules are already in the main kernel tree, the qemu patches should be merged soon I think, and then libvirt support to follow.


Does KVM vfio passthrough work on intel with nvidia?
It seems like a much more promising solution for me.


----------



## dizzy4

Quote:


> Originally Posted by *Xaero252*
> 
> Does KVM vfio passthrough work on intel with nvidia?
> It seems like a much more promising solution for me.


That post was pretty epic, but outdated and biased. I get 95-99% native performance with xen and others have reported it to be superior to KVM. His post makes a lot of assumptions such as kernel hacking being better than one of the largest open source efforts out there. The xen project has code added by LOTS of big names in computing and is a very viable and supported solution.

That post also talks about xen PRE kernel 3.0. With the right kernel before kernel 3.0 the performance was good. The difference is that xen support is now upstream in the kernel so you don't need a special one anymore. Besides, best practices for virtualization usually include not using Domain-0 as a desktop or for other purposes than virtualization and networking.

VFIO is what I am looking into now and should work with KVM and Xen.

All that is to say that Xen is a very viable option and works extremely well.


----------



## nbhusain

Yeah nvidia + vfio works, you can see my archlinux guide if you're interested https://bbs.archlinux.org/viewtopic.php?id=162768


----------



## Natriumatom

Took one last shot at it and got it working with XenServer 6.2SP1









Passed on a set of USB ports, Radeon 6450 VGA and the Radeon HDMI sound.

The only problem I now have is the LG Bluray Writer. I only have a Virtual DVD Drive in the Windows 8.1 VM where I can mount Isos or the actual content of the writer through XenCenter software. The problem is the Bluray Player software doesnt recognize the drive in the VM then.

I only have one SATA controller on the board with all the 6 ports so I cant pass one controller where the bluray drive is attached through.

Anybody has an idea?

Code:



Code:


00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])
        Subsystem: ASRock Incorporation Device 8c02
        Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 1277
        I/O ports at f0d0 [size=8]
        I/O ports at f0c0 [size=4]
        I/O ports at f0b0 [size=8]
        I/O ports at f0a0 [size=4]
        I/O ports at f060 [size=32]
        Memory at f0832000 (32-bit, non-prefetchable) [size=2K]
        Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
        Capabilities: [70] Power Management version 3
        Capabilities: [a8] SATA HBA v1.0
        Kernel driver in use: ahci
        Kernel modules: ahci


----------



## Thorhian

Alright, nevermind my post, I found out my Hardware is supportable (crosshair V Formula plus my Phenom II Thuban). Thanks for the Arch Guide nbhusain


----------



## powerhouse

@Natriumatom: Well, having only one SATA controller sucks. You could get a SATA PCI Express controller card and plug it into a PCI-E slot - they aren't expensive (~$20). I got an ASMedia ASM1062 based card that works well. Once installed, connect the DVD or BR drive and pass it through to Windows.


----------



## Natriumatom

To be honest I got fed up with the whole project and stopped it for this time being.

VMWare ESXi 5.5 - Bluray Worked, GPU passthrough wouldnt for either GPU
Citrix XenServer - GPU and USB passthrough worked, Bluray wouldnt
Linux Mint etc. with Xen - got nothing to work

I just threw Win 8.1 on the machine now and run the media center through this and through VM Workstation I run a Tretflix Appliance.


----------



## Ozieboy

Hi Dizzy,

Thanks for the awesome guide. I really learned a lot and I appreciate the work you put in it. Also thanks to everyone else who contributed further , especially "Powerhouse " and the likes.
Well I got it working sort of:
I got an Intel NUC Haswell, with -i5 and integrated Intel HD 5000 GPU.
I followed the Fedora 20 install, except some typos got through it fine and then everything else and got the Windows 8.1 VM installed using file (NOT LVM).
Got the iGPU passthrough and USB passthrough fine. Rebooted VM a few times, installed windows updates and also GPLPV drivers and it seems to work fine. (Except I tried to install Media portal and it asked for .Net 3.5 features to be enabled, but it seems to fail and wont enable the feature, needs more testing)
So Now I have two issues:
1. When I start or restart the windows 8.1 VM, my screen shows the first initial start window (a window with 4 windows) in the middle of the screen, there is no disk activity that i can notice and it sits there for about 2 minutes and then it starts to get drive activity and windows 8.1 desktop shows in 5-10 secs. So why its taking long time to start the boot process. I am thinking may be installing windows VM in LVM might help, not sure, any help please.
2. The second issue is that I cannot seems to pass through the internal inbuild CIR (infrared) built into the NUC, so that will render it useless if I am using this setup for mediacenter. I suppose I can use the usb IR adapter but I preffer not if I dont have to. Any thoughts?

@ Natriumatom:
Quote:


> Originally Posted by *Natriumatom*
> 
> Took one last shot at it and got it working with XenServer 6.2SP1
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Passed on a set of USB ports, Radeon 6450 VGA and the Radeon HDMI sound.
> 
> The only problem I now have is the LG Bluray Writer. I only have a Virtual DVD Drive in the Windows 8.1 VM where I can mount Isos or the actual content of the writer through XenCenter software. The problem is the Bluray Player software doesnt recognize the drive in the VM then.
> 
> I only have one SATA controller on the board with all the 6 ports so I cant pass one controller where the bluray drive is attached through.
> 
> Anybody has an idea?
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> 00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05) (prog-if 01 [AHCI 1.0])
> Subsystem: ASRock Incorporation Device 8c02
> Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 1277
> I/O ports at f0d0 [size=8]
> I/O ports at f0c0 [size=4]
> I/O ports at f0b0 [size=8]
> I/O ports at f0a0 [size=4]
> I/O ports at f060 [size=32]
> Memory at f0832000 (32-bit, non-prefetchable) [size=2K]
> Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
> Capabilities: [70] Power Management version 3
> Capabilities: [a8] SATA HBA v1.0
> Kernel driver in use: ahci
> Kernel modules: ahci


@ Natriumatom :

I saw you post about Xenserver 6.2 SP1 and how you was successful in passing through the VGA and Audio, this is what I really wanted to do anyway. So I installed xenserver 6.2 SP1.
The VM creating was a breeze and the vm booted fine with windows 8.1. Also worked fine when I tried to enable the .Net 3.5 features.
Now I do have see the GPU icon in the VM properties , but there is no GPU in the server detected. So I cannot pass it through.
Can you please elaborate a bit more on how you was able to get the GPU passthrough. Is it possible to do it in Xencenter or do I need to set it up via the command line. If you like you can link me to the resource you used to set it up or please give some steps on setting it up.

Best Regards,


----------



## dizzy4

Quote:


> Originally Posted by *Ozieboy*
> 
> Hi Dizzy,
> 
> Thanks for the awesome guide. I really learned a lot and I appreciate the work you put in it. Also thanks to everyone else who contributed further , especially "Powerhouse " and the likes.
> Well I got it working sort of:
> I got an Intel NUC Haswell, with -i5 and integrated Intel HD 5000 GPU.
> I followed the Fedora 20 install, except some typos got through it fine and then everything else and got the Windows 8.1 VM installed using file (NOT LVM).
> Got the iGPU passthrough and USB passthrough fine. Rebooted VM a few times, installed windows updates and also GPLPV drivers and it seems to work fine. (Except I tried to install Media portal and it asked for .Net 3.5 features to be enabled, but it seems to fail and wont enable the feature, needs more testing)
> So Now I have two issues:
> 1. When I start or restart the windows 8.1 VM, my screen shows the first initial start window (a window with 4 windows) in the middle of the screen, there is no disk activity that i can notice and it sits there for about 2 minutes and then it starts to get drive activity and windows 8.1 desktop shows in 5-10 secs. So why its taking long time to start the boot process. I am thinking may be installing windows VM in LVM might help, not sure, any help please.
> 2. The second issue is that I cannot seems to pass through the internal inbuild CIR (infrared) built into the NUC, so that will render it useless if I am using this setup for mediacenter. I suppose I can use the usb IR adapter but I preffer not if I dont have to. Any thoughts?
> 
> Best Regards,


Glad to hear it went so smoothly for the intel nuc! I will take a look and see if I can catch those typos.

1. I had issues enabling .net in windows 8 when I first upgraded, but it was a windows issue and had noting to do with the VM. After some long calls to Msoft they told me to just wipe and install fresh which worked. You should be using the "turn windows features on or off" It's easy to find with the search. My problem was that it was blank.

2. Are you sure that the IR device is PCI or USB? In my experience they are often just i2c interfaces. It might also be contained in another pci device that is part of a larger chipset. Virtio might actually work well for you, but my knowledge of virtio is a little lacking for now.


----------



## Mshenay

http://pcpartpicker.com/user/Mshenay/saved/2OoR

My updated parts list, any one have any tips? Going for Air in this unit, ditching Wireless all together, and I'd like to OC the CPU to at least a stable 4.05

I'm hoping with the Full Tower, I can get some SERIOUS Air flow in there... now time to read up on case management via flow >.>

I know I should post this in another thread, but I am building this machine to work as you have intended. A Liniux Machine running full windows Viritulization [So I can game on my Linux Machine <3]

non the less, I'm ofc open to any suggestions, mostly for the case, and for the air flow [and processor]


----------



## Ozieboy

Quote:


> Originally Posted by *dizzy4*
> 
> Glad to hear it went so smoothly for the intel nuc! I will take a look and see if I can catch those typos.
> 
> 1. I had issues enabling .net in windows 8 when I first upgraded, but it was a windows issue and had noting to do with the VM. After some long calls to Msoft they told me to just wipe and install fresh which worked. You should be using the "turn windows features on or off" It's easy to find with the search. My problem was that it was blank.
> 
> 2. Are you sure that the IR device is PCI or USB? In my experience they are often just i2c interfaces. It might also be contained in another pci device that is part of a larger chipset. Virtio might actually work well for you, but my knowledge of virtio is a little lacking for now.


Hi Dizzy,

Thanks for your reply. I am actually not too much concerned about the .Net issues at the moment.
The main issue that I have is that windows take too long to start the boot process once its started. Not sure if its the VM or Xen.
Can you please shed some light on "Virtio". Is it like Xenserver or another package to install on top of the xen package on fedora.
I also tried debian jessie at one stage too. Went fine, Igpu passthrough to windows but then windows could not allocate any resources to the intel HD 5000 and just shows an ! mark under device drivers.
However i did noticed one difference, with Debian windows VM I could also see a USB port (device) emulated in device manager, however with
Fedora in the windows VM there is no usb device in device manager ( this is without passing through any physical usb device ).
About Virtio again, is it your recommendation for my CIR infrared device issues or windows boot time issues ?
I know you said that you do not have much exp with Virtio, but can you please link me to a good guide for that, I will give that a go.
I just tried ununtu server 13.10 last night, not sure whats going on but windows will just stay on the window and wont boot any further to the installation. I am really really confortable with Debian based systems, so really like that one to succeed.
One another question: can the VM config file created for Fedora be used for the VM config file for debian/ubuntu/Mint etc ?
I think i've asked enough for today.
Regards and Thanks.


----------



## Mshenay

any word on which Haswell CPUS work with the instruction sets we needed?

I'm doing a new build here in the fall as you guys know, and I'm wondering if I should go with Haswell [not sure which chip set] Ivy [3770] or Sandy [3820] any tips on which of these offers the best bang for the buck, espically when used for running a virtualized Windows enviorment


----------



## powerhouse

Quote:


> Originally Posted by *Mshenay*
> 
> http://pcpartpicker.com/user/Mshenay/saved/2OoR
> 
> My updated parts list, any one have any tips? Going for Air in this unit, ditching Wireless all together, and I'd like to OC the CPU to at least a stable 4.05
> 
> I'm hoping with the Full Tower, I can get some SERIOUS Air flow in there... now time to read up on case management via flow >.>
> 
> I know I should post this in another thread, but I am building this machine to work as you have intended. A Liniux Machine running full windows Viritulization [So I can game on my Linux Machine <3]
> 
> non the less, I'm ofc open to any suggestions, mostly for the case, and for the air flow [and processor]


The CPU won't work !!! See http://ark.intel.com/search/advanced/?s=t&VTD=true for a list of VT-d enabled Intel CPUs.

The i7 4770 will be OK, but won't overclock. If you are into overclocking, I recommend going with a 2011 board and a 3930K or 4930K processor. If you choose the latter (4930K), make absolutely sure the motherboard BIOS supports that CPU with VT-d. The Asus Rampage IV Extreme with the latest BIOS does support VT-d when using (not so most other Asus boards !!! - see also here). It's probably best to check ASrock as they have reportedly a good record for supporting VT-d.


----------



## powerhouse

Quote:


> Originally Posted by *Ozieboy*
> 
> ...I just tried ununtu server 13.10 last night, not sure whats going on but windows will just stay on the window and wont boot any further to the installation. I am really really confortable with Debian based systems, so really like that one to succeed.
> One another question: can the VM config file created for Fedora be used for the VM config file for debian/ubuntu/Mint etc ?
> I think i've asked enough for today.
> Regards and Thanks.


For a Ubuntu-based system, you could try my how-to here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It should work likewise with Linux Mint 16 and Ubuntu 13.10. My how-to uses the "xm" (xend) toolstack, not the newer "xl" toolstack - the reason why I chose xm over xl is explained there.

Regarding VM config file: there might be some differences between xl and xm toolstack and their configuration files, but generally speaking you can use the config with any distro.


----------



## Mshenay

Quote:


> Originally Posted by *powerhouse*
> 
> The CPU won't work !!! See http://ark.intel.com/search/advanced/?s=t&VTD=true for a list of VT-d enabled Intel CPUs.
> 
> The i7 4770 will be OK, but won't overclock. If you are into overclocking, I recommend going with a 2011 board and a 3930K or 4930K processor. If you choose the latter (4930K), make absolutely sure the motherboard BIOS supports that CPU with VT-d. The Asus Rampage IV Extreme with the latest BIOS does support VT-d when using (not so most other Asus boards !!! - see also here). It's probably best to check ASrock as they have reportedly a good record for supporting VT-d.


Indeed it wont, why can't you OC a 4770, is it due to the low tCore limit

Hell ata max limit 77c I'd be terrified to even stick it into my rig, have they done anything to help it keep lower temps?

Although It may be possible to achive a stable 4.05 with it and keep temps down. I didn't tweak the voltage enough but my 3450k was able to hit 4.1 for about an hour

anyways $300 is my limit for a processor sadly


----------



## powerhouse

Quote:


> Originally Posted by *Mshenay*
> 
> Indeed it wont, why can't you OC a 4770, is it due to the low tCore limit
> 
> Hell ata max limit 77c I'd be terrified to even stick it into my rig, have they done anything to help it keep lower temps?
> 
> Although It may be possible to achive a stable 4.05 with it and keep temps down. I didn't tweak the voltage enough but my 3450k was able to hit 4.1 for about an hour
> 
> anyways $300 is my limit for a processor sadly


I'm no overclocker so I'm afraid I can't comment on the 4770 and its (missing?) overclocking capabilities. I just know from various references that the 4930K, 3930K etc. can be overclocked - and they support VT-d.

However, the absolutely essential part is VT-d.


----------



## fergalom

Possibly a stupid question but can I upgrade to F20 and Xen 4.3 from my current dom0 setup (F18 & Xen 4.1) without much issue?

Is there anything special required such as removing Xen 4.1 first before upgrading from F18 to F20?


----------



## powerhouse

Quote:


> Originally Posted by *fergalom*
> 
> Possibly a stupid question but can I upgrade to F20 and Xen 4.3 from my current dom0 setup (F18 & Xen 4.1) without much issue?
> 
> Is there anything special required such as removing Xen 4.1 first before upgrading from F18 to F20?


I don't think there are stupid questions, but I sure hope I don't post a stupid answer. First of all, I can't really answer the question, as I'm not familiar with Fedora. So here is what I can say on upgrading Linux based on ~17 years Linux experience:

1. Make sure you have a full backup of your current installation, in case things go wrong (and they do occasionally)!
2. Never touch a running system - if F18/Xen 4.1 works fine and does what you need it to do, I wouldn't bother upgrading to F20. The only reasons for upgrading a distro would be if I have unsupported hardware (and the upgrade fixes it), or if I need applications or features that aren't supported in my installed version. I wouldn't upgrade just because there is a newer release available.
3. Most Linux distros allow 2 ways of upgrading: new installation and package upgrade. Though upgrading the packages seems like the natural way to upgrade, it can be problematic. The founder of Linux Mint describes the two methods and the pros and cons here. Using Linux Mint now, I prefer the "new installation" method and always keep a separate home partition - this way I have all my desktop settings in place, in addition to my data.

Fedora starting with F18 offers an upgrade utility called FedUp, see https://fedoraproject.org/wiki/FedUp. Pay special attention to this: https://fedoraproject.org/wiki/FedUp#Why_does_my_upgrade_to_Fedora_20_fail_.28immediately_reboot_to_my_old_Fedora.29.3F.

The FedUp upgrade should not require you to remove anything, but I don't have any experience with it. Hope someone really familiar with this jumps in.

Whatever you do, the first step is to make a backup of your entire system!


----------



## Ozieboy

Quote:


> Originally Posted by *powerhouse*
> 
> For a Ubuntu-based system, you could try my how-to here: http://forums.linuxmint.com/viewtopic.php?f=42&t=112013. It should work likewise with Linux Mint 16 and Ubuntu 13.10. My how-to uses the "xm" (xend) toolstack, not the newer "xl" toolstack - the reason why I chose xm over xl is explained there.
> 
> Regarding VM config file: there might be some differences between xl and xm toolstack and their configuration files, but generally speaking you can use the config with any distro.


Hi Powerhouse,

Thanks for your reply. I now have the windows 8.1 VM running successfully ( without the passthrough) .
This is where I am stuck now at the pciback :
if i run sudo echo "xen-pciback passthrough=1 hide=(00:1d.0)" >> /etc/initramfs-tools/modules

i get : -bash: /etc/initramfs-tools/modules: Permission denied

i ran : modprobe pciback
FATAL: Module pciback not found.

So , does it mean that I do not have the pciback installed, if not then how can i install it.

Please help, as this is driving me nuts.

Best Regards.


----------



## powerhouse

Quote:


> Originally Posted by *Ozieboy*
> 
> Hi Powerhouse,
> 
> Thanks for your reply. I now have the windows 8.1 VM running successfully ( without the passthrough) .
> This is where I am stuck now at the pciback :
> if i run sudo echo "xen-pciback passthrough=1 hide=(00:1d.0)" >> /etc/initramfs-tools/modules
> 
> i get : -bash: /etc/initramfs-tools/modules: Permission denied
> 
> i ran : modprobe pciback
> FATAL: Module pciback not found.
> 
> So , does it mean that I do not have the pciback installed, if not then how can i install it.
> 
> Please help, as this is driving me nuts.
> 
> Best Regards.


1. Check that /etc/initramfs-tools/modules is not set to read-only. You can also edit the file using your favorite editor, just make sure you are root. To check its content, use

Code:



Code:


cat /etc/initramfs-tools/modules

2. First check this:

Code:



Code:


lsmod | grep pciback

The output should look like:

Code:



Code:


[email protected] ~ $ lsmod | grep pciback
xen_pciback            55503  0

To load the pciback module, use:

Code:



Code:


modprobe xen-pciback


----------



## Ozieboy

Hi,

Thanks Powerhouse. I did manged to setup Ubuntu per your instructions, But it seems that XM toolstack have issues with Integrated Intel GPU. So I have to go back to XL.

Dizzy: There is a new Kernal out and so is the new Xen 4.3.2 and because of that it seems that the Kernal and headers are no longer working. Would it be too much of a hassle for you to upload the latest kernal with modules built in, please?
OR , can you please link me to a guide to make my own kernal and headers with latest kernal, xen and pciback built in.

Thanks in Advance.


----------



## powerhouse

Hello Ozieboy,

Just yesterday I switched to the xl toolstack myself. In your situation I would give it a try using xl with the standard kernel / Xen for Ubuntu 13.10, which means Xen 4.3. Another thing is that I've seen reports that Windows 8.1 may have issues with xm, so if you are using Windows 8.1 the xl toolstack might solve the issue.

You will find an update on xl in my how-to (at step 3) and in my last post in that thread. If you already installed everything it's easy to try out xl:
1. Edit your /etc/xen/win.cfg file (replace the file name with yours). Comment out

Code:



Code:


#device_model = '/usr/lib/xen-default/bin/qemu-dm'

and add

Code:



Code:


device_model_version = 'qemu-xen-traditional'

2. Edit /etc/default/xen and change to the following:

Code:



Code:


TOOLSTACK="xl"

3. Reboot your computer.

4. Boot your Windows guest with

Code:



Code:


xl create /etc/xen/win.cfg

See if it works.

EDIT: See the last few posts in my how-to thread. Also, I just updated it to include the steps above. It seems that Windows 8.1 requires xl for VGA passthrough to work.

Furthermore, since you use the IGP, you should not run into the performance/stability issues after domU shutdown and reboot that were reported by AMD graphics card users.


----------



## shruikan

hi
I am new here (and new in linux, and xen







) and got some trouble:
at:

Code:



Code:


sudo xl create win8.sxp

system respond:

Code:



Code:


libxl: error: libxl.c:91:libxl_ctx_alloc: Is xenstore daemon running?
failed to stat /var/run/xenstored.pid: No such file or directory
cannot init xl context

and after reboot there is no internet connection so:

Code:



Code:


sudo service network restart

every start, after that its work fine

if anyone can help i'd appreciate









(Ps.: sorry for my bad english)


----------



## Irritation

A little off topic, but i'm running OSX NAtively on my Hackintosh and Win7 and Linux is soley VM installed via Parallels. I love VM, def going to give THIS a try when i feel comfortable enough with a linux system to ditch windows.


----------



## tjbanf

Other than using Fedora, how does this guide differ from Xen's documentation on creating a HVM? Would the steps laid out in this guide be more capable than those put forth by Xen here?


----------



## powerhouse

Quote:


> Originally Posted by *tjbanf*
> 
> Other than using Fedora, how does this guide differ from Xen's documentation on creating a HVM? Would the steps laid out in this guide be more capable than those put forth by Xen here?


This guide by dizzy4 specifically describes how to set up your PC to achieve VGA passthrough. VGA passthrough is a method allowing your (Windows) guest to gain direct access to a dedicated graphics adaptor. This allows you to run a VM with graphics performance that is the same or very very close to having the same operating system (like Windows) run on bare metal (that is directly on the PC hardware, i.e. not as a VM).

If you don't need this graphic performance, for example for simple office applications or servers, a regular HVM or PV installation is enough. The Xen wiki also describes VGA passthrough, but this how-to and thread accumulates a lot of experience. Unfortunately with VGA passthrough things aren't always as easy as they seem.

On a practical note, read as much as you can before you attempt VGA passthrough.


----------



## powerhouse

For those interested, I tested Xen 4.4 on a 3.13 kernel running the latest Ubuntu 14.04 beta2 release. Similar kernel/Xen releases should be available through other distributions. Here my observations with regard to VGA passthrough using a AMD / ATI HD 6450 video card:

1. xl toolstack works now: no problems rebooting the domU - no performance loss, no crashes (dom0 or domU)
2. To use VGA passthrough one still needs to select device_model_version = 'qemu-xen-traditional'
3. When installing the AMD graphics driver, select Custom installation, then deselect the AMD Catalyst Control Center (CCC), else you might get a BSOD

In general I'm pleased with Xen 4.4 and its support for VGA passthrough, particularly the improvement with AMD cards and the xl toolstack. Finally the point has been reached where the xm toolstack can be discarded. It's just a pity that the new qemu-xen (upstream) still doesn't support VGA passthrough. Perhaps Xen 4.5 addresses this?

More information on the tests can be found here.


----------



## cheuk3

Hi, if domain0 upgrade or change to other distro, does the vm need to reinstall?


----------



## powerhouse

Quote:


> Originally Posted by *cheuk3*
> 
> Hi, if domain0 upgrade or change to other distro, does the vm need to reinstall?


I have upgraded the dom0 several times and no, there is no need to reinstall your VM. Just make sure you save your config files (VM, Xen, networking, grub, initram, whatever) and scripts (if any). The same should go for changing the distro. You can also change from xm to xl toolstack and vice versa.


----------



## mdocod

Bump for an amazing thread about a seriously awesome use of an IOMMU. Will be bookmarking for future reference.


----------



## vuldin

I've made alot of progress on getting a Windows VM to have control over a 290x using Xen, but it's not yet working properly. I have a ton of details I can share on what I've done, but for now I have some specific questions for those who have gone through these steps successfuly on similar hardware.

None of my PCI devices (including the 290x) shows FLR (function level reset) enabled. Is this a requirement for all GPUs that will be passed to a VM or is it just for swapping the card between dom0 and domU?I found this out by running *sudo lspci -vv* looking for my 290x and then finding the section starting with 'DevCap' (FLReset- is present). In fact, all FLReset variables in this output are negative for all devices. First I need to ensure this is critical to successful PCI passthrouth. If so, then this makes me think I need to either 1) enable something in the BIOS or 2) change a kernel or module parameter. I've heard that some motherboards (for instance the Intel DH55HC) have an option to enable FLR in the chipset section of the BIOS. My motherboard (ASRock Z97 Extreme6) has VT-d support, but doesn't have any FLR option in the BIOS.

Does anyone have FLReset+ next to their GPUs?

Here is the output of of *xl dmesg*:

Code:



Code:


 Xen 4.4.0
(XEN) Xen version 4.4.0 ([email protected]) (gcc (GCC) 4.9.0 20140521 (prerelease)) debug=n Sat May 31 15:52:14 PDT 2014
(XEN) Latest ChangeSet: 
(XEN) Bootloader: GRUB 2.02~beta2
(XEN) Command line: /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1 dom0_mem=16G xsave=1
(XEN) Video information:
(XEN)  VGA is text mode 80x25, font 8x16
(XEN)  VBE/DDC methods: V2; EDID transfer time: 1 seconds
(XEN) Disc information:
(XEN)  Found 0 MBR signatures
(XEN)  Found 0 EDD information structures
(XEN) Multiboot-e820 RAM map:
(XEN)  0000000000000000 - 0000000000058000 (usable)
(XEN)  0000000000058000 - 0000000000059000 (reserved)
(XEN)  0000000000059000 - 000000000009f000 (usable)
(XEN)  000000000009f000 - 00000000000a0000 (reserved)
(XEN)  0000000000100000 - 0000000069e4c000 (usable)
(XEN)  0000000069e4c000 - 0000000069e53000 (ACPI NVS)
(XEN)  0000000069e53000 - 000000006ab54000 (usable)
(XEN)  000000006ab54000 - 000000006b085000 (reserved)
(XEN)  000000006b085000 - 000000007d185000 (usable)
(XEN)  000000007d185000 - 000000007d216000 (reserved)
(XEN)  000000007d216000 - 000000007d25d000 (usable)
(XEN)  000000007d25d000 - 000000007d398000 (ACPI NVS)
(XEN)  000000007d398000 - 000000007df7d000 (reserved)
(XEN)  000000007df7d000 - 000000007dfff000 type 20
(XEN)  000000007dfff000 - 000000007e000000 (usable)
(XEN)  000000007f000000 - 00000000bf200000 (reserved)
(XEN)  00000000f0000000 - 00000000f8000000 (reserved)
(XEN)  00000000fec00000 - 00000000fec01000 (reserved)
(XEN)  00000000fed00000 - 00000000fed04000 (reserved)
(XEN)  00000000fed1c000 - 00000000fed20000 (reserved)
(XEN)  00000000fee00000 - 00000000fee01000 (reserved)
(XEN)  00000000ff000000 - 0000000100000000 (reserved)
(XEN)  0000000100000000 - 000000083fe00000 (usable)
(XEN) ACPI: RSDP 000F0010, 0024 (r2 ALASKA)
(XEN) ACPI: XSDT 7D365088, 0094 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FACP 7D376090, 010C (r5 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DSDT 7D3651B8, 10ED6 (r2 ALASKA    A M I       35 INTL 20120711)
(XEN) ACPI: FACS 7D397F80, 0040
(XEN) ACPI: APIC 7D3761A0, 0092 (r3 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: FPDT 7D376238, 0044 (r1 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: SSDT 7D376280, 0539 (r1  PmRef  Cpu0Ist     3000 INTL 20051117)
(XEN) ACPI: SSDT 7D3767C0, 0B74 (r1 CpuRef  CpuSsdt     3000 INTL 20051117)
(XEN) ACPI: SSDT 7D377338, 01C7 (r1  PmRef LakeTiny     3000 INTL 20051117)
(XEN) ACPI: MCFG 7D377500, 003C (r1 ALASKA    A M I  1072009 MSFT       97)
(XEN) ACPI: HPET 7D377540, 0038 (r1 ALASKA    A M I  1072009 AMI.        5)
(XEN) ACPI: SSDT 7D377578, 036D (r1 SataRe SataTabl     1000 INTL 20120711)
(XEN) ACPI: SSDT 7D3778E8, 5B5E (r1 SaSsdt  SaSsdt      3000 INTL 20120711)
(XEN) ACPI: AAFT 7D37D448, 049F (r1 ALASKA OEMAAFT   1072009 MSFT       97)
(XEN) ACPI: BGRT 7D37D8E8, 0038 (r0 ALASKA    A M I  1072009 AMI     10013)
(XEN) ACPI: DMAR 7D37D920, 00B8 (r1 INTEL      BDW         1 INTL        1)
(XEN) ACPI: SSDT 7D37D9D8, 0579 (r1 Intel_ IsctTabl     1000 INTL 20120711)
(XEN) System RAM: 31690MB (32450764kB)
(XEN) Domain heap initialised
(XEN) ACPI: 32/64X FACS address mismatch in FADT - 7d397f80/0000000000000000, using 32
(XEN) Processor #0 7:12 APIC version 21
(XEN) Processor #2 7:12 APIC version 21
(XEN) Processor #4 7:12 APIC version 21
(XEN) Processor #6 7:12 APIC version 21
(XEN) Processor #1 7:12 APIC version 21
(XEN) Processor #3 7:12 APIC version 21
(XEN) Processor #5 7:12 APIC version 21
(XEN) Processor #7 7:12 APIC version 21
(XEN) IOAPIC[0]: apic_id 8, version 32, address 0xfec00000, GSI 0-23
(XEN) Enabling APIC mode:  Flat.  Using 1 I/O APICs
(XEN) Failed to enable Interrupt Remapping: Will not enable x2APIC.
(XEN) Using scheduler: SMP Credit Scheduler (credit)
(XEN) Detected 3598.990 MHz processor.
(XEN) Initing memory sharing.
(XEN) xstate_init: using cntxt_size: 0x340 and states: 0x7
(XEN) spurious 8259A interrupt: IRQ7.
(XEN) Intel VT-d iommu 0 supported page sizes: 4kB.
(XEN) Intel VT-d iommu 1 supported page sizes: 4kB, 2MB, 1GB.
(XEN) Intel VT-d Snoop Control not enabled.
(XEN) Intel VT-d Dom0 DMA Passthrough not enabled.
(XEN) Intel VT-d Queued Invalidation enabled.
(XEN) Intel VT-d Interrupt Remapping enabled.
(XEN) Intel VT-d Shared EPT tables not enabled.
(XEN) I/O virtualisation enabled
(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) Platform timer is 14.318MHz HPET
(XEN) Allocated console ring of 16 KiB.
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN)  - VMCS shadowing
(XEN) HVM: ASIDs enabled.
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) Brought up 8 CPUs
(XEN) *** LOADING DOMAIN 0 ***
(XEN)  Xen  kernel: 64-bit, lsb, compat32
(XEN)  Dom0 kernel: 64-bit, PAE, lsb, paddr 0x1000000 -> 0x1f5b000
(XEN) PHYSICAL MEMORY ARRANGEMENT:
(XEN)  Dom0 alloc.:   0000000820000000->0000000828000000 (4158762 pages to be allocated)
(XEN)  Init. ramdisk: 000000083f32a000->000000083fdff200
(XEN) VIRTUAL MEMORY ARRANGEMENT:
(XEN)  Loaded kernel: ffffffff81000000->ffffffff81f5b000
(XEN)  Init. ramdisk: ffffffff81f5b000->ffffffff82a30200
(XEN)  Phys-Mach map: ffffffff82a31000->ffffffff84a31000
(XEN)  Start info:    ffffffff84a31000->ffffffff84a314b4
(XEN)  Page tables:   ffffffff84a32000->ffffffff84a5b000
(XEN)  Boot stack:    ffffffff84a5b000->ffffffff84a5c000
(XEN)  TOTAL:         ffffffff80000000->ffffffff84c00000
(XEN)  ENTRY ADDRESS: ffffffff818ee1f0
(XEN) Dom0 has maximum 8 VCPUs
(XEN) Scrubbing Free RAM: .....................................................................................................................................................done.
(XEN) Initial low memory virq threshold set at 0x4000 pages.
(XEN) Std. Loglevel: Errors and warnings
(XEN) Guest Loglevel: Nothing (Rate-limited: Errors and warnings)
(XEN) Xen is relinquishing VGA console.
(XEN) *** Serial input -> DOM0 (type 'CTRL-a' three times to switch input to Xen)
(XEN) Freed 260kB init memory.

Here is my /etc/xen/vwin.hvm (vm config):

Code:



Code:


name = 'vwin'
builder = 'hvm'
memory = 8192
vcpus = 2
disk = [ '/home/joshua/vms/vwin.img,raw,xvda,w', 'file:/home/joshua/vms/win81.iso,hdc:cdrom,r' ]
vif = [ 'bridge=xenbr0,ip=192.168.1.115' ]
vnclisten = '0.0.0.0'
vnc = 1
pci = [ '01:00.0', '01:00.1' ]
pci_permissive = 1

When I successfully start the VM, here is the output:

Code:



Code:


Parsing config from /etc/xen/vwin.hvm
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1

The above is what first made me think there is an issue (which led me to find out more regarding FLR). When Windows boots I see the 290x, and when I run the AMD test to see what hardware I have it finds it as being a 290x. I can install the drivers for the card and no errors are reported. I reboot and look at Device Manager and I see the following message associated with the problem 290x device:

Code:



Code:


This device cannot find enough free resources that it can use. (Code 12)

I've looked around for other devices which may be conflicting with the 290x, and there is one other device which has no drivers currently. Other than that there is another Microsoft display device in the same section of the list next to the 290x. Disabling one or both of these devices and rebooting has no effect.

Here's the relevant portion of my grub config:

Code:



Code:


multiboot /xen-4.4.0.gz /xen-4.4.0.gz iommu=1 iommu_inclusive_mapping=1 dom0_mem=16G xsave=1
module /vmlinuz-linux /vmlinuz-linux root=UUID=917153ed-f386-4987-8fd0-9e293f8d969a rw  modprobe.blacklist=radeon i915 quiet rootflags=subvol=rootvol console=tty0 xen-pciback.passthrough=1 xen-pciback.hide=(01:00.0)(01:00.1)

I think 'iommu=1' is related to previous versions of Xen, and I'm not sure what 'iommu_inclusive_mapping=1' even does. I haven't verified that I get the same current functionality without these lines yet.

Please request any other details if you think it would be helpful. This is an up-to-date archlinux, Xen 4.4.0-4 system (specs in sig).


----------



## zirblazer

Quote:


> Originally Posted by *vuldin*
> 
> None of my PCI devices (including the 290x) shows FLR (function level reset) enabled. Is this a requirement for all GPUs that will be passed to a VM or is it just for swapping the card between dom0 and domU?I found this out by running *sudo lspci -vv* looking for my 290x and then finding the section starting with 'DevCap' (FLReset- is present). In fact, all FLReset variables in this output are negative for all devices. First I need to ensure this is critical to successful PCI passthrouth. If so, then this makes me think I need to either 1) enable something in the BIOS or 2) change a kernel or module parameter. I've heard that some motherboards (for instance the Intel DH55HC) have an option to enable FLR in the chipset section of the BIOS. My motherboard (ASRock Z97 Extreme6) has VT-d support, but doesn't have any FLR option in the BIOS.
> 
> Does anyone have FLReset+ next to their GPUs?


FLReset- means that it doesn't support it, FLReset+ that it does. No Radeons support FLR that I'm aware of, and on Xen Wiki it says very few devices actually support it. nVidia Quadro supports it for sure, GeForces no idea. I think it was one of the reasons why some users were modding GeForces into Quadros or GRIDs.
FLReset works pretty much as Hardware reset for hotplugging. If I recall correctly, there was some AMD guy that did some coding for Xen and had Software patch for that. The Xen 4.3.1 package that was in Arch Linux AUR included a file named ati_passthrough.patch, which I recall having applied during installation by uncommenting it from the PKGBUILD file.
One of the issues with Radeons was that restarting the VM could cause degraded GPU performance due to it not reinitializing properly (As the Video Card was not in a fresh state when the VM was restarted), but I never had such performance issue. What I DID notice with my Radeon 5770, is that sometimes GPU-Z wouldn't be able to read the Voltage of the card and it is always running at the default Frequency without switching Power States (Temperature sensors works and is much higher than what it should Idle at on the minimum Power State). Yet if I restart the VM, it may suddently pick it good again. So I may have to restart the VM, even if its the first one created after booting, until I see Voltage reading and switching Power States.

Quote:


> Originally Posted by *vuldin*
> 
> Here is my /etc/xen/vwin.hvm (vm config):
> 
> Code:
> 
> 
> 
> Code:
> 
> 
> name = 'vwin'
> builder = 'hvm'
> memory = 8192
> vcpus = 2
> disk = [ '/home/joshua/vms/vwin.img,raw,xvda,w', 'file:/home/joshua/vms/win81.iso,hdc:cdrom,r' ]
> vif = [ 'bridge=xenbr0,ip=192.168.1.115' ]
> vnclisten = '0.0.0.0'
> vnc = 1
> pci = [ '01:00.0', '01:00.1' ]
> pci_permissive = 1


I think that you're missing there...

device_model_version = "qemu-xen-traditional"

Because the error you said was caused by that. Default qemu does NOT work with VGA Passthrough.


----------



## vuldin

Thanks for the response, and you found something that I thought may also be the issue since I posted previously. In fact I just found out today that the built-in version of qemu that comes with the latest xen still doesn't have vga passthrough capability. I went through the entire install again and this time used the qemu-xen-traditional model version, but I am still getting similar results (with the following messages when I create the VM):

Code:



Code:


Parsing config from /etc/xen/vwin.hvm
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.0
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:01:00.1

There is a difference though... when I create the VM (after ensuring the pci line is inserted into the config) the VM never boots up fully. No signal is sent through the GPU, instead on the VNC session I get the intial Windows symbol and the white circles spinning below. This continues for a few minutes before the screen goes black and I then can't do anything but restart.

I get potentially interesting errors in /var/log/xen/qemu-dm-winimtsy.log:


Spoiler: big log



Code:



Code:


domid: 9
Strip off blktap sub-type prefix to /home/joshua/vms/winimtsy.img (drv 'aio')
Using file /home/joshua/vms/winimtsy.img in read-write mode
Strip off blktap sub-type prefix to /home/joshua/vms/win81.iso (drv 'aio')
Using file /home/joshua/vms/win81.iso in read-only mode
Watching /local/domain/0/device-model/9/logdirty/cmd
Watching /local/domain/0/device-model/9/command
Watching /local/domain/9/cpu
qemu_map_cache_init nr_buckets = 10000 size 4194304
shared page at pfn feffd
buffered io page at pfn feffb
Guest uuid = 06baea53-cc6e-4a1f-9129-7e1dd2cf36f0
populating video RAM at ff000000
mapping video RAM from ff000000
Register xen platform.
Done register platform.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
xs_read(/local/domain/0/device-model/9/xen_extended_power_mgmt): read error
xs_read(): vncpasswd get error. /vm/06baea53-cc6e-4a1f-9129-7e1dd2cf36f0/vncpasswd.
medium change watch on `hdc' (index: 1): aio:/home/joshua/vms/win81.iso
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
Log-dirty: no command yet.
I/O request not ready: 0, ptr: 0, port: 0, data: 0, count: 0, size: 0
vcpu-set: watch node error.
[xenstore_process_vcpu_set_event]: /local/domain/9/cpu has no CPU!
xs_read(/local/domain/9/log-throttling): read error
qemu: ignoring not-understood drive `/local/domain/9/log-throttling'
medium change watch on `/local/domain/9/log-throttling' - unknown device, ignored
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 01:00.0 ...
register_real_device: Disable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x0
pt_register_regions: IO region registered (size=0x10000000 base_addr=0xd000000c)
pt_register_regions: IO region registered (size=0x00800000 base_addr=0xe000000c)
pt_register_regions: IO region registered (size=0x00000100 base_addr=0x0000e001)
pt_register_regions: IO region registered (size=0x00040000 base_addr=0xefc00000)
pt_register_regions: Expansion ROM registered (size=0x00020000 base_addr=0xefc40000)
pci_intx: intx=1
register_real_device: Real physical device 01:00.0 registered successfuly!
IRQ type = INTx
dm-command: hot insert pass-through pci dev 
register_real_device: Assigning real physical device 01:00.1 ...
register_real_device: Disable MSI translation via per device option
register_real_device: Disable power management
pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x1
pt_register_regions: IO region registered (size=0x00004000 base_addr=0xefc60004)
pci_intx: intx=2
register_real_device: Real physical device 01:00.1 registered successfuly!
IRQ type = INTx
pt_iomem_map: e_phys=e0000000 maddr=d0000000 type=8 len=268435456 index=0 first_map=1
cirrus vga map change while on lfb mode
pt_iomem_map: e_phys=f3000000 maddr=e0000000 type=8 len=8388608 index=2 first_map=1
pt_iomem_map: e_phys=f3800000 maddr=efc00000 type=0 len=262144 index=5 first_map=1
pt_iomem_map: e_phys=f3860000 maddr=efc60000 type=0 len=16384 index=0 first_map=1
pt_ioport_map: e_phys=c200 pio_base=e000 len=256 index=4 first_map=1
mapping vram to f0000000 - f0400000
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is rw state.
platform_fixed_ioport: changed ro/rw state of ROM memory area. now is ro state.
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:05:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
pt_pci_read_config: [00:06:0] Error: Failed to read register with invalid access size alignment. [Offset:0eh][Length:4]
ACPI:debug: write addr=0xb044, val=0x0.





The messages like the following:

Code:



Code:


pt_iomul_init: Error: pt_iomul_init can't open file /dev/xen/pci_iomul: No such file or directory: 0x1:0x0.0x1

may not be a concern, since a guy who testing working VGA passthrough on win8 also had these messages in his log:

http://wiki.xensource.com/mediawiki/images/6/63/Xen_VGA_Passthrough_to_Windows_8_Consumer_Preview_64-bit_English_HVM_domU_and_Windows_XP_Home_Edition_SP3_HVM_domU_with_Xen_4.2-unstable_Changeset_25070_and_Linux_Kernel_3.3.0_in_Ubuntu_11.10_oneiric_ocelot_amd64_Final_Release_Dom0.pdf

The errors at the very end related to 'Failed to read register with invalid access size alignment' seems more worrisome. In fact it looks like powerhouse has been involved in a thread on the xen mailing list related to this issue:

http://xen.1045712.n5.nabble.com/VGA-Passthrough-in-Xen-4-4-td5722323.html

I may need to revert to a previous verson of xen as next steps.


----------



## opt1k

Quote:


> Originally Posted by *zirblazer*
> 
> FLReset- means that it doesn't support it, FLReset+ that it does. No Radeons support FLR that I'm aware of, and on Xen Wiki it says very few devices actually support it. nVidia Quadro supports it for sure, GeForces no idea. I think it was one of the reasons why some users were modding GeForces into Quadros or GRIDs.
> FLReset works pretty much as Hardware reset for hotplugging. If I recall correctly, there was some AMD guy that did some coding for Xen and had Software patch for that. The Xen 4.3.1 package that was in Arch Linux AUR included a file named ati_passthrough.patch, which I recall having applied during installation by uncommenting it from the PKGBUILD file.
> One of the issues with Radeons was that restarting the VM could cause degraded GPU performance due to it not reinitializing properly (As the Video Card was not in a fresh state when the VM was restarted), but I never had such performance issue. What I DID notice with my Radeon 5770, is that sometimes GPU-Z wouldn't be able to read the Voltage of the card and it is always running at the default Frequency without switching Power States (Temperature sensors works and is much higher than what it should Idle at on the minimum Power State). Yet if I restart the VM, it may suddently pick it good again. So I may have to restart the VM, even if its the first one created after booting, until I see Voltage reading and switching Power States.
> I think that you're missing there...
> 
> device_model_version = "qemu-xen-traditional"
> 
> Because the error you said was caused by that. Default qemu does NOT work with VGA Passthrough.


So, there is no problem in mobo? If there is no FLR in mobo, is it critical? I do not need gpu passthrough, only usb.
And I want to buy ASRock Z97 Extreme6.

vuldin, any success with yours project?


----------



## vuldin

Quote:


> Originally Posted by *opt1k*
> 
> So, there is no problem in mobo? If there is no FLR in mobo, is it critical? I do not need gpu passthrough, only usb.
> And I want to buy ASRock Z97 Extreme6.
> 
> vuldin, any success with yours project?


I've made some progress but still don't have vga passthrough. I made a post to another related thread on the forum with all the updated details in the hopes that it will get views from additional people.

http://www.overclock.net/t/1338063/vt-d-compatible-motherboards/60#post_22394082

As for the ASRock Z97 Extreme6, I think it is an awesome motherboard. I can't blame it for my current lack of VGA passthrough, and I feel like I will get it working eventually. I think that if there is a hardware issue at this point it lies in the 290x, and it wouldn't be a hardware issue so much as an issue with xen knowing how to communicate with the GPU BIOS of this card. I need to do more investigation into this though.


----------



## powerhouse

Quote:


> Originally Posted by *vuldin*
> 
> I've made some progress but still don't have vga passthrough. I made a post to another related thread on the forum with all the updated details in the hopes that it will get views from additional people.
> 
> http://www.overclock.net/t/1338063/vt-d-compatible-motherboards/60#post_22394082
> 
> As for the ASRock Z97 Extreme6, I think it is an awesome motherboard. I can't blame it for my current lack of VGA passthrough, and I feel like I will get it working eventually. I think that if there is a hardware issue at this point it lies in the 290x, and it wouldn't be a hardware issue so much as an issue with xen knowing how to communicate with the GPU BIOS of this card. I need to do more investigation into this though.


I haven't read all the information you have given, sorry. But a common mistake is to not activate IOMMU (or VT-d) in the motherboard BIOS.

The next step is to turn primary passthrough off in the guest config file. qemu-xen-traditional is what you need to specify too. Before starting the VM, check to see that the VGA devices are assignable.


----------



## Nicolae

Hi,

Is this working on a laptop ? if yes what laptop model / brand / whatever I need to have this ?
I was asking same question you solved here but for a laptop: https://forums.freebsd.org/viewtopic.php?f=3&t=47003&p=264428&hilit=GPU#p264428

I mean what laptop with 2 GPU would allow this "native speed casual gaming" then back to work.


----------



## Rykoshet

On the first page it says: "Now if you had a second card, you could make a second fully functional system!" Does that refer to the video card?

If so - hypothetical question - can I configure a VM with its own graphics card, but share the CPU and RAM (or use its own reserved resources)?

My wife and I enjoy playing D&D type games together, but our laptops cannot keep up with the more demanding games, such as Divinity: Original Sin. I am building a desktop anyway, but if I could just spend a little more (second GPU) and get a segregated gaming machine for the wife, that would be a great solution, no matter how much trouble it might be to set up.

Is it feasible? I am not just thinking about saving money, but also space, and having everything on one MOBO means I can easily convert it into just my own CF gaming setup if need be...

If anyone can confirm this is possible/the idea, I'll start checking this path out. Thanks.


----------



## zirblazer

For a multiseat system, you can count on CPU, RAM and Storage to be easily shareable between VMs. The big problem was the GPU, which with VGA Passthrough you can solve by having one GPU per VM. You also need a Monitor to go with it. For input, you need an extra Keyboard and Mouse, which can also be given to a specific VM via PCI or USB Passthrough (A controller or the individual device, the first was recommended due to lower overhead). Oh, you also need to drop a USB Sound Card so you can pass it to the VM (Integrated Motherboard Sound Card to one VM, and the USB Sound Card to the other one).
Currently, there is also XenGT project, which aims to virtualize the Haswell's GPU to be able to share it among many VMs. But I doubt it has the horsepower to power up two games simultaneously, so even with it, you may need one or two discrete Video Cards.

Also, I correct some info I provided before: FLR is not supported by Quadros, either, I found some guy that posted lspci -vvv output. The MultiOS feature that allows it to properly reset after a VM reboot seems to be entirely Driver related.


----------



## powerhouse

Quote:


> Originally Posted by *Rykoshet*
> 
> On the first page it says: "Now if you had a second card, you could make a second fully functional system!" Does that refer to the video card?


Yes
Quote:


> Originally Posted by *Rykoshet*
> If so - hypothetical question - can I configure a VM with its own graphics card, but share the CPU and RAM (or use its own reserved resources)?


Yes
Quote:


> Originally Posted by *Rykoshet*
> My wife and I enjoy playing D&D type games together, but our laptops cannot keep up with the more demanding games, such as Divinity: Original Sin. I am building a desktop anyway, but if I could just spend a little more (second GPU) and get a segregated gaming machine for the wife, that would be a great solution, no matter how much trouble it might be to set up.


You have to pay close attention to the hardware requirements! Your new PC must support VT-d for Intel CPUs, or AMD-Vi for AMD CPUs. Also, not all graphics cards will work. It is in fact a little tricky. See this how-to and also my link below for more on it.
Quote:


> Originally Posted by *Rykoshet*
> Is it feasible? I am not just thinking about saving money, but also space, and having everything on one MOBO means I can easily convert it into just my own CF gaming setup if need be...


Yes it is. Just be prepared to spend some time to get it working. And make sure you get the right hardware.
Quote:


> Originally Posted by *Rykoshet*
> If anyone can confirm this is possible/the idea, I'll start checking this path out. Thanks.


You need two graphics cards, two screens, two sets of mouse/keyboard, and probably a second sound card. Also, if you plan to play under Windows, you will need a CPU with integrated GPU to run Linux/the Xen hypervisor (at least it makes it easier).

EDIT: Oops, didn't see that zirblazer already answered.


----------



## Rykoshet

Quote:


> Originally Posted by *zirblazer*
> 
> For a multiseat system, you can count on CPU, RAM and Storage to be easily shareable between VMs. The big problem was the GPU, which with VGA Passthrough you can solve by having one GPU per VM. You also need a Monitor to go with it. For input, you need an extra Keyboard and Mouse, which can also be given to a specific VM via PCI or USB Passthrough (A controller or the individual device, the first was recommended due to lower overhead). Oh, you also need to drop a USB Sound Card so you can pass it to the VM (Integrated Motherboard Sound Card to one VM, and the USB Sound Card to the other one).
> Currently, there is also XenGT project, which aims to virtualize the Haswell's GPU to be able to share it among many VMs. But I doubt it has the horsepower to power up two games simultaneously, so even with it, you may need one or two discrete Video Cards.
> 
> Also, I correct some info I provided before: FLR is not supported by Quadros, either, I found some guy that posted lspci -vvv output. The MultiOS feature that allows it to properly reset after a VM reboot seems to be entirely Driver related.


Quote:


> Originally Posted by *powerhouse*
> 
> Yes
> Yes
> You have to pay close attention to the hardware requirements! Your new PC must support VT-d for Intel CPUs, or AMD-Vi for AMD CPUs. Also, not all graphics cards will work. It is in fact a little tricky. See this how-to and also my link below for more on it.
> Yes it is. Just be prepared to spend some time to get it working. And make sure you get the right hardware.
> You need two graphics cards, two screens, two sets of mouse/keyboard, and probably a second sound card. Also, if you plan to play under Windows, you will need a CPU with integrated GPU to run Linux/the Xen hypervisor (at least it makes it easier).
> 
> EDIT: Oops, didn't see that zirblazer already answered.


Thank you both! I already have the monitor/KB/mouse, so this is something I am really excited to try.

I was planning to get a separate 60-80 GB SSD for the wife's VM partition, I assume direct I/O applies to SATA ports as well, correct, so this is a good idea?

Then I am not sure what storage solution to get. If I was building a rig just for myself, I would get a 250 GB SSD, but given that this device will sometimes be used for just me (I hope to CF the cards) and sometimes the system will be running a VM with direct I/O, any suggestions on how many drives to buy, what size, HDD vs SSD, etc?

Thanks.


----------



## subassy

Quote:


> Originally Posted by *Rykoshet*
> 
> On the first page it says: "Now if you had a second card, you could make a second fully functional system!" Does that refer to the video card?
> 
> If so - hypothetical question - can I configure a VM with its own graphics card, but share the CPU and RAM (or use its own reserved resources)?
> 
> My wife and I enjoy playing D&D type games together, but our laptops cannot keep up with the more demanding games, such as Divinity: Original Sin. I am building a desktop anyway, but if I could just spend a little more (second GPU) and get a segregated gaming machine for the wife, that would be a great solution, no matter how much trouble it might be to set up.
> 
> Is it feasible? I am not just thinking about saving money, but also space, and having everything on one MOBO means I can easily convert it into just my own CF gaming setup if need be...
> 
> If anyone can confirm this is possible/the idea, I'll start checking this path out. Thanks.


Seems like everybody already answered this but I have some questions about this since I found Rykoshet's post a little confusing. He mentions him and his wifi having laptops that can't get certain games to run then mentions building a new PC with two graphics cards. Then goes on to mention a "segregated gaming machine".

This could mean he wants to build a separate physical PC. Except he mentions saving space. So I can only conclude he actually wants to having two copies of one game running in two different VMs so he and his wife can play it at the same time. Which would require: a total of 3 GPUs (one of the Xen config stuff, 1 each for the VMs/gaming), two monitors, two keyboards/two mice and two sound cards.

Is this really possible? Wouldn't you need some kind of super-powerful CPU for this? Might be better off with some kind of server magic. As in multiple Xeon machines with a ton of RAM. Then booting off iSCSI with thin clients. Something like that. That wouldn't really be saving space or money though.


----------



## Rykoshet

Quote:


> Originally Posted by *subassy*
> 
> Seems like everybody already answered this but I have some questions about this since I found Rykoshet's post a little confusing. He mentions him and his wifi having laptops that can't get certain games to run then mentions building a new PC with two graphics cards. Then goes on to mention a "segregated gaming machine".
> 
> This could mean he wants to build a separate physical PC. Except he mentions saving space. So I can only conclude he actually wants to having two copies of one game running in two different VMs so he and his wife can play it at the same time. Which would require: a total of 3 GPUs (one of the Xen config stuff, 1 each for the VMs/gaming), two monitors, two keyboards/two mice and two sound cards.
> 
> Is this really possible? Wouldn't you need some kind of super-powerful CPU for this? Might be better off with some kind of server magic. As in multiple Xeon machines with a ton of RAM. Then booting off iSCSI with thin clients. Something like that. That wouldn't really be saving space or money though.


I thought only two GPUs would be required, since one person would be using the actual computer, while the other would be using the VM. I don't plan on making two VMs.


----------



## josipva

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

EDIT:

Nevermind. Got my answer on arch forums.


----------



## subassy

Quote:


> Originally Posted by *Rykoshet*
> 
> I thought only two GPUs would be required, since one person would be using the actual computer, while the other would be using the VM. I don't plan on making two VMs.


Guess I missed that part. The game of course would have to be a linux native (obviously).

I thought there would still be an issue with the CPU and OS trying to figure out where to send which keyboard/mouse signal at the the same time. I thought more trickery would be required to make this work. But I haven't tried to set up the original Xen/gaming VM thing so this may or may not be the case for all I know.


----------



## quipers

Does anyone have any experience with getting vga passthrough to work on a laptop that has an Intel integrated gpu along with an Nvidia Optimus gpu?

Is it doable yet?


----------



## appleboy

I have problems creating the virtual machine, stuck at
$ sudo xl create win8.sxp
"Libxl : error: libxl.c:91:libxl_ctx_alloc: Is xenstore deamon running?
Failed to stat /var/run/xenstore.pid: No such file or directory
Cannot init xl context"

I have manually enable and start xenstore


----------



## smartwart

Quote:


> Originally Posted by *Natriumatom*
> 
> Took one last shot at it and got it working with XenServer 6.2SP1
> 
> 
> 
> 
> 
> 
> 
> 
> ....


I'm inspired by this video post:
https://www.youtube.com/watch?v=8vCH-7AeNvg

How is it possible to connect to VM from host machine?
I want to have a VM for gaming and play around and another one which is running 24/7 with mission critical stuff.
When I was installing xemserver 6.2, I only can connect remote via xencenter from another machine.

How can I achieve a xenserver setup like this?


----------



## Donach

Hello!
I have a question regarding the multi OS support.

What do I would like to do?

I would like to have dual OS of Win 8 (both machine and virtual), while on real machine I'll be playing (or the other way around) and I'll run a another game in virtual machine using Steam (or any other) to stream into another PC. OR I could connect another set of mouse + keyboard to control the virtual OS with another monitor connected too.
Is this even possible?

What would it take to make it happen?
Having 2 discrete GPU's and strong 8-core CPU?

I was thinking of having a GTX 960 or some price AMD equivalent and 8-core AMD CPU like 8350 (since I already have AM3+ motherboard) + 6850 HD GPU I have right now.
I do also have 2 (or even 3) monitors available, so having multiple video output is no problem. I also count in investing in more RAM, 16 GB atleast I suppose.

If it would be possible to run 2 VMs on a Linux, it wouldn't still be such a problem. (If the fx-8350 could handle it).

Any advice would be very helpful!


----------



## F3ERS 2 ASH3S

Quote:


> Originally Posted by *Donach*
> 
> Hello!
> I have a question regarding the multi OS support.
> 
> What do I would like to do?
> 
> I would like to have dual OS of Win 8 (both machine and virtual), while on real machine I'll be playing (or the other way around) and I'll run a another game in virtual machine using Steam (or any other) to stream into another PC. OR I could connect another set of mouse + keyboard to control the virtual OS with another monitor connected too.
> Is this even possible?
> 
> What would it take to make it happen?
> Having 2 discrete GPU's and strong 8-core CPU?
> 
> I was thinking of having a GTX 960 or some price AMD equivalent and 8-core AMD CPU like 8350 (since I already have AM3+ motherboard) + 6850 HD GPU I have right now.
> I do also have 2 (or even 3) monitors available, so having multiple video output is no problem. I also count in investing in more RAM, 16 GB atleast I suppose.
> 
> If it would be possible to run 2 VMs on a Linux, it wouldn't still be such a problem. (If the fx-8350 could handle it).
> 
> Any advice would be very helpful!


sounds like you would want to set up the multiple VMs then have a dummy computer or 2 to remote to the VMs and play off of the VMs via a remote application.. windows rdp or equivalent etc etc... the issue that you would be doing is running into the actual input methods overriding the others.

most of this is just for a low level virtual environment that does not have a high overhead. in addition to that.. it is to allow passthrough commands to access the hardware directly. for example: if you just use virtual pc and try to run a dx11 game, it will not work due to it reading the generic graphics drivers of the virtual machine.

now with the hardware pass through it would allow the virtual machines to use the actual graphics cards as intended so on and so forth,

AS far as I can recall.. now unless I have missed something.. I am not sure that you can actually set the ports to connect to a particular VM.. that would be the only way.. mainly the issue is that you need direct access to the input media without the other virtual machines detecting it.


----------



## burning chrome

I've got a couple questions regarding esxi virtualisation.
I'd like to run vmware esxi on my main rig and run a 'desktop' linux and a 'gaming' win 7 vm on it similar to this http://www.pugetsystems.com/labs/articles/Multi-headed-VMWare-Gaming-Setup-564/

is it possible to access the vsphere web client from inside the linux vm to start the win 7vm?
if i power down the rig do i need a laptop to turn on the 'desktop' vm remotely or is it possible to tell esxi to boot into a vm on powerup?
do i need 2 pairs of mice/keyboards?


----------



## nbhusain

Quote:


> Originally Posted by *burning chrome*
> 
> I've got a couple questions regarding esxi virtualisation.
> I'd like to run vmware esxi on my main rig and run a 'desktop' linux and a 'gaming' win 7 vm on it similar to this http://www.pugetsystems.com/labs/articles/Multi-headed-VMWare-Gaming-Setup-564/
> 
> is it possible to access the vsphere web client from inside the linux vm to start the win 7vm?
> if i power down the rig do i need a laptop to turn on the 'desktop' vm remotely or is it possible to tell esxi to boot into a vm on powerup?
> do i need 2 pairs of mice/keyboards?


You can run a linux + kvm desktop on bare metal, and create a windows VM with a dedicated GPU, instead of having an hypervisor consuming resources under a virtualized desktop: take a look: https://bbs.archlinux.org/viewtopic.php?id=162768


----------



## kitg90

Hey guys reading the first post about the benefits:

Uninterrupted sessions of other VMs
Ability to run the latest games on a VM
Being able to use all that extra Memory and those Cores

Could anyone elaborate more about this? I'm new and would like to understand what are the real benefits to having a Gaming VM.

Thanks!


----------



## burning chrome

Quote:


> Originally Posted by *nbhusain*
> 
> You can run a linux + kvm desktop on bare metal, and create a windows VM with a dedicated GPU, instead of having an hypervisor consuming resources under a virtualized desktop: take a look: https://bbs.archlinux.org/viewtopic.php?id=162768


i'll check it out but i have zero experience with arch. i have however looked a bit into using xen on mint. vmware esxi with pcie passthrough just seems a lot easier to set up, at least in theory.

i have also thought about the keyboard/mouse problem and figured it should be possible to passthrough a usb port to each vm and connect a usb hub to each one.


----------



## nbhusain

Quote:


> Originally Posted by *burning chrome*
> 
> i'll check it out but i have zero experience with arch. i have however looked a bit into using xen on mint. vmware esxi with pcie passthrough just seems a lot easier to set up, at least in theory.
> 
> i have also thought about the keyboard/mouse problem and figured it should be possible to passthrough a usb port to each vm and connect a usb hub to each one.


You can do this with any distro, my guide is pretty much generic, there are some guide for ubuntu too: http://ubuntuforums.org/showthread.php?t=2266916 http://www.pugetsystems.com/labs/articles/Multiheaded-NVIDIA-Gaming-using-Ubuntu-14-04-KVM-585/


----------



## Bradford1040

Okay trying to do this KVM pass through on my system and running into some of the dumbest noob issues, I just can't seem to even get the OS installed! Talk about feeling neutered! I normally can figure out pretty much everything but can't seem to get two cdroms loaded with CMDline "Qemu" I need two ISO's so I can install drivers that are not in windows!

qemu-system-x86_64 -enable-kvm -m 4096 -cdrom win7_64.iso -boot d win7.qcow2

That is the loading line I was going to use but also tried to add another cdrom, anyone know the right way to word it, I have tried everything but the one that works I guess lol. Please help

I have Arch with BlackArch and ArchAssault repos, AUR as well, loaded pretty much everything hardware wise, I have VT-d and x set and loaded on boot, I have Virtio loaded on boot! I have a few GUI frontends but CMDline just works faster for me. Everything is updated daily BTW so running newest kernel


----------



## Bradford1040

Well I got everything to work, I had a corrupt windows install disk/usb lol it seems. I am now down to getting the KVM Passthrough on the GPU working, any help would always be welcomed! I started this with no understanding on having to have two GPU's but not to fret I have another CPU to put into this system so I will be doing that after I get done compiling the kernel 3.19.3.3. I am updated to 4.01 kernel but rolling back to 3.19.3.3 is no big deal for me, worked great anyway! Again any help from anyone is welcome


----------



## fergalom

Perhaps someone can help.

F20 Dom0 with kernel per Dizzys guide

Trying to launch a F18 DomU with a Nvidia Quadro card passed through but it won't pass it through and getting the following message which passes through only part of the card (i.e. 00:03:0.1 but not 00:03:0.0) - any ideas what the issue is?

Code:



Code:


[[email protected] ~]$ sudo xl pci-assignable-add 0000:03:00.0
[sudo] password for xenhost: 
libxl: warning: libxl_pci.c:654:libxl__device_pci_assignable_add: 0000:03:00.0 not bound to a driver, will not be rebound.
libxl: error: libxl_pci.c:350:sysfs_write_bdf: write to /sys/bus/pci/drivers/pciback/bind returned -1: No such device
libxl: error: libxl_pci.c:535:pciback_dev_assign: Couldn't bind device to pciback!: No such device
libxl: error: libxl_pci.c:659:libxl__device_pci_assignable_add: Couldn't bind device to pciback!
[[email protected] ~]$ sudo xl pci-assignable-add 0000:03:00.1
libxl: warning: libxl_pci.c:635:libxl__device_pci_assignable_add: 0000:03:00.1 already assigned to pciback

[[email protected] ~]$ sudo ./F18-RASP.sh
./F18-RASP.sh: line 6:  2323 Killed                  xl pci-assignable-add 00:03:0.0
Parsing config from /etc/xen/F18-RASP.sxp
xc: info: VIRTUAL MEMORY ARRANGEMENT:
  Loader:        0000000000100000->00000000001a2304
  Modules:       0000000000000000->0000000000000000
  TOTAL:         0000000000000000->0000000111800000
  ENTRY ADDRESS: 0000000000100000
xc: info: PHYSICAL MEMORY ALLOCATION:
  4KB PAGES: 0x0000000000000200
  2MB PAGES: 0x000000000000048b
  1GB PAGES: 0x0000000000000002
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
libxl: error: libxl_pci.c:1055:libxl__device_pci_add: PCI device 0:3:0.0 is not assignable
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:03:00.1
Daemon running with PID 2441
[[email protected] ~]$ sudo xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1022     8     r-----      67.0
F18-RASP                                     3  4379     4     r-----      17.9
[[email protected] ~]$ sudo xl destroy 3
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:00:14.0
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:03:00.0
libxl: error: libxl_pci.c:1265:do_pci_remove: xc_deassign_device failed
libxl: error: libxl_pci.c:990:libxl__device_pci_reset: The kernel doesn't support reset from sysfs for PCI device 0000:03:00.1


----------



## zipeldiablo

Hello guys, i'm currently buying my final parts and i am looking at the best option for the os and the virtualization software.
So question is simple, considering xen and kvm (which from my research are the best options) which one will consume the least ressources and give me the maximum power possible?
For the host os i'm thinking archlinux? it seems that it is the one of the least consuming options (or centos maybe, not sure yet).

Anyone got benchmarks by any chance?


----------



## Sleepyza57

I already build a new third rig. I've enabled the IOMMU controller in the UEFI menu. I'm running Ubuntu MATE 16.04 64-bit Beta. I don't know if which KVM, QEMU, or Xen that is the best and easiest to setup the virtualization for me? See below for the computer spec. Do I need another GPU and a second monitor, or I could just use the single monitor? (can switch the video source). Can I use Ubuntu MATE and use your guide? I'm not sure if I know what am I doing. So please tell me what to do. Thanks in advance.

AMD FX-8350
Gigabyte GA-990FXA-UD5 R5
G.Skill Sniper 16GB DDR3 1866
Samsung 840 Pro 128GB SSD (I will add another high capacity SSD or HDD soon)
Gigabyte Windforce Radeon 7950 3GB Ghz Edition
PS/2 Keyboard
Etekcity gaming optical mouse USB


----------



## contay

Hello. I am complete newcomer when it comes to Linux and virtual computing. I don't really seek help or a guide, but I'd like to know if following scenario would be possible:

I'd like to utilize my current rig (4930K and 980ti) on 100% capacity when I am alone. When I have GF at my place, I'd like to share system virtually to two gaming rigs (second 980ti or other gpu). This would help me give up on second rig.

In this way I'd have 4930K, 2x 980ti and multible drives or raid partitions: lets say D1, D2 and D3.

Lets say my primary data and OS reside and D1, secondary rig is D2 and D3 would be for host.

So, normally I'd use D1 as boot drive and don't give a Trump of virtual systems, jusy happily using 12 threads and 980ti SLI.

*Wild GF appears*
*GF uses gaming*

And here begins the magic: could I just reboot with D3 (host) and immideatly have two booting virtual windowses with shared resources (dedicated cores and RAM) and desicated GPU?


----------



## nbhusain

Yes, its totally possible, but since you mentioned you're new to linux, you have to be prepared to spend a lot of time doing some research, i personally recommend KVM over other solutions like XEN because in my opinion its easier to set up, have a look at vfio.blogspot.com, i believe it is fedora centric but theres tons of distro specific guides out there.


----------



## contay

Quote:


> Originally Posted by *nbhusain*
> 
> Yes, its totally possible, but since you mentioned you're new to linux, you have to be prepared to spend a lot of time doing some research, i personally recommend KVM over other solutions like XEN because in my opinion its easier to set up, have a look at vfio.blogspot.com, i believe it is fedora centric but theres tons of distro specific guides out there.


Thank you. I look into that link. I did some research and found out that qemu-kvm would be best bet.

Also, friend of mine appears to have some experience in kind of this. He recommended Debian sid, as he has most experience with it and has some success running something like this.

My current plan is to rig second 980ti and 750ti or similiar like that. 750ti would go with host and 980ti's will go for virtual windowses. Either one each or SLI for one. I might still keep one partition for bootable windows.


----------



## nbhusain

i've seen a lot of debian/ubuntu guides out there so you should be ok, as for the storage setup there tons of interesting configurations to try, personally i just passthrough my raid controler to the windows VM, so i can dual boot and virtualize the same windows install, if you have some ssd's lying around you can also use bcache backing for super fast read/write ops


----------



## contay

Quote:


> Originally Posted by *nbhusain*
> 
> i've seen a lot of debian/ubuntu guides out there so you should be ok, as for the storage setup there tons of interesting configurations to try, personally i just passthrough my raid controler to the windows VM, so i can dual boot and virtualize the same windows install, if you have some ssd's lying around you can also use bcache backing for super fast read/write ops


Thanks for the tip. That sounds interesting option to try, so I might look up for preowned raidcontroller on our local forums. Currentyly I have some other things to do, but I have at least introduced myself to debian (installed and played around a little). If I manage to pull this the way I like, I'll keep you posted.

I thought easier way too: I could just keep my current raid and hook 3 drives to ASMedia controlled sata-ports. They are always ACHI-mode, so it would be easy to make btrfs in debian with them. OR I could try even simpler way and use unraid in the way LTT used here. It is way less complicated to use preconfigured setup and as I have six (yes, six) SSDs to use, it could be done. Altough free version of can use only three, but I guess 3x256 in raid0 is enough for two rigs as there are only few games I and GF play at the time. Also, with 100mbps connection downloading a steam game takes quite little time, which can be used for other beneficial purposes...


----------



## nbhusain

If you're getting a raid controller make sure it has an efi rom, because you will most likely en up using OVMF which is an efi firmware (im using my integrated amd raid controller, i had to dump the efi driver from memory, then convert it to a rom)


----------



## contay

Quote:


> Originally Posted by *nbhusain*
> 
> If you're getting a raid controller make sure it has an efi rom, because you will most likely en up using OVMF which is an efi firmware (im using my integrated amd raid controller, i had to dump the efi driver from memory, then convert it to a rom)


Half that went over my head. Anyway. Current plan is following, if it would just work. Just for simplicity, as I have about zero experience in linux.


I'll keep my raid0 configuration, where my windows currently resides and and continue using it as I have until today.
Plug 4 SSDs to my mobo (Rampage IV Black Edition). It has second controller for four ports (ASMedia) and they are always ACHI so this should work with unraid.
Follow this video from LTT which I mentioned before to create two seperate windowses differing from "main disks".
This would give me little easier configuration while sacrifising disk space. But, since drives aren't problem, this should be easiest approach. When I have more time and less stress, I would like to configure better system with debian and virtual machines. My main consern is this sata-configuration, but since SSDs for unraid are ACHI, this should work, right?

Also: for unraid, you should use different usb-peripherals. I quess this doesn't matter, if I use usb-hub integrated in my display?


----------



## nbhusain

Yeah you should be ok


----------



## contay

I've been tinkering with unraid for a while now. Still having problems with VM. Should I try win10 in virtual machine instead of 8.1, could win10 just work better?


----------



## nbhusain

I dont think so, windows 10 used to work like crap for me under previous qemu versions, now its working fine but i recommend 8.1, what kind of problems are you having?


----------



## contay

Quote:


> Originally Posted by *nbhusain*
> 
> I dont think so, windows 10 used to work like crap for me under previous qemu versions, now its working fine but i recommend 8.1, what kind of problems are you having?


Host (unraid) keeps crashing after launcing VMs. Usb devices dedication pisses off, but it is very hard to work around. There is kind of solution, however: pci usb-card which I will passtrough to vm. It enables hot plug usb for vm.

Crashing is different then, I have to keep investigating it and experimenting with different setups with VM settings.


----------



## spinFX

Nice guide, however I'd consider using KVM over XEN for a home setup.
KVM is built into certain linux kernels (ie. Ubuntu), and due to native support, will be upgraded as you upgrade your linux distro (XEN may become incompatible upon updating kernel as there is no native support for XEN in any linux distro)

For this setup, KVM has enough features to be equal to XEN (I have the same setup as in this tutorial, using KVM - 3d Mark scores are only a few hundred points off the score the system got when not being run as a VM)

Also KVM has red hat backing it, and is becoming standardised across all linux distros, so it will easy to migrate your KVM VM from one linux distro to another.


----------



## Candleflame

hi guys, I hope you can help me out:

I plan to use Linux as my main OS but I want windows a secondary OS to do gaming and some other fairly cpu intensive tasks (i.e. video editing).

Is it best to to run Xen as a type 1 hypervisor and then run BOTH windows and i.e. ubuntu both as VMs or should you just stick to Fedora as your main OS and then run windows from within?

I'm looking for whichever way gives me the most performance.


----------



## zipeldiablo

Quote:


> Originally Posted by *Candleflame*
> 
> hi guys, I hope you can help me out:
> 
> I plan to use Linux as my main OS but I want windows a secondary OS to do gaming and some other fairly cpu intensive tasks (i.e. video editing).
> 
> Is it best to to run Xen as a type 1 hypervisor and then run BOTH windows and i.e. ubuntu both as VMs or should you just stick to Fedora as your main OS and then run windows from within?
> 
> I'm looking for whichever way gives me the most performance.


I don't know if things changed but when i was doing this vga passthrough was next to impossible on xen, it was running just fine on fedora though, no need to switch your os


----------



## Candleflame

Quote:


> Originally Posted by *zipeldiablo*
> 
> I don't know if things changed but when i was doing this vga passthrough was next to impossible on xen, it was running just fine on fedora though, no need to switch your os


Well my plan was to use Fedora + KVM to then launch windows. I read that apparently KVM behaves like a Type 1 hypervisor anyways (so you get the same performance gain) so there is no need to use a bloaty Type 1 ghost-OS which is only there to launch your actual linux OS.

The problem is that most people just use this for gaming under windows where cpu performance really doesnt matter that much so you see conflicting information on which solutions provides the most cpu power.


----------



## zipeldiablo

Quote:


> Originally Posted by *Candleflame*
> 
> Well my plan was to use Fedora + KVM to then launch windows. I read that apparently KVM behaves like a Type 1 hypervisor anyways (so you get the same performance gain) so there is no need to use a bloaty Type 1 ghost-OS which is only there to launch your actual linux OS.
> 
> The problem is that most people just use this for gaming under windows where cpu performance really doesnt matter that much so you see conflicting information on which solutions provides the most cpu power.


Indeed kvm act as an hypervisor, the difference with xen is that the "os" part of xen (xen runs over linux) is very small, but there is not much loss of performance.
Taking gaming as example i only had 3% difference, which were coming for the cpu use of my linux host.
Cpu does matter, well, it depends on your cpu for sure but some games are really heavy on the cpu.
Also there are lot of things you can configure to have more cpu performance on windows.


----------



## Candleflame

Quote:


> Originally Posted by *zipeldiablo*
> 
> Indeed kvm act as an hypervisor, the difference with xen is that the "os" part of xen (xen runs over linux) is very small, but there is not much loss of performance.
> Taking gaming as example i only had 3% difference, which were coming for the cpu use of my linux host.
> Cpu does matter, well, it depends on your cpu for sure but some games are really heavy on the cpu.
> Also there are lot of things you can configure to have more cpu performance on windows.


ok, but how does that compare to KVM? Bearing in mind that with KVM you only need to launch 1) Linux and 2) Windows and not 1) LinuxXen 2) Linux 3) Windows


----------



## zipeldiablo

Quote:


> Originally Posted by *Candleflame*
> 
> ok, but how does that compare to KVM? Bearing in mind that with KVM you only need to launch 1) Linux and 2) Windows and not 1) LinuxXen 2) Linux 3) Windows


My message wasn't clear.
Xen is already running on top of linux, the difference is you cannot access it.
I was talking about kvm performances


----------



## Candleflame

so are you saying that if you install fedora and then install xen then you continue to use fedora as your main OS and launch windows from within?


----------



## zipeldiablo

Quote:


> Originally Posted by *Candleflame*
> 
> so are you saying that if you install fedora and then install xen then you continue to use fedora as your main OS and launch windows from within?


No point using xen if you are on fedora already.
Also pretty sure you will not be able to passthrough your gpu with xen


----------



## Liranan

It is, indeed, impossible to use Xen for passthrough and it's irrelevant anyway as KVM is the better solution now. If you are using Fedora passthrough is quite easy compared with Ubuntu based distro's as it only requires a few commands.


----------



## powerhouse

Quote:


> Originally Posted by *Liranan*
> 
> It is, indeed, impossible to use Xen for passthrough and it's irrelevant anyway as KVM is the better solution now. If you are using Fedora passthrough is quite easy compared with Ubuntu based distro's as it only requires a few commands.


Dismissing Xen is a little unfair.

Xen is an excellent hypervisor that has a lot going for it. I have used it for VGA passthrough for over 3 years and it has been stable with excellent performance, and the setup was a piece of cake compared to KVM.

KVM, on the other side, has caught up and may be the better choice in many cases, particularly on desktops with Nvidia graphics. I myself switched to KVM for a very specific reason - support of Nvidia proprietary driver in Linux. To my knowledge this can now also be achieved in Xen.

With KVM, however, it can be a real pain in the neck to get good (or even decent) performance. The documentation pretty much sucks, with a few exceptions. I have had very bad experiences with virt-manager, a popular GUI to set up KVM virtual machines. Some people report success, however. I have followed many of the published virt-manager based guides and none of them gave me the performance I'm used to. And trust me, I'm not a newcomer to this.

Fedora versus Ubuntu: Perhaps the virt-manager based passthrough tutorials really only work with Fedora/Red Hat, which is a pretty bad sign as virt-manager is developed by Red Hat. I'd expect a software package to work on all major platforms, but all this is pure speculation on my side.
The fact is that Ubuntu and its derivatives like Linux Mint has many users, and quite a few of them managed to get VGA passthrough working using qemu and a little script. I've written how-tos for both Xen VGA passthrough and KVM VGA passthrough for Linux Mint, both of which work for Ubuntu too.

If I were to recommend a distribution for use with VGA passthrough, I'd name Arch Linux simply because of their built-in features and their excellent documentation, and because of their forum.

When I use the word "performance", I can share benchmarks to quantify it, for example: https://forums.linuxmint.com/viewtopic.php?f=225&t=153482.


----------



## JackCY

Expected this to be unRAID, found Xen and KVM, hmm. Why not unRAID? From just technical POV, pricing aside.


----------



## powerhouse

Quote:


> Originally Posted by *JackCY*
> 
> Expected this to be unRAID, found Xen and KVM, hmm. Why not unRAID? From just technical POV, pricing aside.


This is a very old thread/tutorial, back from a time when unRAID wasn't around or didn't support this stuff.

Although Xen is still a viable option, it has been put in second place by KVM. If I'm not mistaken, unRAID uses KVM for VGA passthrough.

unRAID is a commercial solution. Xen and KVM come free. There are more up-to-date tutorials for different Linux distributions, in particular when using KVM.

From a technical POV, it boils down to the hardware you have. If your hardware (CPU and motherboard) doesn't support IOMMU (VT-d in Intel terms), none of the above will work.


----------



## zipeldiablo

Quote:


> Originally Posted by *JackCY*
> 
> Expected this to be unRAID, found Xen and KVM, hmm. Why not unRAID? From just technical POV, pricing aside.


Hum, if I remember correctly I did research on unraid back in the days and it was a real pain.
Not enough customisation and the lots.

Quote:


> Originally Posted by *powerhouse*
> 
> Dismissing Xen is a little unfair.
> 
> Xen is an excellent hypervisor that has a lot going for it. I have used it for VGA passthrough for over 3 years and it has been stable with excellent performance, and the setup was a piece of cake compared to KVM.
> 
> KVM, on the other side, has caught up and may be the better choice in many cases, particularly on desktops with Nvidia graphics. I myself switched to KVM for a very specific reason - support of Nvidia proprietary driver in Linux. To my knowledge this can now also be achieved in Xen.
> 
> With KVM, however, it can be a real pain in the neck to get good (or even decent) performance. The documentation pretty much sucks, with a few exceptions. I have had very bad experiences with virt-manager, a popular GUI to set up KVM virtual machines. Some people report success, however. I have followed many of the published virt-manager based guides and none of them gave me the performance I'm used to. And trust me, I'm not a newcomer to this.
> 
> Fedora versus Ubuntu: Perhaps the virt-manager based passthrough tutorials really only work with Fedora/Red Hat, which is a pretty bad sign as virt-manager is developed by Red Hat. I'd expect a software package to work on all major platforms, but all this is pure speculation on my side.
> The fact is that Ubuntu and its derivatives like Linux Mint has many users, and quite a few of them managed to get VGA passthrough working using qemu and a little script. I've written how-tos for both Xen VGA passthrough and KVM VGA passthrough for Linux Mint, both of which work for Ubuntu too.
> 
> If I were to recommend a distribution for use with VGA passthrough, I'd name Arch Linux simply because of their built-in features and their excellent documentation, and because of their forum.
> 
> When I use the word "performance", I can share benchmarks to quantify it, for example: https://forums.linuxmint.com/viewtopic.php?f=225&t=153482.


The real pain for me was to build my custom kernel.
Once i've done that i was a pain of cake (mainly because i had fixed every problem before).

Back then i couldn't find any documentation on how to setup vga passthrough with xen and the threads i bump into said it wasn't possible, so i dropped the idea (wanted to do passthrough on qubes os)
Virt-manager sucks big time and should ONLY be used to create your config file on the first run.
When you say qemu you mean qemu-kvm right?
What is awesome with qemu is qemu hooks ! Plugging and deplugging your usb peripherals, damn that was good.

You say documentation sucks but there is a 170+pages thread on archlinux forum, about 20 blogpost from a guy working at redhat on virt and a mailing list where people actually answer your questions.

Arch is great for building your custom kernel, was really easy compared to do it on fedora... (also it took me 3 different oses to do what i wanted, since i started on centos but the damn thing being on lts didn't have a kernel recent enough).
There is a great thread on arch forums bug those modos are nazys mate, they closed it because a lot of folks who didn't use arch went on the forum ask for help, such a pity... since then the community apart from the mailing list kinda disolved.

Speaking of benchmarks i did some with 3dmark and i was only 3% behind in terms of performance between native win10 and vm win10.
It was due in fact to the cpu consumption of arch linux.

Anyway if you guys are looking for vm files or stuff like that i can share my former config : https://github.com/tirrorex/Virtual-machine


----------



## powerhouse

Quote:


> Originally Posted by *zipeldiablo*
> 
> The real pain for me was to build my custom kernel.
> Once i've done that i was a pain of cake (mainly because i had fixed every problem before).
> 
> Back then i couldn't find any documentation on how to setup vga passthrough with xen and the threads i bump into said it wasn't possible, so i dropped the idea (wanted to do passthrough on qubes os)
> Virt-manager sucks big time and should ONLY be used to create your config file on the first run.
> When you say qemu you mean qemu-kvm right?
> What is awesome with qemu is qemu hooks ! Plugging and deplugging your usb peripherals, damn that was good.
> 
> You say documentation sucks but there is a 170+pages thread on archlinux forum, about 20 blogpost from a guy working at redhat on virt and a mailing list where people actually answer your questions.
> 
> Arch is great for building your custom kernel, was really easy compared to do it on fedora... (also it took me 3 different oses to do what i wanted, since i started on centos but the damn thing being on lts didn't have a kernel recent enough).
> There is a great thread on arch forums bug those modos are nazys mate, they closed it because a lot of folks who didn't use arch went on the forum ask for help, such a pity... since then the community apart from the mailing list kinda disolved.
> 
> Speaking of benchmarks i did some with 3dmark and i was only 3% behind in terms of performance between native win10 and vm win10.
> It was due in fact to the cpu consumption of arch linux.
> 
> Anyway if you guys are looking for vm files or stuff like that i can share my former config : https://github.com/tirrorex/Virtual-machine


Yes, I mean KVM.

I can feel your pain - building custom kernel and the like. However, given the right hardware, VGA passthrough is pretty easy. This is why I recommend Intels Xeon-based desktop CPUs, that is CPUs for the X299 or the older X99 chipsets. For example the 8-core Intel i7-7820X, or the 6-core i7-7800X. I'm not sure about the i7-7740X, if it offers ACS capabilities then it might be a good and less expensive choice for a gaming rig.

The Z170 / Z270 based CPUs are a mixed bag - some work OK, some need the ACS patch. That is where Arch Linux comes in - you don't need to patch, just select the linux-vfio package. Aside from that and their documentation I don't see a reason to switch from my favorite distribution to Arch. But that's a matter of taste - Arch is most certainly an excellent distribution.

I wouldn't use the harsh words you used (nazis), but the mods of the Arch forum could have been more tolerant and not closed the thread. It was the best thread on kvm and VGA passthrough - ever! I'm sure that quite some people switched to Arch just because of that thread.

Thanks for the VM files - need to look at them.


----------



## zipeldiablo

Quote:


> Originally Posted by *powerhouse*
> 
> Yes, I mean KVM.
> 
> I can feel your pain - building custom kernel and the like. However, given the right hardware, VGA passthrough is pretty easy. This is why I recommend Intels Xeon-based desktop CPUs, that is CPUs for the X299 or the older X99 chipsets. For example the 8-core Intel i7-7820X, or the 6-core i7-7800X. I'm not sure about the i7-7740X, if it offers ACS capabilities then it might be a good and less expensive choice for a gaming rig.
> 
> The Z170 / Z270 based CPUs are a mixed bag - some work OK, some need the ACS patch. That is where Arch Linux comes in - you don't need to patch, just select the linux-vfio package. Aside from that and their documentation I don't see a reason to switch from my favorite distribution to Arch. But that's a matter of taste - Arch is most certainly an excellent distribution.
> 
> I wouldn't use the harsh words you used (nazis), but the mods of the Arch forum could have been more tolerant and not closed the thread. It was the best thread on kvm and VGA passthrough - ever! I'm sure that quite some people switched to Arch just because of that thread.
> 
> Thanks for the VM files - need to look at them.


I have a x99 cpu (i7 5960x) and i did use the linux-vfio package, however I needed some custom things on the kernel (not a clue what it was







) and so I had to patch it (might be related to usb hardware as i was doing usb passthrough too).
I saw some guys with other components, damn i was sorry for them


----------



## powerhouse

Quote:


> Originally Posted by *zipeldiablo*
> 
> I have a x99 cpu (i7 5960x) and i did use the linux-vfio package, however I needed some custom things on the kernel (not a clue what it was
> 
> 
> 
> 
> 
> 
> 
> ) and so I had to patch it (might be related to usb hardware as i was doing usb passthrough too).
> I saw some guys with other components, damn i was sorry for them


Interesting. I got the 3930K and 5960X should not be anything behind mine. I am also passing through 2 USB controllers, one of which is part of my X79 chipset. I never had the need to use the ACS patch.

Can you post your IOMMU groups before you patched the kernel? I'm quite surprised that you need a patch with that CPU.

I'm quite confident that if you were to do VGA passthrough with a modern graphics card that has a UEFI BIOS, using Windows 10 (or Windows 8) with UEFI boot, you should have no problems. In the worst case, you'd need an extra USB3 PCIe card, which is real cheap. Of course, if the motherboard BIOS is screwed, you may run into all sorts of problems. What's your motherboard?

EDIT: Just looked at your config file (xml sucks). I see you do UEFI boot. Haven't had the patience to extract the essence out of the xml garbage. So here is my start script. The interesting part starts at qemu-system...:

#!/bin/bash

configfile=/etc/vfio-pci.cfg
vmname="win10vm"

vfiobind() {
dev="$1"
vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
device=$(cat /sys/bus/pci/devices/$dev/device)
if [ -e /sys/bus/pci/devices/$dev/driver ]; then
echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
fi
echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

if ps -A | grep -q $vmname; then
zenity --info --window-icon=info --timeout=15 --text="$vmname is already running." &
exit 1
else
cat $configfile | while read line;do
echo $line | grep ^# >/dev/null 2>&1 && continue
vfiobind $line
done

export QEMU_AUDIO_DRV=alsa
export QEMU_ALSA_ADC_BUFFER_SIZE=1024 QEMU_ALSA_ADC_PERIOD_SIZE=256
export QEMU_ALSA_DAC_BUFFER_SIZE=1024 QEMU_ALSA_DAC_PERIOD_SIZE=256
export QEMU_AUDIO_DAC_FIXED_SETTINGS=1
export QEMU_AUDIO_DAC_FIXED_FREQ=44100 QEMU_AUDIO_DAC_FIXED_FMT=S16 QEMU_AUDIO_ADC_FIXED_FREQ=44100 QEMU_AUDIO_ADC_FIXED_FMT=S16
export QEMU_AUDIO_DAC_TRY_POLL=1 QEMU_AUDIO_ADC_TRY_POLL=1
export QEMU_AUDIO_TIMER_PERIOD=50

cp /usr/share/OVMF/OVMF_VARS.fd /tmp/my_vars.fd
chown qemu-vga:qemu-vga /tmp/my_vars.fd

#taskset -c 0-9
qemu-system-x86_64 \
-monitor stdio \
-serial none \
-parallel none \
-nodefaults \
-nodefconfig \
-name $vmname,process=$vmname \
-machine q35,accel=kvm,kernel_irqchip=on,mem-merge=off \
-cpu host,kvm=off,hv_vendor_id=1234567890ab,hv_vapic,hv_time,hv_relaxed,hv_spinlocks=0x1fff \
-smp 12,sockets=1,cores=6,threads=2 \
-m 20G \
-mem-path /run/hugepages/kvm \
-mem-prealloc \
-balloon none \
-rtc base=localtime,clock=host \
-vga none \
-nographic \
-soundhw hda \
-device vfio-pci,host=02:00.0,multifunction=on \
-device vfio-pci,host=02:00.1 \
-device vfio-pci,host=00:1a.0 \
-device vfio-pci,host=08:00.0 \
-drive if=pflash,format=raw,readonly,file=/usr/share/OVMF/OVMF_CODE.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd \
-boot order=c \
-drive id=disk0,if=virtio,cache=none,format=raw,aio=native,file=/dev/mapper/lm13-win10 \
-drive id=disk1,if=virtio,cache=none,format=raw,aio=native,file=/dev/mapper/photos-photo_stripe \
-drive id=disk2,if=virtio,cache=none,format=raw,aio=native,file=/dev/mapper/media-photo_raw \
-netdev type=tap,id=net0,ifname=vmtap0,vhost=on \
-device virtio-net-pci,netdev=net0,mac=00:16:3e:00:01:01

#EOF

exit 0
fi

The rest of the setup you find here: https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/


----------



## zipeldiablo

Quote:


> Originally Posted by *powerhouse*
> 
> Interesting. I got the 3930K and 5960X should not be anything behind mine. I am also passing through 2 USB controllers, one of which is part of my X79 chipset. I never had the need to use the ACS patch.
> 
> Can you post your IOMMU groups before you patched the kernel? I'm quite surprised that you need a patch with that CPU.
> 
> I'm quite confident that if you were to do VGA passthrough with a modern graphics card that has a UEFI BIOS, using Windows 10 (or Windows 8) with UEFI boot, you should have no problems. In the worst case, you'd need an extra USB3 PCIe card, which is real cheap. Of course, if the motherboard BIOS is screwed, you may run into all sorts of problems. What's your motherboard?


Mate It was almost a year and a half ago when i said i don't have a clue, i meant i don't remember.

Also i can't post anything, since then i switched to a 4k monitor and before the release of the gtx 1080ti i had a 980ti which wasn't powerfull enought, so i had to go sli









And since it wasn't possible i moved from linux main os to win10 main os.
Also now with linux inside windows i pretty much don't need linux on my main computer, especially since i also have a macbook pro for work.
(also i could finally get rid of my zfs array and move to a 10to hdd o/ )

I had an extra usb 3 pcie card by the way, which i used for usb passthrough.
Speaking of motherboard, i am wondering if i had the same cpu back then, quite possible i was on a different chipset, time flies so...


----------



## powerhouse

Quote:


> Originally Posted by *zipeldiablo*
> 
> Mate It was almost a year and a half ago when i said i don't have a clue, i meant i don't remember.
> 
> Also i can't post anything, since then i switched to a 4k monitor and before the release of the gtx 1080ti i had a 980ti which wasn't powerfull enought, so i had to go sli
> 
> 
> 
> 
> 
> 
> 
> 
> 
> And since it wasn't possible i moved from linux main os to win10 main os.
> Also now with linux inside windows i pretty much don't need linux on my main computer, especially since i also have a macbook pro for work.
> (also i could finally get rid of my zfs array and move to a 10to hdd o/ )
> 
> I had an extra usb 3 pcie card by the way, which i used for usb passthrough.
> Speaking of motherboard, i am wondering if i had the same cpu back then, quite possible i was on a different chipset, time flies so...


Wow, didn't notice that this was so long ago. Glad you found a way to enjoy both worlds - Linux and Windows. It looks like you got some nice hardware, too.

I myself would not want to run Windows as a host OS - too many bad memories from deteriorating disk / general performance to "disk not readable" since Microsoft dropped support for disk compression (moving from Win98 to 2000 or whatever was the next) to - surprise surprise - virus infection, despite commercial anti-virus software etc. The list goes on.

Since I run Windows 10 Pro in a VM on lvm formated drives, backups are easy - create snapshot and backup, even while Windows is running. A 110 GB Windows partition on SSD takes me about 10 minutes to backup to HDD, and about the same time to restore. I have always 2 consecutive backups, plus a "golden copy" when I installed Windows and the software I use. All copies are stored both local on a backup drive and remote on a server. If my CPU/motherboard dies on me, I can replace/upgrade both CPU and m/b and boot my Windows VM as if nothing happened.

Sometimes I install software on Windows for trials, or experiment with different settings inside Windows. If I'm not happy with the results, I simply restore Windows from a backup. This would be a lot more complicated if Windows was installed on bare metal.

Backup and restore, including remote backup/restore, is all done in Linux. Linux has lots of utilities for handling and safekeeping data, and all of them free. I'm not a script wizard and often use GUI applications to get the job done. But sometimes I use scripts since they are unbeatable.

The reason I replied to your previous post was that I couldn't understand why your hardware wouldn't work with VGA passthrough - it should have been rather easy.

In any case, if you're interested, here is my how-to: https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/

Forgive me if I posted the link before.


----------



## zipeldiablo

Quote:


> Originally Posted by *powerhouse*
> 
> The reason I replied to your previous post was that I couldn't understand why your hardware wouldn't work with VGA passthrough - it should have been rather easy.
> 
> In any case, if you're interested, here is my how-to: https://heiko-sieger.info/running-windows-10-on-linux-using-kvm-with-vga-passthrough/


Probably because I had a different hardware back then, it was a real pain in the ass.
You do realize that I already know more on the subject than the sum of the content you posted on your website right?

Anyway, I had windows on Vm too but before switching it as main os I already moved it on baremetal, I needed to run benchmarks to compare performance between vm and host so...


----------



## powerhouse

Quote:


> Originally Posted by *zipeldiablo*
> 
> You do realize that I already know more on the subject than the sum of the content you posted on your website right?
> 
> Anyway, I had windows on Vm too but before switching it as main os I already moved it on baremetal, I needed to run benchmarks to compare performance between vm and host so...


I didn't know. Do you have any place you share your knowledge? I'm always interested to learn more.


----------



## zipeldiablo

Quote:


> Originally Posted by *powerhouse*
> 
> I didn't know. Do you have any place you share your knowledge? I'm always interested to learn more.


No i don't, there is enough place as it is on the web








The blog from the redhat guy is a good start (alex richardson or something, it should be the first result when it comes to vfio), the mailing list is top-notch if you have any issue and you can still find 182 pages of information on arch-linux forum


----------

