This is one of the best enthusiast ESXi virtualization motherboard I have come across so far: ASRock Z87 Extreme6 (maybe second to ASRock Z77 Extreme11).
Some of the highlights:
- Onboard USB header
- VT-d passthrough
- Debug LED
- Dual Intel NICs
I’m very fond of installing ESXi on a USB stick to separate it from the rest of the storage. Previously I have removed the metal bracket from “onboard-USB-header-to-rear-I/O-bracket” (whatever do they call these things?) that used to come along with motherboards a couple of years back, to have an internal USB header.
Secondly, ASRock is one of the few manufacturers that supports VT-d on as many of their motherboards as possible. Even on boards with a chipset not officially supporting VT-d, like the Z77. Here is my previous test of ASRock Z77 Pro4-M. See below for some VT-d tests.
Finally, the debug code LED. This brings some lovely overclocking memories back to life. One of the main reasons for choosing Abit over Asus back in the P3 and P4 days was this very tiny feature. The POST codes proved to be extremely valuable when pushing the very last bit of performance out of a system. Anyway, I’m way off track. Let’s look at some pictures and some of my other findings.
Specifications
- Intel Z87 chipset
- Intel Haswell CPU support, socket 1150
- Four DIMMS, supporting 32GB RAM
- Dual Intel Gigabit LAN (I217V + I211AT)
- USB socket directly on the board
- POST debug code display
- 8x USB3
The dual Intel network controllers is a real positive thing for us ESXi persons and I was really exited about this. But, neither of these controllers are supported by ESXi 5.5 out of the box. The I217 NIC can be made to work if a newer driver is supplied. However, the I211 NIC is a scaled down desktop version, as far as I have read and I have not yet found a way to make work.
Apart from the Z87 based SATA controller, there are also four additional ports connected to two controllers made by ASMedia. Two ports to each controller, whereas one port double as a ESATA port. The name of the controllers are “ASM1062”.
In total, there are eight USB3 ports where four are connected to the Z87 chipset and four connected to a controller also made by ASMedia. The four rear I/O ones are connected to the ASMedia controller and the four internal ones to the Z87 chipset. I am not entirely sure if I think this was the best move by ASRock. I can argue either way.
VT-d testing
While gathering fact about the board on the Internet I came across some comments fearing the removal of the VT-d setting for some boards and with some BIOS versions. This board, with the shipping BIOS (P2.10), has the VT-d option. According to some BIOS release notes, even though the VT-d option is removed it is supposed to be enabled by default if all criterias are met.
ESXi 5.5 is capable of utilizing the VT-d functionality with this board. More specifically DirectPath I/O as VMware calls it. Here are some of my findings:
- Both NICs can be passed on to a VM, even though none of them are supported out of the box by ESXi 5.5, and only one can be made operational with additional drivers.
- Both ASMedia SATA controllers show up and can be passed on to a VM. However, when I tried this with a Ubuntu 13.10 VM I could not get any drive connected to show up. The controller is detected in the VM but that is all. The controller is detected by Ubuntu 13.10 when run natively.
- The ASMedia USB3 controller does not show up in the passthrough view. There are two Z87 USB controllers showing up but it is not clear to me if either or both of them are USB3.
- An LSI 9211-8i card, actually a flashed M1015, inserted into the top most PCI-Express 16x slot did not show up as available for passthrough. The card works in this port when running Ubuntu 13.10 natively. Inserting the card in the second slot made it available for passthrough and it worked well with Ubuntu 13.10.
Here is a screenshot of the default available passthrough devices:
Summary
All in all, a board with great potential. If drivers arrive for the second Intel NIC and some of the passthrough issues are sorted out it will be a killer motherboard. Nothing critical by any means, but the flexibility is reduced if you need to use one PCI(-Express) slot for another NIC, all SATA ports or an separate RAID controller.
I will hopefully be able to present upcoming articles on how to customize an ESXi image for this board and present some storage benchmarks.
*Update 2014-01-04*
Here is how to include a driver the one of the Intel network controllers on this board:
Include Intel network drivers in an ESXi ISO
Also, here is some storage benchmarks related to this board:
Storage performance of ASRock Z87 Extreme6
Hey, could you elaborate on getting that integrated Intel NIC to work? I’m looking at doing a similar setup and this would save me buying a discrete one.
Hello, yes I’m working on writing an article on that topic. Please stay tuned for it! Best Regards – Robert
Here is how to include a driver for one of the NICs
https://blog.kihltech.com/2014/01/how-to-add-intel-nic-drivers-to-an-esxi-5-5-iso/
Very helpful entry – thank you. I’ve recently bought the ASRock Z87E ITX board but it doesn’t seem that vt-d is supported on this version.
Was there any configuration required in the UEFI to enable this? I’m using a i7 4770K CPU and vt-x shows as supported, however under Chip Configuration, vt-d is “unsupported”
Hello! As far as I’ve read, most of the ASRock boards support VT-d, but I have not tried that card myself and I have not searched the Internet for it so I can’t tell.
However, I think the issue you are experiencing is that the “K” version CPUs does not support VT-d. Here is a quick link to VT-d supported CPUs by Intel:
http://ark.intel.com/search/advanced?VTD=true
Did someone test ASRock Z87E-ITX with ESXi? I am interesting to compatibility and passthrough support, because I consider this motherboar for my build instead of GIGABYTE GA-Z87N-WIFI.
I have not tried the ASRock Z87E-ITX board so I can’t comment on it specifically. Generally, ASRock has good support for VT-d/Directed IO/DirectPath/passthrough on many of their boards. Gigabyte also seems to have support for VT-d on some of their boards. Which ever board you chose, let us know your experience with it. Best Regards – Robert
Hi,
I have a Asrock Z87E-ITX and everything except the Lynx SATA controller seems to work with passthrough. And I haven’t seen any workaround so far – did anyone else have that?
Regards,
Aladdin
Hi,
Good info, thanks for the heads up.
I have not tried passing on the SATA controller since I have used it for storage.
Best Regards,
Robert
I will, but it could take some time 😉 I also consider buying HP Microserver Gen8 and update CPU and RAM, but I am not sure if there will work GPU passthrough.
Pingback: Anonymous
Any luck passing through USB ports/controllers? I tried passing through both Lynx USB controllers but the VM doesn’t see them.
Hi Cody,
Yes, I have connected USB peripherals to VMs with this board. I do it by adding a USB Controller in the settings for the specific machine, then it’s possible to add the specific device. However, this is not the same as passing on the entire USB controller/host to the VM. If that is what you need I guess I could try it when I get the opportunity to take the board off “production”.
Best regards,
Robert
Yes, I was referring to passing through the entire controller to a VM. No luck for me when I tried, but I’m still going to mess with it a bit before giving up.
Thanks!
Hi, thanks for all the info!
I’m wondering though, from what I understand, it looks like VT-d isn’t working almost at all, you had to use a separate I/O card to get storage, and even then that card worked only on a specific slot.
Since I’m in the process of building my own VT rig I was looking for some insight on desktop boards that could handle VT-d, but I wasn’t able to get any satisfying info. Did you get that onBoard controller to work in the end?
Thanks!
Hi Marlon, thanks for the feedback.
I would not summarize it exactly the way you did. Using the onboard SATA controller for storage to ESXi works absolutely fine. Using it for passthrough in ESXi would be a catch 22, I would say. (Passing the onboard controller to a VM that resides on the same storage, if no other storage is used). ESXi does not have support for ASMedia controllers. I can only guess why I could not get passthrough to work with the ASMedia controller to a VM and that is the controller itself. PCI/-E devices need to have certain capabilities to work in passthrough mode.
VT-d has some specific usages and I’m not sure I understand how you intend to use it. If you’re just interested in getting a virtual environment up and running, pretty much any 7- or 8-series motherboard will do just fine. An Intel NIC and you’re all set. If you need VT-d and a consumer motherboard there are a few manufacturers that support it, along with some specific CPUs as I’m sure you know.
Let me know what system you ended up with!
Best Regards,
Robert
Asrock Z87 with esxi 5.5 SATA3 mode works?
I installed version esxi 5.5 custom reports only a single network card;
installing any other network card on pci slot is not detected by the custom version of esxi 5.5,
with the standard version of esxi 5.5 does not work the onboard network adapters but work on the pci slot
Yes, I have also only been able to get one of the onboard controllers to work with customized versions of the installer. I have installed the Z87 Extreme 6 with a customized image for one of the onboard controllers and at the same time used two additional single port PCI-Express cards and one dual slot PCI card. I have no explanation to what you experience. What NIC is it and have you tried the other PCI-port?
Best Regards,
Robert
I redid the custom installation cd, now work on PCI add-in cards, so it was a problem with the cd …
Asrock Z87 with esxi 5.5 SATA3 mode works?
thanks
Nice to hear that it worked out. I have tried the onboard Z87 SATA ports together with ESXi and they work. But I have not tried or verified that they operate at SATA3/6G speeds.
Best Regards,
Robert
hi there,
is it possible to passthrough a VGA card in the topmost pcie16x port to a vm-guest?
Hi Bernd,
I have not tried passing through a graphics card so I can’t tell if it is working or not.
Please let me know if and how it works if you try it!
Best regards,
Robert
hi,
so i bought all that stuff i needed. same board, different cpu, lots of rams etc…
with ESXi it was not possible to passthrough anything in the two top most pci-e-16x slots.
but it works with libvirt 🙂
vga in topmost 16x slot
raid controller (areca 1230) in the 2nd.
though, i wasnt able to passthrough the sound device yet.
Interesting, can you elaborate a little more on what type of cards you tried to pass through and to what OS in the VM?
I have not tried anything with graphics cards yet, but there are some really cool things that is possible with it 🙂
Best regards,
Robert
well, i wanted to have a streamingbox for Steam-in-home-streaming… 😀
Host: ubuntu 14.0
virtualization software: libvirt
guestvm1: windows vista 32bit (only windows i have lying around here 🙁 )
guestvm2: ubuntu
first 16x slot: radeon r7 260x
second 16x slot: areca arc 1230 RAID HBA
radeon for the first vm
areca for the second vm.
i am still fixing some stuff for it to work 100%.
the sound-dev of the radeon card seems to be troublesome. i cannot pass it through somehow.
i havent bought the board yet.
Hi,
Anyone already tried the new model Z97 Extreme6?
I’m thinking of buying this one or the z87 model. The reason of doubt is that I really want vt-d to work on Esxi and I’m no sure if it will work on the new Z97 Extreme6.
Good question, I have not looked into the Z97 chipset for this application. If you find something or try it out, be sure to let us know!
Best regards,
Robert
Hello,
i have tested the Z97 Extreme6 with ESXI 5.5.
I could passthrough the asmedia USB PCI Device to a virtual Maschine
and i was able to passthrough a IBM 1018 /lsi to a virtual maschine.
this worked so well, tha i even was able to update the firmware.
Thank you very much for the information Dominik! Did you manage to get the onboard Intel NIC and/or Realtek NIC to work?
Best Regard – Robert
I haven’t tried to pass them through to a virtual maschine yet.
I’ll try this if i find the time this weekend.
btw the passthrough of the IBM Serveraid/LSi worked in the slot in the middle for me.
in the Slot on the edge of the board i’m using an Intel Dual Gigabit Network card.
Thanks again for the info. I meant to ask if you got any of the onboard NICs to work with ESXi, by adding drivers to the ISO or adding them afterwards with a VIB or similar?
Best Regards – Robert
Hi – thanks for the info!
I have recently bought an ASUS Z87 Deluxe as it was for a good price and pretty much appeared to be the same board as this. However when I booted up ESXi I found that pass-through wasn’t available.
After looking in the bios I found the option for VT-d, enabled it and went back in to ESXi, this time pass-through was enabled but I couldn’t see my LSI RAID card in the top PCI-E slot. The pass-through list looked identical to the one in this article,
I am hoping moving it to the second slot tonight will fix this 🙂
Update to last post. Tried the LSI RAID card in the 2nd slot and again it wasn’t detected (not even at boot) but it did work in the 3rd PCI-E slot. I didn’t try changing the speed or generation of the first two slots but it could be something to do with that.
Thanks for the information Adrian!
– Robert