VT-d Verification on ASRock Z87 Extreme6 with ESXi 5.5

This is one of the best enthusiast ESXi virtualization motherboard I have come across so far: ASRock Z87 Extreme6 (maybe second to ASRock Z77 Extreme11).
Some of the highlights:

  • Onboard USB header
  • VT-d passthrough
  • Debug LED
  • Dual Intel NICs

I’m very fond of installing ESXi on a USB stick to separate it from the rest of the storage. Previously I have removed the metal bracket from “onboard-USB-header-to-rear-I/O-bracket” (whatever do they call these things?) that used to come along with motherboards a couple of years back, to have an internal USB header.

Secondly, ASRock is one of the few manufacturers that supports VT-d on as many of their motherboards as possible. Even on boards with a chipset not officially supporting VT-d, like the Z77. Here is my previous test of ASRock Z77 Pro4-M. See below for some VT-d tests.

Finally, the debug code LED. This brings some lovely overclocking memories back to life. One of the main reasons for choosing Abit over Asus back in the P3 and P4 days was this very tiny feature. The POST codes proved to be extremely valuable when pushing the very last bit of performance out of a system. Anyway, I’m way off track. Let’s look at some pictures and some of my other findings.

Extreme6-overview USB-header-post-code-led intel-i211 sata-ports

Specifications

  • Intel Z87 chipset
  • Intel Haswell CPU support, socket 1150
  • Four DIMMS, supporting 32GB RAM
  • Dual Intel Gigabit LAN (I217V + I211AT)
  • USB socket directly on the board
  • POST debug code display
  • 8x USB3

The dual Intel network controllers is a real positive thing for us ESXi persons and I was really exited about this. But, neither of these controllers are supported by ESXi 5.5 out of the box. The I217 NIC can be made to work if a newer driver is supplied. However, the I211 NIC is a scaled down desktop version, as far as I have read and I have not yet found a way to make work.

Apart from the Z87 based SATA controller, there are also four additional ports connected to two controllers made by ASMedia. Two ports to each controller, whereas one port double as a ESATA port. The name of the controllers are “ASM1062”.

In total, there are eight USB3 ports where four are connected to the Z87 chipset and four connected to a controller also made by ASMedia. The four rear I/O ones are connected to the ASMedia controller and the four internal ones to the Z87 chipset. I am not entirely sure if I think this was the best move by ASRock. I can argue either way.

VT-d testing

While gathering fact about the board on the Internet I came across some comments fearing the removal of the VT-d setting for some boards and with some BIOS versions. This board, with the shipping BIOS (P2.10), has the VT-d option. According to some BIOS release notes, even though the VT-d option is removed it is supposed to be enabled by default if all criterias are met.

ESXi 5.5 is capable of utilizing the VT-d functionality with this board. More specifically DirectPath I/O as VMware calls it. Here are some of my findings:

  • Both NICs can be passed on to a VM, even though none of them are supported out of the box by ESXi 5.5, and only one can be made operational with additional drivers.
  • Both ASMedia SATA controllers show up and can be passed on to a VM. However, when I tried this with a Ubuntu 13.10 VM I could not get any drive connected to show up. The controller is detected in the VM but that is all. The controller is detected by Ubuntu 13.10 when run natively.
  • The ASMedia USB3 controller does not show up in the passthrough view. There are two Z87 USB controllers showing up but it is not clear to me if either or both of them are USB3.
  • An LSI 9211-8i card, actually a flashed M1015, inserted into the top most PCI-Express 16x slot did not show up as available for passthrough. The card works in this port when running Ubuntu 13.10 natively. Inserting the card in the second slot made it available for passthrough and it worked well with Ubuntu 13.10.

Here is a screenshot of the default available passthrough devices:

ESXi-ASRockZ87Extreme6

Summary

All in all, a board with great potential. If drivers arrive for the second Intel NIC and some of the passthrough issues are sorted out it will be a killer motherboard. Nothing critical by any means, but the flexibility is reduced if you need to use one PCI(-Express) slot for another NIC, all SATA ports or an separate RAID controller.

I will hopefully be able to present upcoming articles on how to customize an ESXi image for this board and present some storage benchmarks.

*Update 2014-01-04*
Here is how to include a driver the one of the Intel network controllers on this board:
Include Intel network drivers in an ESXi ISO
Also, here is some storage benchmarks related to this board:
Storage performance of ASRock Z87 Extreme6

Some quick HDD and SSD benchmarks

I have been able to run some benchmarks on various hard drives and a solid state drive. Mostly for my own amusement to see how old drives compares to new drives. There are some desktop drives as well as some enterprise drives. Perhaps the numbers can be useful for someone.

The drives

Listed in some kind of old/slow to new/fast

  • Seagate Barracuda 7200.11 (ST31500341AS), 1.5TB
  • Seagate Barracuda 7200.12 (ST500DM002), 500GB
  • Samsung SpinPoint F3 (HD502HJ), 500GB
  • Hitachi GST Deskstar 7K1000.D (HDS721010DLE630), 1TB
  • Western Digital Red (WD20EFRX), 2TB
  • Seagate SV35.6 Series (ST2000VX000), 2TB
  • Seagate Constellation CS (ST2000NC000), 2TB
  • Seagate Constellation ES (ST3000NM0033), 3TB
  • Intel 520 SSD (SSDSC2CW240A3), 240GB

Test system

  • Asus P8Z68-V (Intel Z68 chipset)
  • Intel 2600K
  • 2x4GB RAM
  • Ubuntu Desktop 10.04.3
  • Ubuntu Disk application used for the benchmarks

The system is kind of old, but I have collected the numbers for some time and wanted to run the drives on the same platform. The drives were connected to the onboard SATA-III/6G ports, connected to the Z68 chipset.

*Update* I believe I have screwed up and actually used the SATA 3G ports for some of the drives. I will rerun the benchmark with the Constellation ES and SSD drive and update this post. The other drives are in production and I’m unable to test them.

Results

ST31500341AS ST500DM002 HD502HJ HDS721010DLE630 WD20EFRX ST2000VX000 ST2000NC000 ST3000NM0033 SSDSC2CW240A3

Reflections

I am not going to do an in depth analysis of the results, since I realize the procedure was way too sloppy. There are some really strange write results for the Constellation ES drive shown here. I tried running the same benchmark with Ubuntu 12.04 and it was more consistent with less spikes/dips.

Hopefully I will be able to post some other interesting benchmarks soon.