Maxim DS9490R 1-Wire + Ubuntu 18.04 Server + python-ow

After having some challenges finding a simple way to use a Maxim (previous Dallas) DS9490R USB to 1-Wire/iButton adapter on Ubuntu 18.04 LTS Server (Bionic Beaver) with python-ow I decided to write down my experiences here.

Orientation

Plug it in! You should see something like:

$ lsusb
Bus 002 Device 004: ID 04fa:2490 Dallas Semiconductor DS1490F 2-in-1 Fob, 1-Wire adapter

Run the following command to load the built in kernel module:

$ sudo modprobe ds2490

You will now be able to reach the 1-wire bus master at this location:

$ ls -l /sys/bus/w1/

If you have a working temperature sensor, like the DS18B20, you can run the following command and get its reading

cat /sys/bus/w1/28*/w1_slave
71 01 4b 46 7f ff 0f 10 56 : crc=56 YES
71 01 4b 46 7f ff 0f 10 56 t=22125

I would like to use Python to handle the temperature in order to make good use of the temperature and it’s quite possible to just read this value from the file system. However, I would like to use the python-ow package instead.

python-ow

Start off with installing the python-ow package with:

$ sudo apt-get install python-ow

This will also install the owfs-common package. Launch a python terminal to verify the installation:

$ python
Python 2.7.15rc1 (default, Apr 15 2018, 21:51:34)
[GCC 7.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import ow
>>> ow.init('u')
 Could not open the USB bus master. Is there a problem with permissions?
Traceback (most recent call last):
  File "", line 1, in 
  File "/usr/lib/python2.7/dist-packages/ow/__init__.py", line 224, in init
    raise exNoController
ow.exNoController

import ow should work but you are most likely going to run into the permission issue, as can be seen here after the ow.init(‘u’) command. It is possible to run python as root with sudo python and it will not throw an error if all works correctly. However, it’s a bad idea to run all scripts with root so we need to take care of this issue.

libusb permissions

A kernel module (ds2490) is loaded by default which is why the first step works fine. However, python-ow and owfs use the libusb library and we need to disable the kernel module for this to work. Edit the kernel module blacklist:

$ sudo nano /etc/modprobe.d/blacklist.conf

and add the following to the end of the file:

# Disable auto loading of Dallas USB-1-wire adapter
blacklist ds2490

Next we need to allow a user to use this specific USB device. To do this we create a udev rule:

$ sudo nano /etc/udev/rules.d/99-one-wire.rules

and add the following content:

ATTRS{idVendor}=="04fa", ATTRS{idProduct}=="2490", GROUP="plugdev", MODE="0664"

What this does is giving the user group plugdev access to the USB adapter. idVendor and idProduct points to the USB device as can be seen with the lsusb command shown earlier. Normal users are members of the plugdev group by default – verify with the groups command.

More information on the section can be found here:
https://www.maximintegrated.com/en/app-notes/index.mvp/id/5917

Important: Reboot the OS in order for all changes to take effect.

Verify functionality

Let’s see if it works. You can launch a python shell and enter the snippets shown earlier. Here is a more complete code section to try out:
(Stolen from https://raspberrypi.stackexchange.com/questions/62292/usage-of-python-ow-one-wire-file-system-python-package-for-reading-1-wire-ds18b2 and modified to taste)

#!/usr/bin/python
import ow
ow.init('u')

sensorlist = ow.Sensor('/').sensorList()

for sensor in sensorlist:
   print('Device Found')
   print('Address: ' + sensor.address)
   print('Family: ' + sensor.family)
   print('ID: ' + sensor.id)
   print('Type: ' + sensor.type)
   if sensor.type == 'DS18B20':
      print('Temperature: ' + sensor.temperature)
   print(' ')

SSH: Conditional host address based on network or location

I have been searching extensively to find a solution to ease management and administration of my servers from multiple location. Let me present my environment and then my use cases

Environment

  • Site One (10.0.1.0\24)
    • Server A
    • Server B
    • Server C
  • Site Two (10.0.2.0\24)
    • Server D
    • Server E
  • Site Three (10.0.3.0\24)
    • Server F

Use cases

I manage these servers from any of the sites and even other external networks and the two main issues I have had are

  1. Reaching any server from any other server, workstation or laptop in the field with:
    ssh server-x
  2. Be able to bridge multiple server jumps. Normally, only “A”, “D” and “F” have an externally port forwarded to it.
    For example, sending a file from “B” to “E” is troublesome with

    rsync local_file server-e:.

My solution

Perhaps my Google skills are lacking, but finding a solution that is efficient across all locations and dynamic was hard. There were quite a few samples of having two settings in the ssh config file for each host, i.e. server-x-local and server-x-remote. Sure, doable but a pain to maintain across all sites and not transparent enough.

The solution to this problem was the Match keyword for the ssh config. This is the configuration for Server A:

Match Originalhost server-a Exec "ifconfig | grep 10.0.1"
     Hostname 10.0.1.2
Host server-a
     Hostname external.siteone.com

Let’s explain this construct. The match clause executes the ifconfig and pipes the output to grep in order to see if we are on this subnet. This will evaluate to true if I’m located on any host on subnet One. If the match is true it will set the hostname to the local IP of the server. If the clause evaluates to false, the default (external DNS) hostname of the server will be used. According to the ssh_config man page, the first match of a setting will be used. Therefor, the Match clause needs to be before the Host.

This works great from my initial testing and makes it possible to use the very same ssh config file regardless of client.

The second issue: Jump/Proxy/Bastion setup.

In order to reach other servers behind the externally connected servers easily I had previously been using SSH tunneling to solve some tricky cases. Now that I had some Google flow I attempted to take a stab at that challenge as well.

As it turns out, this was an easier fix thanks to the ProxyCommand feature in later SSH versions. From my understanding, this executes a custom command as a precondition to the final SSH connection.

Expanding on the previous solution, this is what I came up with. First, the complete Server A ssh configuration, same as before.

Match Originalhost server-a Exec "ifconfig | grep 10.0.1"
     Hostname 10.0.1.2
Host server-a 
     Hostname external.siteone.com

Then, the Server B configuration:

Match Originalhost server-b Exec "ifconfig | grep 10.0.1"
     ProxyCommand none
Host server-b
     Hostname 10.0.1.3
     ProxyCommand ssh -W %h:%p server-a

The idea here is the same. The default case is connecting from an external site and the ProxyCommand initiates an ssh connection to server-a first and uses it as a proxy to then connect to server-b. If, on the other hand, we are located on the local subnet already, the ProxyCommand is disabled and no proxy connection to server-a is made.

All that remains now is to configure all different hosts at one client and the roll out the very same config file to any host that is going to connect to any of the servers.

Conclusion

I’m really happy with this solution and it has greatly improved my efficiency when connecting to the different servers. Now the connections are really transparent and flexible!

Let me know if you have any success with this solution!

Credits

http://sshmenu.sourceforge.net/articles/transparent-mulithop.html

https://unix.stackexchange.com/questions/150002/only-apply-match-keyword-to-single-host-in-ssh-config/

 

VT-d Passthrough Verification: ASRock Z97 Extreme 6 + Intel Core i7 4790K

asrock-extreme6-and-intel-4790k

I have had the ASRock Z97 Extreme 6 + Intel Core i7 4790K combination in my workstation for a time and only briefly tried running ESXi 5.5 while testing some network and RAID drivers. Previously I have had great use for VT-d in my server, but since this combo was for my workstation I chose a CPU only based on performance. Now, many months later, I stumbled on the fact that this very CPU supports VT-d, in contrast to previous ‘K’ model CPU. Since VT-d is not widely used, I couldn’t find anyone else that has tested the VT-d support on this hardware, which is why I decided to find out if it does.

System specification

BIOS Setup of the ASRock Z97 Extreme 6

The settings for VT-x and VT-d are hidden under different menus in the UEFI BIOS. VT-x is found under CPU configuration as “Intel Virtualization Technology” and VT-d under Chipset configuration with the same name. VT-x was enabled by default but VT-d was disabled. Here is also a note showing VT-d support or not. As we can see in the image below, VT-d is supported with this CPU and motherboard. asrock_z97_extreme6_bios_vt-d

ESXi Installation

To get support for both of the onboard NICs, please see my other post regarding driver inclusion to an ESXi ISO.

I usually install ESXi on a USB thumbdrive to separate the ESXi installation from the datastore. This is made even more easy on some of ASRock’s motherboards since they have an onboard USB header. For this simple test, I decided to use a single SSD connected to the onboard SATA controller.

Once ESXi is installed the passthrough settings can be found under ConfigurationAdvanced Settings. For this test I chose to pass through the following devices:

  • Onboard Realtek NIC
  • Onboard ASMedia USB3 controller
  • LSI 9211-8i SAS HBA

VT-d verification

I began with just launching a VM with an Ubuntu 14.04 live ISO. As we can see with lspci, all devices were found correctly:ubuntu_lspci

I connected an external 500GB disk to the USB controller and a 3TB disk and they were both found. I actually had a USB mouse connected to the USB controller by chance, and it worked as well šŸ™‚ubuntu_disk_by_pathubuntu_dmesg

Moving on to Windows and installing Windows 7 x64 SP1. After some driver installation all was working fine:windows7_device_managerwindows7_system_properties

Final words

Everything was working really well with this combination and it is a welcomed surprise to see Intel enable VT-d for the Devil’s Canyon CPUs. Only the Core i7-4790K was tested at this point but it should be pretty safe to assume that the Core i5-4690K will work as well.

Manage an LSI MegaRAID card in ESXi host remotely with MSM

Here is a quick post on how to remotely manage an LSI MegaRAID card in an ESXi host with MegaRAID Storage Manager, aka MSM.

Setup

  • ESXi 5.5 u2 host
  • LSI MegaRAID SAS 9261-8i (this guide will work on most 926x and 927x cards)
  • Windows 7 SP1 physical client

Required software

How to get it working

I have read multiple guides on doing this very simple thing. However, most of the tricks did not work or was not an issue for me. Here is what was needed for me to get it working with this setup.

  1. Make sure the LSI SMIS provider is working. Do you get health indications from the RAID card in vSphere?Ā Installed software components. If not; stop here, install it and make sure it is working.
  2. Enable SSH on the host and connect to the host over SSH
  3. View the hosts file with cat /etc/hosts
  4. Copy the line with the IP address to the server, for example:
    172.17.1.1Ā  hostname.domain hostname
  5. On the client Windows machine, edit the hosts file* and add a row for each hostname found in step 4, for example:
    172.17.1.1Ā  hostname.domain
    172.17.1.1Ā  hostname
  6. Start LSI MSM from the client. Change the search setting to ESXIMON servers, save, and then enter the IP of the local machine (not the host IP) in the search field.MegaRAID_storage_manager_host_configuration
  7. Hit search and the host should appear with the correct hostname and IP.MSM-search-results

(*) Right click on a link to Notepad (or a custom text editor) and choose Run as administrator. Select File – Open, and enter the following file name:

C:\Windows\System32\drivers\etc\hosts

I hope this helps the RAID administration! Let me know if you succeed or not in the comments.

ESXi 5.5 u2 on ASRock Z97 Extreme6 with dual NIC support

I finally got around trying out the ASRock Z97 Extreme 6 motherboard and how it is supported by ESXi 5.5-u2. There seems to be quite a few issues with getting the Intel I218 to work on some Z97 boards, while the very same driver have been tested to work well on Z87 boards with the Intel I217 controller, for example the ASRock Z87 Extreme 6.

I recently found that a VMWare forum user called GLRoman had managed to compile an updated Intel e1000e driver that is required for this NIC. By the way, I encourage everyone interested in getting drivers to work for ESXi to read through this thread and the threads it’s linking too. Very interesting!

How to add Intel and Realtek drivers to an ESXi 5.5 U2 ISO

While I was at it I also decided to try to add Realtek drivers to the ESXi 5.5-u2 ISO in an attempt to get both NICs on the Z97 ASRock Extreme6 board to work. Please see my previous blog post on adding the updated Intel driver to an ISO. I used the very same method for this test with the addition of the Realtek drivers. In summary, these are the packages, including links to them, that are required to get both NIC operational.

The Intel driver is an offline bundle while the Realtek driver packages are VIB files. The Realtek VIBs are compressed to a zip file and need to be extracted for the EXSi-Customizer-PS script.

How to make ASRock Z97 Extreme 6 boot ESXi when installed on USB

As I have said earlier, I’m fond of installing ESXi to a USB stick to make it separated from the datastores. From what I have understood, ESXi is using GPT by default and I did not manage to get it to boot with UEFI in that way. I found that it is possible to add an option to the installation process which uses MBR instead of GPT.

During boot of the installation media, press SHIFT + O when promted. A prompt with “runweasel” will appear. Press space and add “formatwithmbr“, press enter to continue the installation as normal.

ESXi support for onboard AHCI SATA controller

In the previous article I covered the issue of VMWare removing support for some onboard SATA controllers. I did not test whether or not ESXi 5.5 U2 would detect the onboard SATA AHCI controller on this motherboard without the sata-xahci driver package since I decided to include it right away. As can be seen from the screenshot below, the onboard Intel SATA controller is detected and it is possible to connect a HDD/SSD to these ports and use them for datastore. ESXi does not support onboard SATA RAID since it is a kind of software RAID.

sata_ahci

VT-d verification on the ASRock Z97 Extreme 6

Unfortunately, at the time of this test I did not have a VT-d capable CPU installed in the system. Therefore, I can not verify that VT-d is working with this board. From what I have read, it is possible to get VT-d to work on the Z97 chipset and other persons have managed to get it working on similar boards. Hopefully, I will have the chance to confirm this at a later point.

Conclusion

It is possible to make both onboard NICs on ASRock Z97 Extreme 6 available to ESXi 5.5 U2 by adding drivers for them to the ESXi ISO image. It is also possible to use the onboard SATA controller to connect drives and use them as datastores.

extreme6_nics

drivers

Adding multiple drivers to an ESXi 5.5 u2 ISO

The calm before the storm

My VMware ESXi based home server has been working really well for the last 2 years and I have not felt the need to upgrade it. That is the longest I can handle “don’t fix it if it ain’t broken” šŸ˜‰

Background and motivation

The main reason for this upgrade is to move from a solution with a single HDD connected to the onboard SATA controller, to a battery backed hardware RAID controller (LSI 9261-8i). I guess I have been lucky, but let’s not celebrate too early!

Anyway, this article is about preparing a new ESXi ISO with all the drivers needed for the transition from ESXi 5.1 to ESXi 5.5-u2. Although I have not been reading very actively on this topic lately, I do have picked up a few things to consider for this upgrade:

  • VMware have been removing SATA AHCI mappings for some onboard controllers. This might be an issue since I intend to move the existing VMs from the single drive to the RAID:ed disks.
  • ESXi 5.5 u2 have drivers for the LSI 9261-8i card but there are newer ones, I might as well include the latest release.
  • Speaking of the RAID controller, there is also a “SMIS provider” VIB to be able to manage the RAID controller in the host remotely. To be honest, I need to learn more about this and the first step is to include this functionality.
  • Still no native support for one of the Intel NICs on this board (Intel DQ77MK motherboard with Intel 82579LM and 82574L NICs). Therefore, NIC drivers need to be included.
  • Finally, since this motherboard is more than 2 years old, at least since I last updated the BIOS, I’m going to include a CPU microcode update pack.

Preparing to include drivers in an ESXi 5.5-u2 ISO

Previously, I have been using the graphical ESXi-Customizer tool to add drivers to an ISO, but this time I will attempt to use the PowerShell script version

Here is a recipe for the tool chain:

Here is a summary of the drivers to be included.

The ESXi-Customizer-PS has a really nice feature where it is possible to make it load an entire directory with files to be included. I created an offline_bundles subfolder from where I put the ESXi-Customizer-PS script and copied all files to it.

Executing the ESXi-Customizer-PS script

I’m a complete PowerShell noob and it took me a while to just make it run the script. I had to run the following command to allow script:

Set-ExecutionPolicy Unrestricted

Navigate to the ESXi-Customizer script folder and run the following command:

.\ESXi-Customizer-PS-v2.3.ps1 -pkgDir .\offline_bundles -izip .\update-from-esxi5.5-5.5_update02-2068190.zip -nsc

To make it easier to read, here it is again broken into multiple rows:

.\ESXi-Customizer-PS-v2.3.ps1 
-pkgDir .\offline_bundles
-izip .\update-from-esxi5.5-5.5_update02-2068190.zip 
-nsc

A successful output looks something like this:

Script to build a customized ESXi installation ISO or Offline bundle using the VMware PowerCLI ImageBuilder snapin
(Call with -help for instructions)

Running with PowerShell version 2.0 and VMware vSphere PowerCLI 5.8 Release 1 build 2057893

Adding base Offline bundle .\update-from-esxi5.5-5.5_update02-2068190.zip ... [OK]

Getting ImageProfiles, please wait ... [OK]

Using ImageProfile ESXi-5.5.0-20140902001-standard ...
(dated 08/23/2014 06:46:46, AcceptanceLevel: PartnerSupported,
For more information, see http://kb.vmware.com/kb/2079732.)

Loading Offline bundles and VIB files from .\offline_bundles\ ...
   Loading D:\Linux\ESXi5.5u2VIBs\offline_bundles\cpu-microcode-1.5.0-1-offline_bundle.zip ... [OK]
      Add VIB cpu-microcode 1.5.0-1 [OK, added]
   Loading D:\Linux\ESXi5.5u2VIBs\offline_bundles\igb-5.2.7-1331820-offline_bundle-2157967.zip ... [OK]
      Add VIB net-igb 5.2.7-1OEM.550.0.0.1331820 [OK, replaced 5.0.5.1.1-1vmw.550.1.15.1623387]
   Loading D:\Linux\ESXi5.5u2VIBs\offline_bundles\megaraid_sas-6.605.00.00-offline_bundle-2132901.zip ... [OK]
      Add VIB scsi-megaraid-sas 6.605.00.00-1OEM.550.0.0.1331820 [OK, replaced 5.34-9vmw.550.2.33.2068190]
   Loading D:\Linux\ESXi5.5u2VIBs\offline_bundles\net-e1000e-3.1.0.2-glr-offline_bundle.zip ... [OK]
      Add VIB net-e1000e 3.1.0.2-glr [New AcceptanceLevel: CommunitySupported] [OK, replaced 1.1.2-4vmw.550.1.15.1623387]
   Loading D:\Linux\ESXi5.5u2VIBs\offline_bundles\sata-xahci-1.24-1-offline_bundle.zip ... [OK]
      Add VIB sata-xahci 1.24-1 [OK, added]
   Loading D:\Linux\ESXi5.5u2VIBs\offline_bundles\VMW-ESX-5.5.0-lsiprovider-500.04.V0.53-0003-offline_bundle-2152533.zip ... [OK]
      Add VIB lsiprovider 500.04.V0.53-0003 [OK, added]

Exporting the ImageProfile to 'ESXi-5.5.0-20140902001-standard-customized.iso'. Please be patient
 ...

All done.

Now you have an installation image ready to be tested!

Align GPT partitions on 4K sector disks

I’m upgrading the storage in a offsite backup server to two new disks. The new disks are of 3TB each which pose some challenges when it comes to partitioning. Here is a quick background to this issue.

Why is it an issue to partition disks larger than 2TB?

Historically, data stored on the actual disks have been stored in 512 byte chunks, called a sector. 32 bit addressing of sectors creates the following limit:

512 bytes * 2^32 = 2199023255552 bytes = 2T bytes

And there you have it. Newer disks have transitioned to “4K”/4096 bytes physical sectors which extends this limit to 16TB. But…

Why is partition alignment crucial to storage performance?

To complicated things further, disks often expose 512 bytes logical sectors to the operating system for legacy support. This might cause tools to believe it is okay to begin and end a partition on any 512 byte sector border, which might not be a 4K byte sector border that is stored on the disk.

Hardware.info has a good article illustrating this.
Wikipedia on 4K / Advanced Format

How do you align partitions in Ubuntu with GNU Parted?

GNU Parted is a tool that supports GUID Partition Table, GPT, setup under Linux. Parted have some parameters to aid in the alignment of partition starts and ends. Let’s launch parted with:

$ sudo parted --align optimal /dev/sdX

Where sdX is the drive we intend to view and/or modify. The –align optimal is the aid in the alignment. In parted we can view the current partition table with the command print:

(parted) print
Model: ATA ST3000VX000-1CU1 (scsi)
Disk /dev/sdX: 3001GB
Sector size (logical/physical): 512B/4096B

As we can see, the drive has 4K physical sectors but presents 512 logical sectors. A tricky part I struggled with for hours was to calculate the partition sizes with the unit set to sectors. In my opinion, parted could be more clear on what sector size it presents to the user. To figure this out I issued the following:

(parted) unit B
(parted) print
Model: ATA ST3000VX000-1CU1 (scsi)
Disk /dev/sdX: 3000592982016B
...
(parted) unit s
(parted) print
Model: ATA ST3000VX000-1CU1 (scsi)
Disk /dev/sdX: 5860533168s

Making the calculation, bytes per sectors:

3000592982016B / 5860533168s = 512 byte/sector

So, even though this is a 4K drive, parted is using 512 byte sectors for viewing partition starts, ends and sizes.

Setting up partitions with parted

First, let’s setup a gpt partition table with the following command:

(parted) mklabel gpt

This was the partition layout I wanted to achieve:

Partition Size Usage
sdX1 8GB swap
sdX2 250GB /
sdX3 1200GB raid
sdx4 1542GB raid

Initially, I tried calculating the partition sizes using the sector unit to make sure that each partition border aligned with the physical sectors. Often, parted complained about the alignment with:

Warning: The resulting partition is not properly aligned for best performance.

What helped was to use the unit MB for the starts and ends. Here is the final parted commands:

mkpart primary 1 0% 8000MB
mkpart primary 2 8000MB 258000MB
mkpart primary 3 258000MB 1458000MB
mkpart primary 4 1458000MB 100%

Notes: Using 0% default to the first 1MB border that is correctly aligned. The same goes for 100% which makes sure the last partition aligns with the end of the disk. Here is the resulting partition layout:

(parted) unit s
(parted) print
Model: ATA ST3000VX000-1CU1 (scsi)
Disk /dev/sdX: 5860533168s
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 2048s 15624191s 15622144s 1 
2 15624192s 503906303s 488282112s 2
3 503906304s 2847655935s 2343749632s 3 raid
4 2847655936s 5860532223s 3012876288s 4 raid

(parted) unit compact
(parted) print
Model: ATA ST3000VX000-1CU1 (scsi)
Disk /dev/sdX: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number Start End Size File system Name Flags
1 1049kB 8000MB 7999MB 1
2 8000MB 258GB 250GB 2
3 258GB 1458GB 1200GB 3 raid
4 1458GB 3001GB 1543GB 4 raid

To verify that the partitions are aligned, the following command can be executed, with P being the partition number:

(parted) align-check optimal P
P aligned

This became a long post. In the future I will try to cover handling alignment between the filesystem layer and the partitions.

Let me know how it goes for you!

How to add Intel NIC drivers to an ESXi 5.5 ISO

Modifying ESXi ISO images to include network drivers

Using new hardware with not-yet supported or unsupported drivers is often required when using consumer grade and/or desktop components. Most of the motherboards I have presented here on this blog (ASRock Z87 Extreme6, ASRock Z77 Pro4-M, Intel DQ77KB, Intel DQ77MK) have had unsupported network controllers.

There are ways to add support for a NIC after ESXi has been installed. However, to install ESXi from the beginning requires at least one supported network controller. Adding or updating a driver directly in the ESXi ISO solves this issue.

Tools required

To be able to perform the driver inclusion the following tools are needed:

When it comes to what driver to include I have to be honest to report that I have not been able to fully figure it out. Intel seems to have two lines of drivers for Linux, igb and e1000. from what I understand, the igb drivers are mostly for server NICs and the e1000 drivers for desktop NICs. Some desktop controllers use the igb driver though. Here is an excerpt from Intel’s website:

e1000e.x.x.x.tar.gz is designed to work with the IntelĀ® 82563/6/7 Gigabit Ethernet PHY, 82571/2/3/4/7/8/9,Ā 82583 Gigabit Ethernet Controller, and I217/I218Ā controllers under Linux*.Ā  The latest version and earlier versions of this driver are available from SourceForge.

If your adapter/connection is notĀ 82563, 82566, 82567, 82571, 82572, 82573, 82574, 82577,Ā 82578, 82579,Ā 82583 -based, you should use one of the following drivers:

– igb-x.x.x.tar.gz driver supports all IntelĀ® 82575/6-, 82580-, I350-, or I210/1-based Gigabit Network Adapters/Connections.
– e1000-x.x.x.tar.gz driver supports all IntelĀ® 8254x-based PCI and PCI-X Gigabit Network Adapters/Connections.

Furthermore, the drivers for ESXi needs to be recompiled from the above versions. TheĀ ASRock Z87 Extreme6 board, that will be used in this example, have the following two controllers:

  • IntelĀ® Ethernet Connection I217-V, e1000e driver.
  • IntelĀ® Ethernet Controller I211 Series, igb driver.

I will only demonstrate how to get the e1000e driver to work, simply because I have not yet found a newly compiled version of the igb driver. The newest e1000e driver I can find on the Internet is:

http://shell.peach.ne.jp/~aoyama/wordpress/download/net-e1000e-2.3.2.x86_64.vib

Credit goes to the following site:http://shell.peach.ne.jp/aoyama/archives/2907

Using the ESXi-Customizer

Here is how I set up the ESXi-Customizer

  1. Start the ESXi-Customizer
  2. Load the ESXi 5.5 ISO
  3. Load the net-e1000e-2.3.2.x86_64.vib driver package
  4. Uncheck UEFI bootable. (I have read both suggestions both for and against. Unchecked has worked well for me)
  5. Hit Run! to create the new ISO
  6. Burn the ISO to a disc or make a bootable USB out of it.

This is what the ESXi-Customizer looks like for me.

ESXi-Customizer

I hope everything works out. Either way, let me know in the comments below!

Storage performance: Intel Z87 vs. ASMedia ASM1062 vs. LSI 9211-8i

During my VT-d verification on the ASRock Z86 Extreme6 I took the opportunity to compare the performance of three different storage controllers, namely:

  • Intel Z87 (onboard)
  • ASMedia ASM1062 (onboard)
  • LSI 9211-8i (PCI-Express 8x add in card)

Below is a summary of the test setup and the results of the tests.

Test System

Native performance

Comparison of the three controllers are done with the simple hard disk benchmark tool in Ubuntu 13.10.

SSD Performance

Average read [MB/s] Average write [MB/s] Average access time [ms]
Intel Z87 516.6 527.4 0.03
ASMedia ASM1062 402.2 398.4 0.04
LSI 9211-8i 546.9 521.8 0.04

Intel-SSD ASMedia-SSD LSI-SSD

HDD Performance

Average read [MB/s] Average write [MB/s] Average access time [ms]
Intel Z87 140.3 136.1 12.4
ASMedia ASM1062 140.3 136.0 12.5
LSI 9211-8i 140.3 136.6 12.4

Intel-HDD ASMedia-HDD LSI-HDD

Passthrough Performance

Passthrough performance is measured with ESXi 5.5 installed on a USB memory and the LSI card passed through to a VM. The VM is running the same version as in the above benchmarks, ubuntu 13.10. The performance is only run with the LSI card. I really tried getting passthrough working with the ASMedia controller as this would open up to some interesting storage opportunities with this board. However, Ubuntu recognized the controller but did not find any disk attached to it. Also, now that I think about it, I have no idea why I did not think about trying to pass through the Z87 controller. Anyway, here is the comparison, SSD and HDD combined.

Average read [MB/s] Average write [MB/s] Average access time [ms]
SSD – Native 546.9 521.8 0.04
SSD – Passthrough 519.3 520.2 0.06
HDD – Native 140.3 136.6 12.4
HDD – Passthrough 140.3 136.4 12.4

LSI-SSD LSI-SSD-passt

LSI-HDD LSI-HDD-passt

Final thoughts

The ASMedia controller is not capable of handling the performance of modern SSDs. For mechanical drives there is practically no difference between the three different controllers.

I had an idea of using the Intel controller for the ESXi datastore and pass through the ASMedia controller to a VM. Then it would be possible to setup software RAID for the drives connected to the ASMedia controller. This is a solution working very well for me today with the LSI card, but it would have been nice to have an all-in-one solution.

There are some performance impacts on reads when passing through the LSI card to a VM. I have not investigated this further but it might very well be benchmark technical reasons behind it.

VT-d Verification on ASRock Z87 Extreme6 with ESXi 5.5

This is one of the best enthusiast ESXi virtualization motherboard I have come across so far: ASRock Z87 Extreme6 (maybe second to ASRock Z77 Extreme11).
Some of the highlights:

  • Onboard USB header
  • VT-d passthrough
  • Debug LED
  • Dual Intel NICs

I’m very fond of installing ESXi on a USB stick to separate it from the rest of the storage. Previously I have removed the metal bracket from “onboard-USB-header-to-rear-I/O-bracket” (whatever do they call these things?) that used to come along with motherboards a couple of years back, to have an internal USB header.

Secondly, ASRock is one of the few manufacturers that supports VT-d on as many of their motherboards as possible. Even on boards with a chipset not officially supporting VT-d, like the Z77. Here is my previous test of ASRock Z77 Pro4-M. See below for some VT-d tests.

Finally, the debug code LED. This brings some lovely overclocking memories back to life. One of the main reasons for choosing Abit over Asus back in the P3 and P4 days was this very tiny feature. The POST codes proved to be extremely valuable when pushing the very last bit of performance out of a system. Anyway, I’m way off track. Let’s look at some pictures and some of my other findings.

Extreme6-overview USB-header-post-code-led intel-i211 sata-ports

Specifications

  • Intel Z87 chipset
  • Intel Haswell CPU support, socket 1150
  • Four DIMMS, supporting 32GB RAM
  • Dual Intel Gigabit LAN (I217V + I211AT)
  • USB socket directly on the board
  • POST debug code display
  • 8x USB3

The dual Intel network controllers is a real positive thing for us ESXi persons and I was really exited about this. But, neither of these controllers are supported by ESXi 5.5 out of the box. The I217 NIC can be made to work if a newer driver is supplied. However, the I211 NIC is a scaled down desktop version, as far as I have read and I have not yet found a way to make work.

Apart from the Z87 based SATA controller, there are also four additional ports connected to two controllers made by ASMedia. Two ports to each controller, whereas one port double as a ESATA port. The name of the controllers are “ASM1062”.

In total, there are eight USB3 ports where four are connected to the Z87 chipset and four connected to a controller also made by ASMedia. The four rear I/O ones are connected to the ASMedia controller and the four internal ones to the Z87 chipset. I am not entirely sure if I think this was the best move by ASRock. I can argue either way.

VT-d testing

While gathering fact about the board on the Internet I came across some comments fearing the removal of the VT-d setting for some boards and with some BIOS versions. This board, with the shipping BIOS (P2.10), has the VT-d option. According to some BIOS release notes, even though the VT-d option is removed it is supposed to be enabled by default if all criterias are met.

ESXi 5.5 is capable of utilizing the VT-d functionality with this board. More specifically DirectPath I/O as VMware calls it. Here are some of my findings:

  • Both NICs can be passed on to a VM, even though none of them are supported out of the box by ESXi 5.5, and only one can be made operational with additional drivers.
  • Both ASMedia SATA controllers show up and can be passed on to a VM. However, when I tried this with a Ubuntu 13.10 VM I could not get any drive connected to show up. The controller is detected in the VM but that is all. The controller is detected by Ubuntu 13.10 when run natively.
  • The ASMedia USB3 controller does not show up in the passthrough view. There are two Z87 USB controllers showing up but it is not clear to me if either or both of them are USB3.
  • An LSI 9211-8i card, actually a flashed M1015, inserted into the top most PCI-Express 16x slot did not show up as available for passthrough. The card works in this port when running Ubuntu 13.10 natively. Inserting the card in the second slot made it available for passthrough and it worked well with Ubuntu 13.10.

Here is a screenshot of the default available passthrough devices:

ESXi-ASRockZ87Extreme6

Summary

All in all, a board with great potential. If drivers arrive for the second Intel NIC and some of the passthrough issues are sorted out it will be a killer motherboard. Nothing critical by any means, but the flexibility is reduced if you need to use one PCI(-Express) slot for another NIC, all SATA ports or an separate RAID controller.

I will hopefully be able to present upcoming articles on how to customize an ESXi image for this board and present some storage benchmarks.

*Update 2014-01-04*
Here is how to include a driver the one of the Intel network controllers on this board:
Include Intel network drivers in an ESXi ISO
Also, here is some storage benchmarks related to this board:
Storage performance of ASRock Z87 Extreme6