With the release of vSphere Update 1, we can now take advantage of Software iSCSI MPIO using the new DvSwitch functionality in vSphere.  Before now, we were limited to only being able to do MPIO on a standard vSwitch.  Using a DvSwitch is a little more involved, here are the steps to set-up the DvSwitch, MPIO and Jumbo Frames: (Please let me know if anything is unclear or incorrect!).

1. Create a new DvSwitch.  We called ours “Storage_dvSwitch”.
2. After creating your new DvSwitch, Edit the settings and change the Max MTU to 9000 to support Jumbo frames.
   
3. Now create 2 (or however many you want) new Port Groups.  Nothing special here, we named ours iSCSI1 and iSCSI2.  VLAN configuration may vary depending on network configurations.
4. This part is important – we need to set-up the pNIC fail-over settings properly.  You should have ONE active dvUplink per iSCSI port-group, NO standby NICs and the remaining NICs should be Unused.  Each port-group should get a different dvUplink. This can be changed in the port-group settings page.
5. This concludes the vCenter specific configuration.  The following steps will apply to EACH host that you plan to use with Jumbo frames.
6. If you haven’t already, you will need to add the Host’s physical NIC’s to the new DvSwitch we created. You can do this from the ‘Add Host’ link on the DvSwitch summary page.
7. Now we need to create 1 VMkernel port for each Port-Group we created. This is done in the HOST configuration tab, in the Networking section. Select the ‘Distributed Virtual Switch’ view and click the ‘Manage Virtual Adapters…’ link.
8. Click ‘Add’ and add one VMkernel port to each iSCSI port-group you created on the storage DvSwitch.  There may be some benefits to over-subscription here (many VMkernel ports on 1 port-group), but that will vary by environment.  When you are done, your Port-Groups should look something like this:
9. Now we need to find the Port number assigned to each vmk port.  You can either click the ‘i’ icons next to each vmk port or by running “esxcfg-vswitch -l” from the CLI.  In this example, vmk1 has a port ID of 1, and vmk2 has a port ID of 129.
10. Now, from the CLI, delete the VMkernel ports you just created in the GUI (Awesome, isn’t it?) by running “esxcfg-vmknic -d -s ‘<your dv switch name>’ -v <dv port number>”.  You will need to run this for each VMkernel port you created.  So in our example, it would be:
    • esxcfg-vmknic -d -s ‘Storage_dvSwitch’ -v 1
    • esxcfg-vmknic -d -s ‘Storage_dvSwitch’ -v 129
11. Now re-add the VMkernel ports (using the port numbers you just deleted) with jumbo frames enabled by running: “esxcfg-vmknic -a -s ‘<your dv switch name>’ -i <ip addr> -n <subnet> -m 9000 -v <dv port number>”.  So in our example:
    • esxcfg-vmknic -a -s ‘Storage_dvSwitch’ -i 10.255.38.151 -n 255.255.255.0 -m 9000 -v 1
    • esxcfg-vmknic -a -s ‘Storage_dvSwitch’ -i 10.255.38.201 -n 255.255.255.0 -m 9000 -v 129
12. Enable the software iSCSI initiator if you haven’t already and find the vmhba number assigned to it.  In our example, it is vmhba35:
13. Now bind the new VMkernel ports to the iSCSI HBA: “esxcli swiscsi nic add -n vmkx -d vmhba35″  Adjust the vmkernel name (vmk0, vmk1, etc) and the iSCSI initiator HBA number (vmhba33, vmhba35, etc) to fit your environment.  Do this for each VMkernel port you will be using for MPIO.  For this example, we run:
    • esxcli swiscsi nic add -n vmk1 -d vmhba35
    • esxcli swiscsi nic add -n vmk2 -d vmhba35
14. You can verify that your vmk ports were successfully added to the pNIC’s by running “esxcli swiscsi nic list -d vmhbaxx” (in our case, esxcli swiscsi nic list -d vmhba35).  Each vmk port should be listed and should be assigned ONE pNIC.