Setting LUN Policy and PSP in ESXi 4.1

For customers with many LUNs, it can be a serious pain to create, modify, and maintain all the multipathing settings in VMware ESXi... I have a client with four active iSCSI adapters on each host, and four active iSCSI frontend ports on their Compellent SAN, which means a delightful 16 paths per LUN. Couple that with 2-3 LUNs per guest (raw device mappings), and 15 or so guests... that's a lot of clicking.

This is where VMware's vCLI is a godsend... you can script all of those monotonous clicks. For example, here's a pair of scripts I used to convert all of their Path Selection Policies and default PSP's to Round Robin.

1) Download and install vCLI
2) type "connect-viserver hostname" (where hostname is the name of the ESX host you want to make the changes on)
3) type "get-scsilun -vmhost (get-vmhost hostname) -luntype disk | set-scsilun -multipathpolicy "roundrobin""
4) Open an SSH prompt to the ESX host in question (you may need to enable SSH access in your hosts Configuration > Security Profile in vSphere).
5) In the SSH prompt, type "esxcli nmp satp setdefaultpsp --psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA

NOTE: The --satp type may vary depending on what SAN you use... you can check what your SATP currently is by clicking "manage paths" of any one of your LUNs in vSphere... under the path selection type it displays it as "storage array type".

That's it... now all the LUNs on the host should be round robin, and all future volumes should default to round robin as well!

Update:

For ESXi 5, the SSH command syntax has changed slightly. You'll want to type the following from the console of your ESXi 5 server:

esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA
Also, the part about vCLI is no longer necessary! Once you switch the default path policy with he above command, you can just reboot the host and it should apply the policy to all existing volumes. Thanks, VMware!

Exchange 2010 Edge Role + 2 NIC cards + DNS issues

I am trying out a few technologies that are new to me for one of our clients, namely Exchange 2010 with the Edge role on a DMZ server, as well as installing Threat Management Gateway on the Edge server... good times! I am sure there will be several posts on TMG coming soon... I can feel it now...

Anyway, I ran into a DNS problem with my fresh Exchange 2010 install that took a little bit of doing to figure out what was going on. The basic layout of the environment is that they have a single Exchange 2010 server on their LAN with the CAS/HUB/MBX roles installed, and a single EDGE role on a DMZ server that is on the domain. This Edge server has two NIC cards... one for the inside network so it can talk to the domain and do windows authentication, and one for the outside network so that it can receive mail and publish the Exchange web interface.

Shortly after installation, mail started queuing up on my edge server. Taking immediate advantage of the tools in front of me, I used the Queue viewer and noticed a few DNS warnings that certain domains could not be resolved. Thinking I was smart, I proceeded to put in hosts entries on the Edge server... nope, didn't solve it. I then thought that perhaps it was using the external interface's DNS server to try and resolve internal server names... so I took the DNS servers away from the external NIC card... still no luck.

Then, I stumbled upon a setting that suddenly cleared it all up: on the edge server, if you open up the Exchange management console, right click on the Edge server in the middle pane, and go to properties, there is a tab for 'External DNS lookups.' I set this to the Inside NIC card, and bam, mail started flowing.

Dell Server P2V stuck at loading screen

Almost every P2V I have done to date has gone flawlessly. I frankly expected more trouble, but have been very pleased with VMware's product thus far. Almost every virtualization that experienced trouble was a result of third party software, or in this case, drivers for third party hardware.

The specific server in question was a Dell PowerVault 700 series NAS... it was running Windows Server 2003 Appliance edition, an OEM only version of windows 2003. Not really supported for virtualization, but the intention was to upgrade it to Datacenter Edition anyway once all was said and done... it was just a matter of getting there. After the virtualization process finished, the server powered on, and would not get past that animated windows logo loading screen with the grey dots moving across the bottom. I let it sit for up to 30 minutes before I was fairly sure there was no hope. The processor spiked at 100% the entire time, so I knew it was some kind of driver or service misbehaving. The system booted into safe mode just fine, but wouldn't boot into normal mode, even in diagnostic startup mode.

I worked with Microsoft, they didn't have a clue either. Time for process of elimination! I opened up MSINFO32 and made an export of all the Software Environment > System Drivers section for safe keeping, then went to work disabling all of them, section by section, starting with the ones marked as 'Stopped' with 'Start Mode' Set to Boot. I knew it had to be a 'stopped' one as windows was starting fine in safe mode, but not when everything started up normally. You can change them by opening up regedit and going to HKLM\System\Services. Expand each service name, and under it is a key called 'Startup' with a value of 0-4. 0 is 'Boot', 1 is 'System', 2 is 'Automatic', 3 is 'Manual', and 4 is 'Disabled'. Make sure to write down the original value of each 'startup' item as you set it to 4, because you will need to revert the changes.


Either way, disabling all the 'Boot' startup items didn't fix it, so I moved on to disabling all of the ones with 'start mode' set to 'System'. With all Boot and System items set to disabled, bingo! It started up. Now I put them each back to their original startup mode, one by one, rebooting into normal mode each time to see if the server would start. It finally narrowed down to one service, called "Msdisp." The description? "DELL LED DRIVER". Go figure! It was a driver for the hardware chassis to control the LED lights on the front of it, and it apparently blew up when it couldn't find whatever LED controller it was looking for. I left the service disabled, returned all others to their original values, and I was off to the races!

Hopefully this process of elimination method helps someone else out there!

Performing a P2V on a Domain Controller

Yes, I know, it's not recommended. Heck, it's not even supported. But what if you want to throw caution to the wind, and do it anyway? Here's the method I used that worked successfully. A few notes though:

- This only works in environments with two or more domain controllers
- There may still be problems with clients authenticating should you be ballsy enough to do this during business hours, as I was. Rebooting usually fixes them.

1) You'll first of all want to transfer all of your FSMO roles to another DC in the environment. You can see which servers are hosting the FSMO roles in your environment by opening up a command prompt on your domain controller and typing "netdom query fsmo". You'll get back a neat list of which services are where. Transfer them to other domain controllers (I'll leave the specifics out of this article, google has the answers.)

2) Open a command prompt on the DC you wish to P2V and type dcpromo, then hit enter. Proceed through the process to remove active directory from this server. You may get an error the first time... it's okay... just run it again. It's complaining that services didn't stop or start in a timely fashion, which is normal for a DC. Reboot once you are done.

3) Uninstall DNS and WINS using Add/Remove programs. Also, if you are running DHCP, you may want to start up a split scope on another server, as you should disable the dhcp scope on this server for the time being.

4) Change the DNS on your server you wish to P2V so that it's primary DNS points to a VALID server other than itself... otherwise, your P2V will fail immediately. This is because the server being virtualized needs to be able to resolve the DNS names of the destination ESX server and VirtualCenter server. Change the DNS for your VirtualCenter server and ESX server so that they are all pointing to the same DNS server, just to be safe.

5) Run the P2V, it should go off without a hitch now that DNS is right.

6) Now that the server is back up and virtual, run dcpromo again to make it a domain controller once more. This should reinstall DNS for you, if you uninstalled it earlier. Fix up DHCP and WINS, and you are back in business. Don't forget to transfer back your FSMO roles, if you want them to be on your virtual server!

7) Readjust your DNS settings on the re-promoted domain controller to point to itself, and fix your VirtualCenter/ESX server DNS settings too, if you had to change them.

Configuring Dell PowerConnect 6224 Switches for iSCSI traffic to a Compellent

This is the second time I have set up a Compellent, and this time I figured I would go all the way down the rabbit's hole and delve into switch optimization for iSCSI traffic. My last project was with Cisco switches, this time, it's with Dell. A pair of PowerConnect 6224 switches to be precise... main reason being, Dell purchased Compellent not long ago, so like-products, yadda yadda. Alas, I digress.

In previous talks with Compellent, I was give the '10 commandments' of switch configuration from at least two different representatives upon having iSCSI issues. These commandments roughly consisted of:

• Gigabit Full Duplex connectivity between Storage Center and all local Servers
• Auto-Negotiate for all switches that will correctly negotiate at Gigabit Full Duplex
• Gigabit Full Duplex hard set for all iSCSI ports, for both Storage Center and Servers for switches that do not correctly negotiate
• Bi-Directional Flow Control enabled for all Switch Ports that servers or controllers are using for iSCSI traffic.
• Bi-Directional Flow Control enabled for all Server Ports used for iSCSI (Storage Center and QLogic HBA's automatically enable it).
• Bi-Directional Flow Control enabled for all ports that handle iSCSI traffic. This includes all devices in between two sites that are used for replication.
• Separate VLAN for iSCSI.
• Two separate networks or VLANs for multipathed iSCSI.
• Two separate IP subnets for the seperate networks or VLANs in multipathed iSCSI.
• Unicast storm control disabled on every switch that handles iSCSI traffic.
• Multicast disabled at the switch level for any iSCSI VLANs.
o Multicast storm control enabled (if available) when multicast can not disabled.
• Broadcast disabled at the switch level for any iSCSI VLANs.
o Broadcast storm control enabled (if available) when broadcast can not disabled.
• Routing disabled between regular network and iSCSI VLANs.
• Do not use Spanning Tree (STP or RSTP) on ports which connect directly to end nodes (the server or Compellent controllers iSCSI ports.) If you must use it, enable the Cisco PortFast option on these ports so that they are configured as edge ports.
• Ensure that any switches used for iSCSI are of a non-blocking design.
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments.
• Verify optimal MTU for replications. Default is 1500 but sometimes WAN circuits or VPNs can create additional overhead which can cause packet fragmentation. This fragmentation can suboptimal performance. The MTU is adjustable via the GUI in 5.x Storage Center Firmware.
For Jumbo Frame Support
• Some switches have limited buffer sizes and can only support Flow Control or Jumbo Frames, but not both at the same time. Compellent strongly recommends choosing Flow Control.
• All devices connected through iSCSI need to support 9k jumbo frames.
• All devices used to connect iSCSI devices need to support it.
o This means every switch, router, WAN Accelerator and any other network device that will handle iSCSI traffic needs to support 9k Jumbo Frames.
• If the customer is not 100% positive that every device in their iSCSI network supports 9k Jumbo Frames, then they should NOT turn on Jumbo Frames.
• QLogic 4010 series cards (Early Compellent iSCSI Cards) do not support Jumbo Frames.
o In the Storage Center GUI default screen, expand the tree in the following order Controllers->SN#(for the controller)->IO Cards->iSCSI->Highlight the port and the general tab should list the model number in the description.
• Because devices on both sides (server and SAN) need Jumbo Frames enabled, the change to enable to disable Jumbo Frames is recommended during a maintenance window. If servers have it enabled first, the Storage Center will not understand their packets. If Storage Center enables it first, servers will not understand its packets.

Okay, so it was more than 10. Anyway, all that roughly distills down to the following needs:

1) Set spanning-tree mode rstp and enable portfast on all ports, or disable spanning-tree all together.
2) Enable jumbo frame support on all iSCSI ports
3) Disable unicast storm-control on all iSCSI ports
4) Enable multicast storm-control on all iSCSI ports
5) Enable broadcast storm-control on all iSCSI ports
6) Enable flow control

For each switch, the commands are different... but for Dell PowerConnect 6224 in particular, these needs translated into the following commands (in order, considering I used ports 1-12 for iSCSI)

enable
configure
spanning-tree mode rstp
interface 1/g1-1/g12
spanning-tree portfast
mtu 9216
no storm-control unicast
storm-control multicast
storm-control broadcast
exit
flowcontrol

That was it, switch optimized, on we go!

How to fix a blackberry sync issue (the secret CNFG menu)

So the title is a bit cryptic, but I ran into an issue where a certain blackberry decided to no longer sync it's contacts with our newly migrated BES server. No idea why, tried rebooting the phone, toggling wireless sync of contacts, etc etc... no luck. The user had a bunch of stuff on the phone, so I didn't really want to wipe it or reactivate it particularly. Found this little tidbit of info on Expert's Exchange at:

http://www.experts-exchange.com/Hardware/Handhelds_-_PDAs/Blackberry/Q_23024284.html

All you need to do is go to Options > Advanced Options > Enterprise Activation and in the email field press and hold the ALT key and type CNFG. Once you enter this a hidden menu will appear and you need to change "Wireless Sync" to No, now exit this menu and wait 30 seconds and repeat the process but turn sync back to Yes. Once you've changed this setting you will see a slow sync will automatically start and it will repair all the wireless sync settings. In the rare case that this fails you just need to wipe the device and reactivate.

Moving ESXi guests without vCenter

vCenter is a very cool application, enabling you to do many things that would otherwise take hours to figure out and execute, especially in a multi-host environment (which is pretty typical for VMware). One such feature is the ability to move guest VM's between hosts. Using vCenter, it's as simple as right clicking the guest and picking 'Migrate', then picking the new destination and waiting. But what if you don't HAVE vCenter, or in my case, the server you want to move IS your vCenter server. Not so easy then!

Veeam to the rescue! If you don't know what Veeam is, you haven't been working with VMware very long. It's a great company that produces lots of useful software for managing VMware environments... and some of their best tools are free! In this case, their free product FastSCP is what we are after. Register for it and install it on any other server than the one you want to move. Setup is simple, and adding hosts is even simpler. Once you add a host, you can expand it's storage just like the storage browser in vCenter or the vSphere client, and moving servers is just a simple copy/paste operation. Well, okay, there are a few more steps:

1) If the server you are moving is part of a vCenter environment, it is highly recommended that you unregister it first. Even if it's not, you should unregister it anyway. Either from vCenter, or using vSphere to connect directly to the host, shut down the guest machine then right click and select 'Remove from Inventory'. Don't click delete! It will warn you about some stuff, just click yes and proceed. You'll notice the VM is no longer listed under your host. It's now just a bunch of files in a folder on the datastore.

2) Start up Veeam, and add the source and destination hosts to the manager if you haven't already. Expand the source host, right click on the folder for the virtual machine you want to move, and click copy. Then, expand the destination host, and right click in the right hand side's empty area, and click paste.

NOTE: If you installed Veeam on an x64 server, you may get errors when trying to copy and paste folders, or create new folders. It's a bug, and Veeam disclaimers that x64 installations are purely experimental at this point. There are two workarounds... either create the folder first using vSphere by browsing to the datastore on the host and making a new folder, then copying only files from one folder to another, or follow this method to fix up your server so that you won't get that error:

http://www.virtualvcp.com/content/view/26/1/ details below

1. Download the Microsoft .NET Framework 2.0 SDK (THE 64-BIT VERSION!) from the Microsoft Website. This download is about 300MB if I remember correctly.
2. Install the SDK on the 64-bit machine that you would like to run Veeam FastSCP on.
3. Now, open a command prompt (Start -> Run -> Type "cmd" -> OK)
4. Change directory to: C:\Program Files\Microsoft.NET\SDK\v2.0 64bit\Bin
5. Now Run: corflags "C:\Program Files (x86)\Veeam\Veeam Backup and FastSCP\VeeamShell.exe" /32BIT+
6. Now when you try and run Veeam FastSCP again, it should work fine.


3) Anyway, now that the files are moved, all you need to do is register the virtual machine on the new host. Open up vSphere, connect directly to the destination host, then browse the datastore. Find the .vmx file inside of the folder you copied and right click on it, then select 'Import' to add it to your inventory. You may get asked something to the effect of "Where did this virtual machine come from?"... in our case, you will want to pick "I moved it".

That's it, start up your server, it should work fine!



Just wait for the copy to finish, then delete the source... migration complete!