Setting up PCNS 3.0.0 for ESXi 4.1 update 1

Note: An updated version of these instructions for ESXi 5.0 and PCNS 3.0.1 is available in a new post here: http://tsbraindump.blogspot.com/2012/08/installing-and-configuring-apc.html

I've done this setup 3 times now, and every time I end up having to resort to trolling the internet for instructions, so I figured it's about time I write down my own. Note these are from personal experience, contacting support, etc etc.

1) Download stuff. There's a bunch of it!
a) Download PCNS 3.0.0 from the APC website. It's free now, it just requires an apc.com login.
b) While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
c) Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 4.1 released in 2010.
d) Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap.

2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.

3) Create one vMA virtual machine FOR EACH HOST by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need a 5GB volume (I usually make the volume 10GB for snapshot space), 1 processor, and a gig or so of ram. Note that I said "for each host". I spoke to APC support, and they admitted (finally) that the vifp 'fastpass' stuff does not work. My personal experiences confirm this. For some reason it only shuts down the first host on it's list!

4) Open the console of the vMA and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local.

5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.

6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.

7) Back in the vMA console, browse to the tmp directory and run "gunzip pcns300ESXi.tar.gz", then run "tar -xf pcns300ESXi.tar" to extract the file. Browse into the ESXi folder that is created, and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts. Continue with all the defaults. When you get to the part about entering an IP, just press "q" to skip it.

8) Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful. Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver". You will be asked to enter the password for your host, so vMA will have it on file.

9) Now, we need to configure PCNS. Open a web browser and go to https://:6547 and follow the configuration wizard. The only step worth menitoning is that you should NOT check the box for "Turn off the UPS after shutdown finishes." If you check it, there's a good chance the UPS will turn off while your hosts are still shutting down. The caveat to leaving this unchecked is that your ESXi servers most likely will not turn back on if power is restored before the UPS battery loses all of it's charge... you'll have to use iLO or DRAC to get in and turn your hosts on instead.

10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)

11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.

12) Finally, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down!

13 Repeat these steps for each host in your cluster. Yes, you need one VMA per host! Don't believe me? Fine. Try it yourself. Or call APC. You'll see.

That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!

Setting LUN Policy and PSP in ESXi 4.1

For customers with many LUNs, it can be a serious pain to create, modify, and maintain all the multipathing settings in VMware ESXi... I have a client with four active iSCSI adapters on each host, and four active iSCSI frontend ports on their Compellent SAN, which means a delightful 16 paths per LUN. Couple that with 2-3 LUNs per guest (raw device mappings), and 15 or so guests... that's a lot of clicking.

This is where VMware's vCLI is a godsend... you can script all of those monotonous clicks. For example, here's a pair of scripts I used to convert all of their Path Selection Policies and default PSP's to Round Robin.

1) Download and install vCLI
2) type "connect-viserver hostname" (where hostname is the name of the ESX host you want to make the changes on)
3) type "get-scsilun -vmhost (get-vmhost hostname) -luntype disk | set-scsilun -multipathpolicy "roundrobin""
4) Open an SSH prompt to the ESX host in question (you may need to enable SSH access in your hosts Configuration > Security Profile in vSphere).
5) In the SSH prompt, type "esxcli nmp satp setdefaultpsp --psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA

NOTE: The --satp type may vary depending on what SAN you use... you can check what your SATP currently is by clicking "manage paths" of any one of your LUNs in vSphere... under the path selection type it displays it as "storage array type".

That's it... now all the LUNs on the host should be round robin, and all future volumes should default to round robin as well!

Update:

For ESXi 5, the SSH command syntax has changed slightly. You'll want to type the following from the console of your ESXi 5 server:

esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA
Also, the part about vCLI is no longer necessary! Once you switch the default path policy with he above command, you can just reboot the host and it should apply the policy to all existing volumes. Thanks, VMware!