Installing and Configuring APC PowerChute Network Shutdown 3.0.1 for ESXi 5.0

I wrote this article original for ESXi 4.1 and PCNS 3.0.0, but enough has changed that I figured it warranted a new article... enjoy!

Shutting down windows servers with the APC network shutdown software is a no-brainer... but what about virtual machines? Sure, you could install the network shutdown software on every virtual machine, but that would be wasteful. Fortunately, there's a centralized way that you can shut down your virtual guests AND hosts... and the best part? It's (mostly) free! All you need is an APC battery backup unit, an APC network shutdown card, and a bunch of software. Here's what you need to do:

1) Download stuff. There's a bunch of it!

1. Download PCNS 3.0.1 from the APC website. It's free now, and you do not even need to register for it (finally, somebody gets it!). You can find it here: http://www.apc.com/tools/download/software_comp.cfm?sw_sku=SFPCNS301&id=127&swfam=127
2. While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
3. Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 5.0.0.2-724898.
4. Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap. Note that fastscp is now bundled into the full backup and recovery tool... so grab that.

2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.

3) Create one vMA virtual machine by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need to pick a datastore for its 5GB volume. I usually make the volume 10GB for snapshot space. There are options to give it an IP, but i found that it didn't end up mattering due to the following error when trying to power on the newly created vMA:

• The only workaround I found for this error was to edit the newly created vMA, go to the Options tab, and under vApp Options select "Disabled". You'll get a nag screen indicating it is removing properties, but the only options that I could find set were items about DHCP, which we aren't using...



4) Open the console of the vMA, power it on, and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local. Once you are completed, you should be dropped to a blue and grey login screen.

5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.

6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.

7) Back in the vMA console, choose the option to "login", enter the vi-admin credentials, and do the following:

1. Browse to the tmp directory (for linux noobs, type cd /tmp) and run "gunzip pcns301ESXi.tar.gz" to unzip the file.
2. In the same directory, run "tar -xf pcns300ESXi.tar" to extract the file.
3. Browse into the newly created "ESXi" folder and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0.
4. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts.
5. Accept the license agreement, and accept mostly defaults. When it gets to the part about java, be sure to let the installer install it's own bundled version of java.  
6. When you get to the part about entering an IP, just press "q" to skip it.
7. Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful.

8) Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver hostIPaddress". You will be asked to enter the password for your host, so vMA will have it on file. Do this for each host in the order you would like them shut down.

• IMPORTANT: Make sure that the vMA is located on the last host you add to the fastpass list. This makes sure that your vMA is one of the last things shut down, and that it is able to give your final host the sutdown command, which will in turn shut down the vMA. You'll probably want to disable vMotion for the vMA too, or only map it's storage to the final host, so that it cannot be moved.

9) Now, we need to configure PCNS. Open a web browser and go to https://IPofVMA:6547 and follow the configuration wizard. The only step worth menitoning is that you should NOT check the box for "Turn off the UPS after shutdown finishes." If you check it, there's a good chance the UPS will turn off while your hosts are still shutting down. The caveat to leaving this unchecked is that your ESXi servers most likely will not turn back on if power is restored before the UPS battery loses all of it's charge... you'll have to use iLO or DRAC to get in and turn your hosts on instead.

10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)

11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.

12) If you are using VMware HA... you have a few more steps to follow in order to get your VMs to shut down cleanly. See the following article for more details and a script that you should add to your battery event: http://nam-en.apc.com/app/answers/detail/a_id/11622/related/1/session/L2F2LzEvdGltZS8xMzQ1ODI3Njg1L3NpZC9fZG9xUXY0bA%3D%3D

In short, it has you upload the following script to your vMA, then add it as a shutdown option:

shutdownvms.sh:

#!/bin/sh
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib
export LD_LIBRARY_PATH
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
hosts=(10.216.252.167 10.216.252.168)
ups_vm="vSphere Management Assistant \(vMA\)"
for host in ${hosts[@]}; do
echo $host
source /opt/vmware/vma/bin/vifptarget -s $host
for i in `vmware-cmd -l --username xxxxx --password xxxxx`; do
if [ `vmware-cmd $i getstate | egrep -c "on"` -eq 1 ]; then
echo $i
if [ `echo $i | egrep -c $ups_vm` -eq 1 ]; then
echo "Skip shut down of VMA"
else
echo "Shutting down $i"
vmware-cmd "$i" stop soft
fi
fi
done
source /opt/vmware/vma/bin/vifptarget -c
done
 

I completed this by downloading the script file they had precreated, then used Veeam FastSCP to upload the file to the /tmp directory. Once there, I changed the permissions as indicated in the instructions (sudo chmod +x shutdownvms.sh), and followed that up by moving the file to the /opt/APC folder, just so I didn't forget what it was for. After that, I just went into the "Configure Shutdown" page on the PowerChute network Shutdown web interface and entered the script and it's path in the "Run this command" box. I gave my system 300 seconds to shut everything down, which seems to be sufficient.

13) If you are not using HA, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down! (Note, this doesn't apply to HA configurations, as HA will keep disabling your shutdown options.)


That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!

Note: If your hosts still refuse to shut down, check out this article for a probable fix: http://nam-en.apc.com/app/answers/detail/a_id/11621/related/1. The long and short of the article is that you need to add the following line of code to the /opt/APC/PowerChute/group1/bin/shutdown file right after the line that says "export LD_LIBRARY_PATH":

export PERL_LWP_SSL_VERIFY_HOSTNAME=0

For linux noobs... you can edit the file by giving the command "sudo vi pathAndFileName", then pressing "i" to "insert" text... then edit the file like normal. Once you are done, press escape, then type ":" and "wq" then hit enter. Simple fix.