Friday, August 24, 2012

Installing and Configuring APC PowerChute Network Shutdown 3.0.1 for ESXi 5.0

I wrote this article original for ESXi 4.1 and PCNS 3.0.0, but enough has changed that I figured it warranted a new article... enjoy!

Shutting down windows servers with the APC network shutdown software is a no-brainer... but what about virtual machines? Sure, you could install the network shutdown software on every virtual machine, but that would be wasteful. Fortunately, there's a centralized way that you can shut down your virtual guests AND hosts... and the best part? It's (mostly) free! All you need is an APC battery backup unit, an APC network shutdown card, and a bunch of software. Here's what you need to do:

1) Download stuff. There's a bunch of it!
  1. Download PCNS 3.0.1 from the APC website. It's free now, and you do not even need to register for it (finally, somebody gets it!). You can find it here: http://www.apc.com/tools/download/software_comp.cfm?sw_sku=SFPCNS301&id=127&swfam=127
  2. While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
  3. Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 5.0.0.2-724898.
  4. Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap. Note that fastscp is now bundled into the full backup and recovery tool... so grab that.

2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.

3) Create one vMA virtual machine by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need to pick a datastore for its 5GB volume. I usually make the volume 10GB for snapshot space. There are options to give it an IP, but i found that it didn't end up mattering due to the following error when trying to power on the newly created vMA:

  • The only workaround I found for this error was to edit the newly created vMA, go to the Options tab, and under vApp Options select "Disabled". You'll get a nag screen indicating it is removing properties, but the only options that I could find set were items about DHCP, which we aren't using...

4) Open the console of the vMA, power it on, and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local. Once you are completed, you should be dropped to a blue and grey login screen.

5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.

6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.

7) Back in the vMA console, choose the option to "login", enter the vi-admin credentials, and do the following:
  1. Browse to the tmp directory (for linux noobs, type cd /tmp) and run "gunzip pcns301ESXi.tar.gz" to unzip the file.
  2. In the same directory, run "tar -xf pcns300ESXi.tar" to extract the file.
  3. Browse into the newly created "ESXi" folder and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0.
  4. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts.
  5. Accept the license agreement, and accept mostly defaults. When it gets to the part about java, be sure to let the installer install it's own bundled version of java.  
  6. When you get to the part about entering an IP, just press "q" to skip it.
  7. Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful.
8) Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver hostIPaddress". You will be asked to enter the password for your host, so vMA will have it on file. Do this for each host in the order you would like them shut down.
  • IMPORTANT: Make sure that the vMA is located on the last host you add to the fastpass list. This makes sure that your vMA is one of the last things shut down, and that it is able to give your final host the sutdown command, which will in turn shut down the vMA. You'll probably want to disable vMotion for the vMA too, or only map it's storage to the final host, so that it cannot be moved.
9) Now, we need to configure PCNS. Open a web browser and go to https://IPofVMA:6547 and follow the configuration wizard. The only step worth menitoning is that you should NOT check the box for "Turn off the UPS after shutdown finishes." If you check it, there's a good chance the UPS will turn off while your hosts are still shutting down. The caveat to leaving this unchecked is that your ESXi servers most likely will not turn back on if power is restored before the UPS battery loses all of it's charge... you'll have to use iLO or DRAC to get in and turn your hosts on instead.

10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)

11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.

12) If you are using VMware HA... you have a few more steps to follow in order to get your VMs to shut down cleanly. See the following article for more details and a script that you should add to your battery event: http://nam-en.apc.com/app/answers/detail/a_id/11622/related/1/session/L2F2LzEvdGltZS8xMzQ1ODI3Njg1L3NpZC9fZG9xUXY0bA%3D%3D

In short, it has you upload the following script to your vMA, then add it as a shutdown option:

shutdownvms.sh:
#!/bin/sh
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib
export LD_LIBRARY_PATH
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
hosts=(10.216.252.167 10.216.252.168)
ups_vm="vSphere Management Assistant \(vMA\)"
for host in ${hosts[@]}; do
echo $host
source /opt/vmware/vma/bin/vifptarget -s $host
for i in `vmware-cmd -l --username xxxxx --password xxxxx`; do
if [ `vmware-cmd $i getstate | egrep -c "on"` -eq 1 ]; then
echo $i
if [ `echo $i | egrep -c $ups_vm` -eq 1 ]; then
echo "Skip shut down of VMA"
else
echo "Shutting down $i"
vmware-cmd "$i" stop soft
fi
fi
done
source /opt/vmware/vma/bin/vifptarget -c
done
 
I completed this by downloading the script file they had precreated, then used Veeam FastSCP to upload the file to the /tmp directory. Once there, I changed the permissions as indicated in the instructions (sudo chmod +x shutdownvms.sh), and followed that up by moving the file to the /opt/APC folder, just so I didn't forget what it was for. After that, I just went into the "Configure Shutdown" page on the PowerChute network Shutdown web interface and entered the script and it's path in the "Run this command" box. I gave my system 300 seconds to shut everything down, which seems to be sufficient.

13) If you are not using HA, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down! (Note, this doesn't apply to HA configurations, as HA will keep disabling your shutdown options.)


That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!

Note: If your hosts still refuse to shut down, check out this article for a probable fix: http://nam-en.apc.com/app/answers/detail/a_id/11621/related/1. The long and short of the article is that you need to add the following line of code to the /opt/APC/PowerChute/group1/bin/shutdown file right after the line that says "export LD_LIBRARY_PATH":
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
For linux noobs... you can edit the file by giving the command "sudo vi pathAndFileName", then pressing "i" to "insert" text... then edit the file like normal. Once you are done, press escape, then type ":" and "wq" then hit enter. Simple fix.

14 comments:

  1. Hi, was read through your guide and try but not successful shutdown my ESXi host, if my ESXi in HA environment, do I need to test all ESXi host or can I just test single ESXi host will perform the same result? Because I was not "vifp addserver" for all ESXi host, I only add single ESXi host ip and test, and my shutdownvms.sh script also not add for all ESXi host.
    Need your advice, thank you.

    ReplyDelete
    Replies
    1. Hello,

      You shouldn't need to add all of your hosts in order to test shutdown... I believe it will only shut down the hosts that you have added using the "vifp addserver" command while setting up the vMA.

      Did you check the link at the bottom of the post about "if your host still won't shut down"? That's the most common problem, though I did not encounter it... I'll go over this process again in another environment soon, and I'll post back if I encounter any difficulties in it!

      Delete
  2. Hi,

    Thanks for the info, the KB from APC and the help from support couldn't provide me with the answers but in the KB http://www2.schneider-electric.com/resources/sites/SCHNEIDER_ELECTRIC/content/live/FAQS/159000/FA159780/en_US/PCNS%203.0.1%20on%20ESXi-vMA%205.pdf it says

    "Next you need to add the server to the fasspass list the command is vifptarget –s
    To verify there is communications between the client and the host run the command Vicfg-nics –l (lower
    case letter l) should display a list of the host NICs."

    Is this required? I don't see it mentioned in your guide or is that step merely to test the fasspass list?

    Thanks

    Ed

    ReplyDelete
  3. Infact I think I have answered my question above, it appears to be just to test as the shutdownvm script goes through and changes the target as needed.

    Also, in the shutdownvms.sh script. Do I need to change the username and password? If so, which ones? the vsphere host ones?

    Thanks

    Ed

    ReplyDelete
  4. Hi Andrew,
    great post - thanks for sharing.

    For those which would like to properly configure SSL with internal CA, I share the required commands to handle JRE into PCNS. Thanks to another webiste I found the APC keystore password, otherwise it was going to be impossible...

    Ciao,
    Marco

    APC Network Shutdown SSL procedure

    Links:
    http://tsbraindump.blogspot.dk/2012/08/installing-and-configuring-apc.html
    http://securitybraindump.blogspot.dk/2010/10/hacking-fix.html

    Keystore password:
    0a15k8mk1oiw26uv4np29def73x0bjsh



    --


    to list current certs
    vi-admin@vMA01:/opt/APC/PowerChute/jre/bin> sudo ./keytool -list -v -keystore ../../group1/keystore

    to delete the self signed cert
    vi-admin@vMA01:/opt/APC/PowerChute/jre/bin> sudo ./keytool -delete -alias securekey -keystore ../../group1/keystore

    to import signed cert
    vi-admin@vMA01:/opt/APC/PowerChute/jre/bin> sudo java -classpath ../../group1/lib/jetty-6.1.26.jar org.mortbay.jetty.security.PKCS12Import "/tmp/ssl/your.pfx" ../../group1/keystore
    Enter input keystore passphrase:
    Enter output keystore passphrase: 0a15k8mk1oiw26uv4np29def73x0bjsh
    Alias 0: le-67af823c-d0d9-4020-941b-bc08c6f1c424
    Adding key for alias le-67af823c-d0d9-4020-941b-bc08c6f1c424

    to change the random alias, le-67af823c-d0d9-4020-941b-bc08c6f1c424
    vi-admin@vMA01:/opt/APC/PowerChute/jre/bin> sudo ./keytool -changealias -alias le-67af823c-d0d9-4020-941b-bc08c6f1c424 -destalias securekey -keystore ../../group1/keystore

    to import ca cert into keystore
    vi-admin@vMA01:/opt/APC/PowerChute/jre/bin> sudo ./keytool -import -trustcacerts -alias cacert -file /tmp/ssl/ca.cer -keystore ../../group1/keystore

    to delete an already available alias into the keystore
    vi-admin@vMA01:/opt/APC/PowerChute/jre/bin> sudo ./keytool -delete -alias securekey -keystore ../../group1/keystore

    ReplyDelete
  5. Does this also shut down the ESXi host?

    ReplyDelete
    Replies
    1. Yep, that's exactly what this is designed to do... shut down your guests, and once the guests are off it shuts down the host.

      Delete
  6. @Anonymous - Yes it will shut down your ESXi Host.

    ReplyDelete
  7. I was searching for Configuring APC PowerChute, And came across your blog. It helped!
    APC back-ups

    ReplyDelete
  8. Many thanks Andrew, this helped out a whole bunch!

    ReplyDelete
  9. Great article, not a linux guy and this really helped put it all together. I was wondering if there is a way to shutdown the vm's in a certain order.

    ReplyDelete
  10. Can you install PowerChute Network Shutdown standalone on each VM?
    Meaning, no central management, just each VM has its own install and configuration querying/receiving power status via the UPS Management Card?

    ReplyDelete