Thursday, July 28, 2011

Setting up PCNS 3.0.0 for ESXi 4.1 update 1

Note: An updated version of these instructions for ESXi 5.0 and PCNS 3.0.1 is available in a new post here:

I've done this setup 3 times now, and every time I end up having to resort to trolling the internet for instructions, so I figured it's about time I write down my own. Note these are from personal experience, contacting support, etc etc.

1) Download stuff. There's a bunch of it!
a) Download PCNS 3.0.0 from the APC website. It's free now, it just requires an login.
b) While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
c) Download the latest VMware vMA from the website under 'tools'. As of this writing, it's 4.1 released in 2010.
d) Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap.

2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.

3) Create one vMA virtual machine FOR EACH HOST by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need a 5GB volume (I usually make the volume 10GB for snapshot space), 1 processor, and a gig or so of ram. Note that I said "for each host". I spoke to APC support, and they admitted (finally) that the vifp 'fastpass' stuff does not work. My personal experiences confirm this. For some reason it only shuts down the first host on it's list!

4) Open the console of the vMA and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local.

5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.

6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.

7) Back in the vMA console, browse to the tmp directory and run "gunzip pcns300ESXi.tar.gz", then run "tar -xf pcns300ESXi.tar" to extract the file. Browse into the ESXi folder that is created, and run "sudo ./" to start the installation of pcns 3.0.0. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts. Continue with all the defaults. When you get to the part about entering an IP, just press "q" to skip it.

8) Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful. Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver". You will be asked to enter the password for your host, so vMA will have it on file.

9) Now, we need to configure PCNS. Open a web browser and go to https://:6547 and follow the configuration wizard. The only step worth menitoning is that you should NOT check the box for "Turn off the UPS after shutdown finishes." If you check it, there's a good chance the UPS will turn off while your hosts are still shutting down. The caveat to leaving this unchecked is that your ESXi servers most likely will not turn back on if power is restored before the UPS battery loses all of it's charge... you'll have to use iLO or DRAC to get in and turn your hosts on instead.

10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)

11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.

12) Finally, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down!

13 Repeat these steps for each host in your cluster. Yes, you need one VMA per host! Don't believe me? Fine. Try it yourself. Or call APC. You'll see.

That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!


  1. Thank you for this post, it was very helpful.
    The description for "UPS: On Battery" is "Do you want to shut down the PCNS operating system when the selected event happens?"

    It doesn't make sense to me that the PCNS operating system would be shut down - I want my ESXi host to shut down. Poorly worded or a misunderstood configuration option?

  2. The guide for PCNS 3 on ESXi shows that you can add hosts to be shut down by adding servers to the "fasspass"
    Is this not reliable? I don't necessarily want a vMA for each host, and it would be more handy if the vMA were able to handle multiple hosts.

  3. it worked for me... with one vMA for a cluster of 3 vmhosts.
    just make sure that the vmhost where you have vMA is the last on the list (vifp listservers --long).

    1. Hrm, that would definitely make sense. The next time I am at one of my sites, I will give this a go and update my post if it works out well.

      The reason I ended up ultimately going with one per host was due to the comments that several different APC engineers made about 'fastpass' not actually working correctly in several cases. Not very reassuring, for a system that needs to be bulletproof. :P

  4. the PCNS is on a vMA VM on a ESXI host that is configured to shutdown??? How can the host get shutdown gracefully if the PCNS-vMA VM that it exists on is already off? (in my list of VMs it's is last to shutdown but that still comes before the shutdown of the host.

    In addition to all this... How to you drive a script to a SAN from the PCNS if both the VMs and Hosts are off, or does the script come from the NMC card on the UPS?

    1. I'll admit I haven't had to revisit this yet, but I will be doing so soon for another customer.

      I anticipate that we are able to shut down all other hosts/VM's from within one VM because of the way VMware handles shutdown requests. You would first of all need to configure the 'vm startup/shutdown' options on your host. Once those are configured, if your host receives a shutdown command, it will in turn shut down all your guests before shutting itself down. It's like a recursive call to a program, if you will. The command to shut down the host has already been made at the point that the vMA guest is being shut down to the host, so it is no longer needed to complete the shutdown. The key I missed on my first attempt at this was that I was not ensuring the host with the vMA was the last host in the sequence... so I expect the vMA was getting shut down before it was able to tell my other hosts to shut down.

      Anyway, I'll be able to confirm this first hand before too much longer.

    2. As to your question about shutting down a SAN... that command would have to come from your UPS, as you cannot really tell your SAN to shut down before all of your guests and hosts are off... but once all your hosts and guests are off you have nothing left to issue the command.

      I have not successfully shut down a SAN via UPS before. I am sure it can be done, and it's certainly worth looking into...

  5. Just a note in case you missed the edit to the above post... there is a new post for ESXi 5 available here:

  6. does PCNS have to be installed on each of the virtual machines? or is this controlled by the Virtual Machine Startup/Shutdown? What if one of these virtual machines is a database server and you need to run a batch job to shutdown the database before you can shut down the server?