There's a lot of mis-information out there on the net right now about how to properly do redirection of the Outlook Web Access URL from an HTTP request to an HTTPS request. Most of this is due to how Exchange and OWA have changed over the past few versions... but some of this confusion comes from Microsoft itself, and their Technet articles! If you check out the Technet article "Simplify the Outlook Web App URL" at http://technet.microsoft.com/en-us/library/aa998359%28v=exchg.150%29.aspx, the article guides you through modifying the SSL settings, changing the HTTP redirect option, then going back through and cleaning up the mess you made in IIS due to properties inheritance on sub directories of the default website. What's worse, it doesn't even work!
As it turns out, this process is actually just a copy/paste from the Exchange 2010 process: http://social.technet.microsoft.com/wiki/contents/articles/921.simplify-the-outlook-web-app-url-in-exchange-server-2010.aspx
So... what's the right way to do this? A custom error page! It's so much simpler than all of the mess above, and the changes that are made to IIS7 to make it happen are much less intrusive, requiring no cleanup or anything.
1) First of all, log into your Exchange 2013 CU2 CAS server and open IIS7.
2) In the IIS7 Management Console, expand sites and select the Default Web Site (not the Exchange Back End site!)
3) In the center area, double click on the Error Pages icon, then on the right hand side under Actions click on "Add".
4) In the Add Custom Error Page window, under status code, enter 403.4. For the response action, select the third option "Respond with a 302 redirect" and under the Absolute URL field enter your full url, such as https://email.domain.com/owa. Don't forget the /owa on the end! Click OK to close the window.
5) Open up a command prompt and type "iisreset" to apply the changes.
That's it! Now try browsing to http://email.domain.com and validate that you are indeed redirected to https://email.domain.com/owa. So much easier! The other real benefit is that you can leave the "require SSL" box checked on the default website and OWA subdirectory using this process, and you don't have to mess with the HTTP redirect option, which has the tendency to break all of your subdirectories by applying the redirection to them as well!
Thursday, November 7, 2013
Saturday, April 20, 2013
Port Address Translation and NAT in Cisco ASA 9.1
I spent way too much time on this issue not to make an entry about it... hopefully this will save someone else the 12 hours of my life that Cisco snatched away from me with this little gem.
I recently had the (dis)pleasure of upgrading an older Cisco ASA 5510 from ASA 8.3 to the latest release... ASA 9.1. The system was stuck several revisions behind due to the memory limitations they imposed after 8.3, which required adding 1GB of memory to the system. The upgrade went well enough, no errors and internet access worked fine on first boot... however... it was quickly noted that outgoing email wasn't working. A bit of poking around revealed that the customer's spam filtering smart host, which was situated in the DMZ, was not sending email over the IP it was NAT'ed to... it was sending it out over the interface IP instead, causing SPF failures and mail rejection.
I spent hours building and rebuilding the NAT (which is completely different from the syntax in 8.3 mind you, so there was some learning curve here)... the lack of documentation online for our particular situation only exacerbated the problem. The key here was that the customer had a single external IP address that was previously responsible for both SMTP and HTTPS traffic... however... they had two internal hosts, a spam filter for SMTP in the DMZ, and an Exchange server for HTTPS in the LAN. In 8.3, it was a simple matter of doing a dynamic PAT for each host, assigning the IP that we wanted to NAT to and specifying the port it should use, with a matching ACL. Not so in 9.1!
Anyway, I'll spare you the endless futile attempts to figure out what eventually worked... and jump right into the code. First, I had to create TWO objects for each host that I wanted to have a PAT to (you'll see why in a minute):
Next, I set up my access control lists:
Finally, I created my NATs:
So... see what I did there? In the NAT statements, I ended up having to NAT the ip of each server TWICE... once as dynamic to the external IP address I wanted it to be NAT'ed to in the first two nat statements, and then again to the same ip as the first NAT, except this time specifying the incoming port for each server as static. Clear as mud? Yeah, same for me.
Apparently, this causes the incoming traffic to utilize the second two NAT statements (effectively the PAT statement from the 8.3), but the other two regular NAT statements are required to have the machines use the same IP for outgoing traffic.
The final two NAT statements above are just my regular interface dNAT entries, telling all other LAN and DMZ devices to utilize the interface IP for outgoing traffic.
Hopefully this helps someone else out there!
I recently had the (dis)pleasure of upgrading an older Cisco ASA 5510 from ASA 8.3 to the latest release... ASA 9.1. The system was stuck several revisions behind due to the memory limitations they imposed after 8.3, which required adding 1GB of memory to the system. The upgrade went well enough, no errors and internet access worked fine on first boot... however... it was quickly noted that outgoing email wasn't working. A bit of poking around revealed that the customer's spam filtering smart host, which was situated in the DMZ, was not sending email over the IP it was NAT'ed to... it was sending it out over the interface IP instead, causing SPF failures and mail rejection.
I spent hours building and rebuilding the NAT (which is completely different from the syntax in 8.3 mind you, so there was some learning curve here)... the lack of documentation online for our particular situation only exacerbated the problem. The key here was that the customer had a single external IP address that was previously responsible for both SMTP and HTTPS traffic... however... they had two internal hosts, a spam filter for SMTP in the DMZ, and an Exchange server for HTTPS in the LAN. In 8.3, it was a simple matter of doing a dynamic PAT for each host, assigning the IP that we wanted to NAT to and specifying the port it should use, with a matching ACL. Not so in 9.1!
Anyway, I'll spare you the endless futile attempts to figure out what eventually worked... and jump right into the code. First, I had to create TWO objects for each host that I wanted to have a PAT to (you'll see why in a minute):
object network Barracuda
host 192.168.12.3
object network Exchange
host 192.168.11.11
object network Exchange-SPAT
host 192.168.11.11
object network Barracuda-SPAT
host 192.168.12.3
Next, I set up my access control lists:
access-list outside_in extended permit tcp any4 object Exchange eq https
access-list outside_in extended permit tcp any4 object Barracuda eq smtp
access-list DMZ_access_in extended permit tcp object Barracuda object Exchange eq smtp
Finally, I created my NATs:
!
object network Barracuda
nat (DMZ,OUTSIDE) dynamic ExchangeExtIP
object network Exchange
nat (INSIDE,OUTSIDE) dynamic ExchangeExtIP
object network Exchange-SPAT
nat (INSIDE,OUTSIDE) static ExchangeExtIP service tcp https https
object network Barracuda-SPAT
nat (DMZ,OUTSIDE) static ExchangeExtIP service tcp smtp smtp
!
nat (INSIDE,OUTSIDE) after-auto source dynamic any interface
nat (DMZ,OUTSIDE) after-auto source dynamic any interface
So... see what I did there? In the NAT statements, I ended up having to NAT the ip of each server TWICE... once as dynamic to the external IP address I wanted it to be NAT'ed to in the first two nat statements, and then again to the same ip as the first NAT, except this time specifying the incoming port for each server as static. Clear as mud? Yeah, same for me.
Apparently, this causes the incoming traffic to utilize the second two NAT statements (effectively the PAT statement from the 8.3), but the other two regular NAT statements are required to have the machines use the same IP for outgoing traffic.
The final two NAT statements above are just my regular interface dNAT entries, telling all other LAN and DMZ devices to utilize the interface IP for outgoing traffic.
Hopefully this helps someone else out there!
Friday, September 7, 2012
2008 R2 Domain Controllers and Scanning to a Folder
I am really surprised I haven't run into this before... but I guess there is a weird bug with many copiers out there that prevents scan to folder from working correctly under the following circumstances:
1) The source copier is using domain authentication
2) The source copier is trying to scan to a folder
3) The destination shared folder is on a Windows 2008 or 2008 R2 server
4) The destination server is also a domain controller
5) The destination shared folder is shared using 'authenticated users' as it's share permission, rather than specific users or the 'Everyone' group.
When these conditions are met, the scanner seems unable to scan to a folder. I found the two fixes so far:
1) Change the 'share' level permissions to 'Everyone' and 'full access'. This allows the copier to send files without authentication, thus bypassing the problem.
2) Change the user specified in the copier as the authentication account from a DOMAIN\Username or just 'Username' to their UPN format... Username@domain.suffix.
Either of those two changes seem to do the trick!
1) The source copier is using domain authentication
2) The source copier is trying to scan to a folder
3) The destination shared folder is on a Windows 2008 or 2008 R2 server
4) The destination server is also a domain controller
5) The destination shared folder is shared using 'authenticated users' as it's share permission, rather than specific users or the 'Everyone' group.
When these conditions are met, the scanner seems unable to scan to a folder. I found the two fixes so far:
1) Change the 'share' level permissions to 'Everyone' and 'full access'. This allows the copier to send files without authentication, thus bypassing the problem.
2) Change the user specified in the copier as the authentication account from a DOMAIN\Username or just 'Username' to their UPN format... Username@domain.suffix.
Either of those two changes seem to do the trick!
Friday, August 24, 2012
Installing and Configuring APC PowerChute Network Shutdown 3.0.1 for ESXi 5.0
I wrote this article original for ESXi 4.1 and PCNS 3.0.0, but enough has changed that I figured it warranted a new article... enjoy!
Shutting down windows servers with the APC network shutdown software is a no-brainer... but what about virtual machines? Sure, you could install the network shutdown software on every virtual machine, but that would be wasteful. Fortunately, there's a centralized way that you can shut down your virtual guests AND hosts... and the best part? It's (mostly) free! All you need is an APC battery backup unit, an APC network shutdown card, and a bunch of software. Here's what you need to do:
1) Download stuff. There's a bunch of it!
2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.
3) Create one vMA virtual machine by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need to pick a datastore for its 5GB volume. I usually make the volume 10GB for snapshot space. There are options to give it an IP, but i found that it didn't end up mattering due to the following error when trying to power on the newly created vMA:
5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.
6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.
7) Back in the vMA console, choose the option to "login", enter the vi-admin credentials, and do the following:
10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)
11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.
12) If you are using VMware HA... you have a few more steps to follow in order to get your VMs to shut down cleanly. See the following article for more details and a script that you should add to your battery event: http://nam-en.apc.com/app/answers/detail/a_id/11622/related/1/session/L2F2LzEvdGltZS8xMzQ1ODI3Njg1L3NpZC9fZG9xUXY0bA%3D%3D
In short, it has you upload the following script to your vMA, then add it as a shutdown option:
shutdownvms.sh:
13) If you are not using HA, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down! (Note, this doesn't apply to HA configurations, as HA will keep disabling your shutdown options.)
That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!
Note: If your hosts still refuse to shut down, check out this article for a probable fix: http://nam-en.apc.com/app/answers/detail/a_id/11621/related/1. The long and short of the article is that you need to add the following line of code to the /opt/APC/PowerChute/group1/bin/shutdown file right after the line that says "export LD_LIBRARY_PATH":
Shutting down windows servers with the APC network shutdown software is a no-brainer... but what about virtual machines? Sure, you could install the network shutdown software on every virtual machine, but that would be wasteful. Fortunately, there's a centralized way that you can shut down your virtual guests AND hosts... and the best part? It's (mostly) free! All you need is an APC battery backup unit, an APC network shutdown card, and a bunch of software. Here's what you need to do:
1) Download stuff. There's a bunch of it!
- Download PCNS 3.0.1 from the APC website. It's free now, and you do not even need to register for it (finally, somebody gets it!). You can find it here: http://www.apc.com/tools/download/software_comp.cfm?sw_sku=SFPCNS301&id=127&swfam=127
- While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
- Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 5.0.0.2-724898.
- Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap. Note that fastscp is now bundled into the full backup and recovery tool... so grab that.
2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.
3) Create one vMA virtual machine by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need to pick a datastore for its 5GB volume. I usually make the volume 10GB for snapshot space. There are options to give it an IP, but i found that it didn't end up mattering due to the following error when trying to power on the newly created vMA:
- The only workaround I found for this error was to edit the newly created vMA, go to the Options tab, and under vApp Options select "Disabled". You'll get a nag screen indicating it is removing properties, but the only options that I could find set were items about DHCP, which we aren't using...
4) Open the console of the vMA, power it on, and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local. Once you are completed, you should be dropped to a blue and grey login screen.5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.
6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.
7) Back in the vMA console, choose the option to "login", enter the vi-admin credentials, and do the following:
- Browse to the tmp directory (for linux noobs, type cd /tmp) and run "gunzip pcns301ESXi.tar.gz" to unzip the file.
- In the same directory, run "tar -xf pcns300ESXi.tar" to extract the file.
- Browse into the newly created "ESXi" folder and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0.
- You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts.
- Accept the license agreement, and accept mostly defaults. When it gets to the part about java, be sure to let the installer install it's own bundled version of java.
- When you get to the part about entering an IP, just press "q" to skip it.
- Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful.
- IMPORTANT: Make sure that the vMA is located on the last host you add to the fastpass list. This makes sure that your vMA is one of the last things shut down, and that it is able to give your final host the sutdown command, which will in turn shut down the vMA. You'll probably want to disable vMotion for the vMA too, or only map it's storage to the final host, so that it cannot be moved.
10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)
11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.
12) If you are using VMware HA... you have a few more steps to follow in order to get your VMs to shut down cleanly. See the following article for more details and a script that you should add to your battery event: http://nam-en.apc.com/app/answers/detail/a_id/11622/related/1/session/L2F2LzEvdGltZS8xMzQ1ODI3Njg1L3NpZC9fZG9xUXY0bA%3D%3D
In short, it has you upload the following script to your vMA, then add it as a shutdown option:
shutdownvms.sh:
#!/bin/shI completed this by downloading the script file they had precreated, then used Veeam FastSCP to upload the file to the /tmp directory. Once there, I changed the permissions as indicated in the instructions (sudo chmod +x shutdownvms.sh), and followed that up by moving the file to the /opt/APC folder, just so I didn't forget what it was for. After that, I just went into the "Configure Shutdown" page on the PowerChute network Shutdown web interface and entered the script and it's path in the "Run this command" box. I gave my system 300 seconds to shut everything down, which seems to be sufficient.
LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/vmware/vma/lib64:/opt/vmware/vma/lib
export LD_LIBRARY_PATH
export PERL_LWP_SSL_VERIFY_HOSTNAME=0
SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
hosts=(10.216.252.167 10.216.252.168)
ups_vm="vSphere Management Assistant \(vMA\)"
for host in ${hosts[@]}; do
echo $host
source /opt/vmware/vma/bin/vifptarget -s $host
for i in `vmware-cmd -l --username xxxxx --password xxxxx`; do
if [ `vmware-cmd $i getstate | egrep -c "on"` -eq 1 ]; then
echo $i
if [ `echo $i | egrep -c $ups_vm` -eq 1 ]; then
echo "Skip shut down of VMA"
else
echo "Shutting down $i"
vmware-cmd "$i" stop soft
fi
fi
done
source /opt/vmware/vma/bin/vifptarget -c
done
13) If you are not using HA, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down! (Note, this doesn't apply to HA configurations, as HA will keep disabling your shutdown options.)
That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!
Note: If your hosts still refuse to shut down, check out this article for a probable fix: http://nam-en.apc.com/app/answers/detail/a_id/11621/related/1. The long and short of the article is that you need to add the following line of code to the /opt/APC/PowerChute/group1/bin/shutdown file right after the line that says "export LD_LIBRARY_PATH":
export PERL_LWP_SSL_VERIFY_HOSTNAME=0For linux noobs... you can edit the file by giving the command "sudo vi pathAndFileName", then pressing "i" to "insert" text... then edit the file like normal. Once you are done, press escape, then type ":" and "wq" then hit enter. Simple fix.
Sunday, August 19, 2012
Installing Dell Openmanage on an ESXi Host
Found this when paroozing the VMware communities sites for more info on Dell Openmanage for ESX.
Section 1: Install the vSphere Client and put the target host in maintenance mode
1. Connect to your Host Server via Internet address.
2. Download the vSphere Client appropriate for your version of ESXi
3. Install the vSphere Client
4. Open the vSphere Client and connect to your vCenter or directly to the host
5. Shut down all the guests on the target host, or migrate them to another host.
6. Right click on the target host in the left hand pane and click "Enter Maintenance Mode."
Section 2: Install Dell OpenManage
1. Download and install ‘vSphere CLI’ from VMware website (you will need to create an account in order to login and download the file):
http://www.vmware.com/support/developer/vcli/
Most recent at time of this post is: VMware-vSphere-CLI-5.0.0-615813.exe (https://my.vmware.com/group/vmware/get-download?downloadGroup=VCLI50U1)
2. Obtain the Dell OpenManage package for ESXi:
v4.0 OM-SrvAdmin-Dell-Web-6.3.0-2075.VIB-ESX40i_A00.9.zip
v4.1 OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
v5.0 OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip
ESXi 4.0: http://ftp.dell.com/sysman/OM-SrvAdmin-Dell-Web-6.3.0-2075.VIB-ESX40i_A00.9.zip
ESXi 4.1: http://ftp.dell.com/sysman/OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
ESXi 5.0: http://www.dell.com/support/drivers/us/en/84/DriverDetails/DriverFileFormats/Product/poweredge-2950?DriverId=Y0WHR&FileId=2990390081&urlProductCode=False
3. Launch ‘vSphere CLI’ via Start > VMware > VMware vSphere CLI > Command Prompt.
4a. On ESXi 4.X, do the following:
- In the CLI window, type "cd bin" and hit enter to change the directory to the "bin" subfolder
- type "Vihostupdate.pl --server IpAddressOfHost -i -b PathToVIBinstallPackage" and hit enter.
(eg: Vihostupdate.pl --server 192.168.0.1 -i -b C:\OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip)
4b. On ESXi 5.X, do the following:
- Copy the zip file containing the VIB to a datastore that the host can access. I used the vSphere client to connect to the host, then hit the configuration tab, Storage, then right clicked on the default local storage and hit "browse datastore". From that window I created a new folder named "updates" and uploaded the file into it.
- Take note of the Datastore location and name that is shown under "Datastore Details" in the bottom pane while the datastore you uploaded the file to is selected. My local store was /vmfs/volumes/502e86dd-dfa6c7f0-9ce2-0022197a5bef ...
- In the CLI window, type "esxcli --server IpAddressOfHost software vib install -d PathToVIBinstallPackage" and hit enter.
(eg: C:\Program Files (x86)\VMware\VMware vSphere CLI>esxcli --server 172.16.8.201 software vib install -d /vmfs/volumes/502e86dd-dfa6c7f0-9ce2-0022197a5bef/updates/OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip)
5. The system will prompt for the username ( root - then press enter) and the root password (then press enter). The update will now be applied to the ESXi server. The update will take approx 5mins on ESXi 4, and only moments on ESXi 5. (You can view the progress on the vSphere Client screen if you have it open).
Note, you may need to enable CIM providers as well... 4.1 and 5.0 do this automatically, but I think certain builds of 4.0 require you do it manually. You can do this by opening up the CLI again and typing CIMoemProviderEnabled = 1 (thanks to a comment below for pointing this out!)
6. Once completed, close the command prompt screen and reboot the ESXi server. Once the server has come back from being rebooted, using the vSphere Client, right click on the server which is in (maintenance mode) and choose “Exit Maintenance Mode”.
Section 3: Connect to the server via Server Administrator
1. Download the latest version of Dell Open Manage Server Administrator by entering the following into your web browser:
(Version 6.3) http://search.dell.com/results.aspx?s=biz&c=us&l=en&cs=555&k=managednode%2Cv.6.3.0&cat=sup
(Version 7.1) http://downloads.dell.com/FOLDER00574377M/1/OM-SrvAdmin-Dell-Web-WIN-7.1.0-5304_A00.exe
and searching for the right package for your needs.
For windows it is:
(6.3) OM-SrvAdmin-Dell-Web-WIN-6.3.0-2075_A00.20.exe (125MB)
(7.1) OM-SrvAdmin-Dell-Web-WIN-7.1.0-5304_A00 (186MB)
2. Install Server Administrator
3. Run Server Manager and choose "Manage a Remote Node", connect via IP Address, enter in the root and root password, and ensure "Ignore Certificate Warnings" is ticked.
NOTE: If your OMSA is responding extremely slow, as mine was, and you are using the Dell Customized version of ESXi 5.0 update 1, look at this article, which details an update to fix some CIM issues. You'll need to use vSphere Update Manager to install a patch on your hosts. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016538
Section 1: Install the vSphere Client and put the target host in maintenance mode
1. Connect to your Host Server via Internet address.
2. Download the vSphere Client appropriate for your version of ESXi
3. Install the vSphere Client
4. Open the vSphere Client and connect to your vCenter or directly to the host
5. Shut down all the guests on the target host, or migrate them to another host.
6. Right click on the target host in the left hand pane and click "Enter Maintenance Mode."
Section 2: Install Dell OpenManage
1. Download and install ‘vSphere CLI’ from VMware website (you will need to create an account in order to login and download the file):
http://www.vmware.com/support/developer/vcli/
Most recent at time of this post is: VMware-vSphere-CLI-5.0.0-615813.exe (https://my.vmware.com/group/vmware/get-download?downloadGroup=VCLI50U1)
2. Obtain the Dell OpenManage package for ESXi:
v4.0 OM-SrvAdmin-Dell-Web-6.3.0-2075.VIB-ESX40i_A00.9.zip
v4.1 OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
v5.0 OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip
ESXi 4.0: http://ftp.dell.com/sysman/OM-SrvAdmin-Dell-Web-6.3.0-2075.VIB-ESX40i_A00.9.zip
ESXi 4.1: http://ftp.dell.com/sysman/OM-SrvAdmin-Dell-Web-6.5.0-2247.VIB-ESX41i_A01.zip
ESXi 5.0: http://www.dell.com/support/drivers/us/en/84/DriverDetails/DriverFileFormats/Product/poweredge-2950?DriverId=Y0WHR&FileId=2990390081&urlProductCode=False
3. Launch ‘vSphere CLI’ via Start > VMware > VMware vSphere CLI > Command Prompt.
4a. On ESXi 4.X, do the following:
- In the CLI window, type "cd bin" and hit enter to change the directory to the "bin" subfolder
- type "Vihostupdate.pl --server IpAddressOfHost -i -b PathToVIBinstallPackage" and hit enter.
(eg: Vihostupdate.pl --server 192.168.0.1 -i -b C:\OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip)
4b. On ESXi 5.X, do the following:
- Copy the zip file containing the VIB to a datastore that the host can access. I used the vSphere client to connect to the host, then hit the configuration tab, Storage, then right clicked on the default local storage and hit "browse datastore". From that window I created a new folder named "updates" and uploaded the file into it.
- Take note of the Datastore location and name that is shown under "Datastore Details" in the bottom pane while the datastore you uploaded the file to is selected. My local store was /vmfs/volumes/502e86dd-dfa6c7f0-9ce2-0022197a5bef ...
- In the CLI window, type "esxcli --server IpAddressOfHost software vib install -d PathToVIBinstallPackage" and hit enter.
(eg: C:\Program Files (x86)\VMware\VMware vSphere CLI>esxcli --server 172.16.8.201 software vib install -d /vmfs/volumes/502e86dd-dfa6c7f0-9ce2-0022197a5bef/updates/OM-SrvAdmin-Dell-Web-7.1.0-5304.VIB-ESX50i_A00.zip)
5. The system will prompt for the username ( root - then press enter) and the root password (then press enter). The update will now be applied to the ESXi server. The update will take approx 5mins on ESXi 4, and only moments on ESXi 5. (You can view the progress on the vSphere Client screen if you have it open).
Note, you may need to enable CIM providers as well... 4.1 and 5.0 do this automatically, but I think certain builds of 4.0 require you do it manually. You can do this by opening up the CLI again and typing CIMoemProviderEnabled = 1 (thanks to a comment below for pointing this out!)
6. Once completed, close the command prompt screen and reboot the ESXi server. Once the server has come back from being rebooted, using the vSphere Client, right click on the server which is in (maintenance mode) and choose “Exit Maintenance Mode”.
Section 3: Connect to the server via Server Administrator
1. Download the latest version of Dell Open Manage Server Administrator by entering the following into your web browser:
(Version 6.3) http://search.dell.com/results.aspx?s=biz&c=us&l=en&cs=555&k=managednode%2Cv.6.3.0&cat=sup
(Version 7.1) http://downloads.dell.com/FOLDER00574377M/1/OM-SrvAdmin-Dell-Web-WIN-7.1.0-5304_A00.exe
and searching for the right package for your needs.
For windows it is:
(6.3) OM-SrvAdmin-Dell-Web-WIN-6.3.0-2075_A00.20.exe (125MB)
(7.1) OM-SrvAdmin-Dell-Web-WIN-7.1.0-5304_A00 (186MB)
2. Install Server Administrator
3. Run Server Manager and choose "Manage a Remote Node", connect via IP Address, enter in the root and root password, and ensure "Ignore Certificate Warnings" is ticked.
NOTE: If your OMSA is responding extremely slow, as mine was, and you are using the Dell Customized version of ESXi 5.0 update 1, look at this article, which details an update to fix some CIM issues. You'll need to use vSphere Update Manager to install a patch on your hosts. http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016538
Thursday, July 12, 2012
More on Exchange 2003 and 2010 Migration and Coexistence
A while back I made a post on migrating from SBS2003 to SBS2011, and the steps required to get coexistence working under that specific scenario, namely revolving around getting OWA redirection working. Those requirements definitely still apply here, but there were a few assumptions made based on some of the default setup work that SBS does for you. Here are some additional 'notes' about doing the same migration and coexistence without SBS factored in (So just a strait Exchange 2003 standard to Exchange 2010 standard migration.)
1) You need to configure your Exchange 2003 legacy URL. There isn't a GUI option for this... you have to issue the following command:
2) This took me a while to figure out because the fix is completely non-intuitive. Upon configuring your Exchange 2003 legacy url and testing it out, you get an "HTTP 500 Internal Server Error" with no further information. The url stops at https://(legacyurl)/exchweb/bin/auth/owaauth.dll.
To fix this you must enable FBA (forms based authentication) on your Exchange 2003 frontend server before OWA redirection will work. I found the following video showing the steps required: http://www.youtube.com/watch?v=B8NAmFqGOl4
Basically, you need to follow all the steps in my previous post, and in addition open up the Exchange 2003 management console, expand your way down to Administrative groups > first administrative group > servers > servername > Protocols > HTTP > Exchange Virtual Server, right click on the virtual server and go to properties. Under the settings tab, check the box for "Enable Forms Based Authentication" and click ok. After that, issue an IISRESET to restart your default website.
IMPORTANT: If you have only one Exchange 2003 backend server, and you configure SSL or FBA, you will likely break any activesync connections when you enable FBA. Here is an article explaining the situation from microsoft: http://support.microsoft.com/default.aspx?scid=kb;en-us;817379
Now... that seems like the proper way to go about things, but the major issue I noticed with my particular environment is that it was authentication that was broken for the mobile uses. A quick investigation of IIS after enabling FBA showed me that the 'windows authentication' option for the Exchange virtual directory was turned off. Checking the box fixed my problem! I think the only reason this worked for this particular site was because they did not have the "require SSL" box checked for the Exchange virtual directory. Not best practices, but this server will be gone soon anyway. As always, your mileage may vary. It's probably safer (or at least more supported) to follow the article I linked above.
1) You need to configure your Exchange 2003 legacy URL. There isn't a GUI option for this... you have to issue the following command:
Set-OwaVirtualDirectory -identity "Exchange2010\owa (default web site)" -Exchange2003Url https://legacy.domain.com/exchangeObviously, you should put your own server and domain name in there. This step is a no brainer... I just felt like documenting it here so that I don't have to look up the command syntax anymore. :P
2) This took me a while to figure out because the fix is completely non-intuitive. Upon configuring your Exchange 2003 legacy url and testing it out, you get an "HTTP 500 Internal Server Error" with no further information. The url stops at https://(legacyurl)/exchweb/bin/auth/owaauth.dll.
To fix this you must enable FBA (forms based authentication) on your Exchange 2003 frontend server before OWA redirection will work. I found the following video showing the steps required: http://www.youtube.com/watch?v=B8NAmFqGOl4
Basically, you need to follow all the steps in my previous post, and in addition open up the Exchange 2003 management console, expand your way down to Administrative groups > first administrative group > servers > servername > Protocols > HTTP > Exchange Virtual Server, right click on the virtual server and go to properties. Under the settings tab, check the box for "Enable Forms Based Authentication" and click ok. After that, issue an IISRESET to restart your default website.
IMPORTANT: If you have only one Exchange 2003 backend server, and you configure SSL or FBA, you will likely break any activesync connections when you enable FBA. Here is an article explaining the situation from microsoft: http://support.microsoft.com/default.aspx?scid=kb;en-us;817379
Now... that seems like the proper way to go about things, but the major issue I noticed with my particular environment is that it was authentication that was broken for the mobile uses. A quick investigation of IIS after enabling FBA showed me that the 'windows authentication' option for the Exchange virtual directory was turned off. Checking the box fixed my problem! I think the only reason this worked for this particular site was because they did not have the "require SSL" box checked for the Exchange virtual directory. Not best practices, but this server will be gone soon anyway. As always, your mileage may vary. It's probably safer (or at least more supported) to follow the article I linked above.
Wednesday, May 9, 2012
SBS 2011 and SBS 2003 Exchange Coexistence Mode
This is another nitpicky little set of steps that I always forget about and end up googling to get a solution to... so I figured I would finally write it down!
During the migration from SBS 2003 to SBS 2011, there is likely some point where you will have a number of users still on 2003, and a number of users on your new 2011 box. Wouldn't it be great if everyone could get their email, access OWA, etc etc regardless of their mailbox location, thus lessening the pressure to get everything done in one 'big bang' migration? Sure it would. Well, here's the steps (although admittedly not very detailed... just high level... individual steps can be googled for more info):
1) Follow the SBS2003 to SBS2011 migration guide from microsoft up to the point where you are ready to Migrate your mailboxes. At this point, Microsoft guides you through a disruptive cutover rather than a smooth migration, so some tweaks are needed.
2) Make sure your SSL is set up appropriately on both servers. There are a few details that are paramount in getting this right. For one, you need to have an SSL certificate on both servers. Each server must also have a unique SSL certificate, as they cannot both resolve to the same name... otherwise this won't work right. If you want to migrate your existing SSL cert from SBS2003 to SBS2011 that's fine... just make sure you remove it from 2003 at the end of it, and generate a new one (either paid, which is a waste, or using the SBS2003 "Connect to the Internet" wizard to generate a self-signed one).
3) Make sure your DNS zones are set up appropriately so that each server is resolvable by their SSL certificate name. If you followed the SBS2011 wizard, it should have done this for you automatically by setting up an authoritative zone for servername.domain.com in your internal DNS server. You should do the same for your legacy SBS2003 server and it's new SSL certificate name.
4) Go into the properties of the Exchange virtual directory under your Default Website in IIS on your SBS 2003 box and check the Directory Security tab. Hit the Edit button under Authentication and Access Control and ensure that "Windows Authentication" is checked, in addition to whatever else was already there. If it's not checked... check it! This allows the two exchange servers to pass authentication to eachother... without it, your users will get prompted for authentication twice.
5) On your SBS 2011 server, open up your Exchange 2010 management shell and enter the following command: SetVirtualDirectory "Servername\owa (Default Web Site)" -Exchange2003Url https://OldServerName.Domain.com/Exchange . That tells your Exchange 2010 server where to send OWA requests when the user's mailbox is on the 2003 server.
Done! Now, just reconfigure your firewall to send the HTTPS/SMTP/Whatever requests to your new exchange server instead of your old one, and you should be off to the races. Resume following the SBS2011 migration guide, ignoring any steps you already did!
During the migration from SBS 2003 to SBS 2011, there is likely some point where you will have a number of users still on 2003, and a number of users on your new 2011 box. Wouldn't it be great if everyone could get their email, access OWA, etc etc regardless of their mailbox location, thus lessening the pressure to get everything done in one 'big bang' migration? Sure it would. Well, here's the steps (although admittedly not very detailed... just high level... individual steps can be googled for more info):
1) Follow the SBS2003 to SBS2011 migration guide from microsoft up to the point where you are ready to Migrate your mailboxes. At this point, Microsoft guides you through a disruptive cutover rather than a smooth migration, so some tweaks are needed.
2) Make sure your SSL is set up appropriately on both servers. There are a few details that are paramount in getting this right. For one, you need to have an SSL certificate on both servers. Each server must also have a unique SSL certificate, as they cannot both resolve to the same name... otherwise this won't work right. If you want to migrate your existing SSL cert from SBS2003 to SBS2011 that's fine... just make sure you remove it from 2003 at the end of it, and generate a new one (either paid, which is a waste, or using the SBS2003 "Connect to the Internet" wizard to generate a self-signed one).
3) Make sure your DNS zones are set up appropriately so that each server is resolvable by their SSL certificate name. If you followed the SBS2011 wizard, it should have done this for you automatically by setting up an authoritative zone for servername.domain.com in your internal DNS server. You should do the same for your legacy SBS2003 server and it's new SSL certificate name.
4) Go into the properties of the Exchange virtual directory under your Default Website in IIS on your SBS 2003 box and check the Directory Security tab. Hit the Edit button under Authentication and Access Control and ensure that "Windows Authentication" is checked, in addition to whatever else was already there. If it's not checked... check it! This allows the two exchange servers to pass authentication to eachother... without it, your users will get prompted for authentication twice.
5) On your SBS 2011 server, open up your Exchange 2010 management shell and enter the following command: SetVirtualDirectory "Servername\owa (Default Web Site)" -Exchange2003Url https://OldServerName.Domain.com/Exchange . That tells your Exchange 2010 server where to send OWA requests when the user's mailbox is on the 2003 server.
Done! Now, just reconfigure your firewall to send the HTTPS/SMTP/Whatever requests to your new exchange server instead of your old one, and you should be off to the races. Resume following the SBS2011 migration guide, ignoring any steps you already did!
Monday, May 7, 2012
Allow Logon Through Terminal Services vs. Remote Desktop Users group: Which and why?
I always end up forgetting which permissions I need to grant users and where to do so when setting up terminal services... so I go googling, and end up finding the answer eventually... but I figured it's about time I left a note for myself (and others!) about which permissions you need to grant users, where they are, and why.
For starters, this article does a great job of explaining:
http://blogs.technet.com/b/askperf/archive/2011/09/09/allow-logon-through-terminal-services-group-policy-and-remote-desktop-users-group.aspx
I'll do a quick summary of the contents, though, and borrow the images in case that site goes missing at some point. All credit goes to the original poster, of course. :P
First, there's the "Allow Logon Through Terminal Services" GPO, located under Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment\. This policy is what controls granting access to the particular machine. When you assign this GPO to a particular machine and add a group to it, that group automatically gains rights to log on to this computer, access local resources, etc. By default, Administrators and Remote Desktop Users are assigned to this policy.
Second, there's the Remote Desktop Users group. This group, as you saw above, is already a member of the "Allow Logon Through Terminal Services" security setting on most servers by default (except for domain controllers, I believe the default domain controller policy overrides this setting allowing only Domain Admins... but I could be wrong here.). The other thing this group does is grant access to connect to the RDP-TCP service on the server. You can see/change which users and groups have access to the RDP-TCP listener by opening the Terminal Services Configuration snap in and checking the Security tab, as shown below:
Finally, here's a quick recap of the typical error messages you see, and what that generally means:
1) "To log on to this remote computer, you must have Terminal Server User Access permissions on this computer..." or "The Requested session access is denied": This error means that the user that tried to connect has been assigned to the GPO correctly for "Allow Logon Through Terminal Services", but the user is not a member of the Remote Desktop Users group, or otherwise does not have permissions to the RDP-TCP listener on that machine. Go check the Terminal Services Configuration snap in.
2) "To log on to this remote computer, you must be granted the Allow Logon Through Terminal Services right..." or "The connection was denied because the user account is not authorized for remote logon." This error is pretty strait forward... the user is not assigned to the "Allow Logon Through Terminal Services" GPO.
For starters, this article does a great job of explaining:
http://blogs.technet.com/b/askperf/archive/2011/09/09/allow-logon-through-terminal-services-group-policy-and-remote-desktop-users-group.aspx
I'll do a quick summary of the contents, though, and borrow the images in case that site goes missing at some point. All credit goes to the original poster, of course. :P
First, there's the "Allow Logon Through Terminal Services" GPO, located under Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment\. This policy is what controls granting access to the particular machine. When you assign this GPO to a particular machine and add a group to it, that group automatically gains rights to log on to this computer, access local resources, etc. By default, Administrators and Remote Desktop Users are assigned to this policy.
Second, there's the Remote Desktop Users group. This group, as you saw above, is already a member of the "Allow Logon Through Terminal Services" security setting on most servers by default (except for domain controllers, I believe the default domain controller policy overrides this setting allowing only Domain Admins... but I could be wrong here.). The other thing this group does is grant access to connect to the RDP-TCP service on the server. You can see/change which users and groups have access to the RDP-TCP listener by opening the Terminal Services Configuration snap in and checking the Security tab, as shown below:
Finally, here's a quick recap of the typical error messages you see, and what that generally means:
1) "To log on to this remote computer, you must have Terminal Server User Access permissions on this computer..." or "The Requested session access is denied": This error means that the user that tried to connect has been assigned to the GPO correctly for "Allow Logon Through Terminal Services", but the user is not a member of the Remote Desktop Users group, or otherwise does not have permissions to the RDP-TCP listener on that machine. Go check the Terminal Services Configuration snap in.
2) "To log on to this remote computer, you must be granted the Allow Logon Through Terminal Services right..." or "The connection was denied because the user account is not authorized for remote logon." This error is pretty strait forward... the user is not assigned to the "Allow Logon Through Terminal Services" GPO.
Friday, March 16, 2012
Assigning Trusted sites in IE via group policy preferences
Assigning sites to a user's trusted sites has always been a thorn in my side, mostly because the process for using group policies was such a pain... creating a blank profile, setting it up right, exporting it, blah blah... and to top it all off, assigning a policy prevented the user from adding any more sites to their trusted sites, making the policy settings more of a hinderance than a help.
Group policy preferences greatly enhances this process by 'adding' sites rather than enforcing only your selections. Here's a process I found off of a microsoft forum post:
1. First off create the registry entries manually, as shown below, on a reference machine. I've done this on the local machine where i need the keys to be added for all users. NOTE: The reference machine does not need to be where the keys have to be located.Manual Creation Steps:
a. launch regedit and go to: hkcu/software/microsoft/windows/currentversion/internet settings/zonemap/domains/
b.create a new key called microsoft.com. In the new key create a reg_dword(32) value called * and change the data to 2 hex.
c. repeat for any other domains the need to be trusted2. launch group policy management (again i did this from the machine where i need the keys but it is not required)
3. go to your GPO and select edit.
4. go to user preferences / windows settings / registry
5. right click registry / new / registry wizard
6. select local computer if you are on the computer where you created the reg entries and are running the GPO management gui. Otherwise choose another computer and select the reference machine from step 1.
7. the wizard will guide you through choosing the required entries, check off all required items. These entries are the ones created in step 1.
Location: hkcu/software/microsoft/windows/currentversion/internet settings/zonemap/domains//
8. Click Finish
9. You can then go back to this GPO preference and select its properties and utilize client side targeting if only certain AD groups need the values.
10. perform a replication to all DC's then a GP update /force on the machines in question; you will be asked to log out for the preferences to take. (or reboot)
Group policy preferences greatly enhances this process by 'adding' sites rather than enforcing only your selections. Here's a process I found off of a microsoft forum post:
1. First off create the registry entries manually, as shown below, on a reference machine. I've done this on the local machine where i need the keys to be added for all users. NOTE: The reference machine does not need to be where the keys have to be located.Manual Creation Steps:
a. launch regedit and go to: hkcu/software/microsoft/windows/currentversion/internet settings/zonemap/domains/
b.create a new key called microsoft.com. In the new key create a reg_dword(32) value called * and change the data to 2 hex.
c. repeat for any other domains the need to be trusted2. launch group policy management (again i did this from the machine where i need the keys but it is not required)
3. go to your GPO and select edit.
4. go to user preferences / windows settings / registry
5. right click registry / new / registry wizard
6. select local computer if you are on the computer where you created the reg entries and are running the GPO management gui. Otherwise choose another computer and select the reference machine from step 1.
7. the wizard will guide you through choosing the required entries, check off all required items. These entries are the ones created in step 1.
Location: hkcu/software/microsoft/windows/currentversion/internet settings/zonemap/domains/
8. Click Finish
9. You can then go back to this GPO preference and select its properties and utilize client side targeting if only certain AD groups need the values.
10. perform a replication to all DC's then a GP update /force on the machines in question; you will be asked to log out for the preferences to take. (or reboot)
Wednesday, February 1, 2012
SBS 2011 Remote Web Workplace "Login attempt failed" bug
Spent a few hours of my life that I never should have, figuring out this little gem. I had a freshly installed SBS 2011 server, with exchange mailboxes migrated and most of the migration wizard completed, when suddenly I noticed I could no longer use the remote web workplace to remote control any of my workstations or servers from the outside. I would get the login credentials box correctly identifying the server i was connecting to, but after entering my username and password it would come back indicating "The logon attempt failed." It worked just fine in tests earlier on, so something in the setup and migration wizard must have incorrectly tweaked something.
Turns out, it was an IIS configuration issue that appeared most likely after running the Network Preparation / Outlook Anywhere wizards. I opened IIS, expanded the default website, and inspected the authentication options for RPC and RPC with cert... both had only 'basic authentication' checked, which I was fairly certain would not work with Remote Desktop web access services. Enabling 'Windows Authentication' and performing an iisreset did the trick... but after a few minutes, it got set back to disabled! An event in the logs from "MSExchange RPC Over HTTP Autoconfig" source indicated that the settings for Outlook Anywhere had been updated, setting it back to Basic only!
The fix that eventually worked (and stuck) was to make the following change in the Exchange Management Shell:
After making that change, it may not show up in IIS right away... wait a few minutes, and that same Autoconfig service will update IIS for you, this time adding both Basic and Windows authentication. Voila!
Turns out, it was an IIS configuration issue that appeared most likely after running the Network Preparation / Outlook Anywhere wizards. I opened IIS, expanded the default website, and inspected the authentication options for RPC and RPC with cert... both had only 'basic authentication' checked, which I was fairly certain would not work with Remote Desktop web access services. Enabling 'Windows Authentication' and performing an iisreset did the trick... but after a few minutes, it got set back to disabled! An event in the logs from "MSExchange RPC Over HTTP Autoconfig" source indicated that the settings for Outlook Anywhere had been updated, setting it back to Basic only!
The fix that eventually worked (and stuck) was to make the following change in the Exchange Management Shell:
Get-OutlookAnywhere | Set-OutlookAnywhere –IISAuthenticationMethods: Basic, ntlm
After making that change, it may not show up in IIS right away... wait a few minutes, and that same Autoconfig service will update IIS for you, this time adding both Basic and Windows authentication. Voila!
Thursday, January 19, 2012
Recovering from 'Exclusive Access' rights when redirecting documents
One very useful, yet much hated, option when redirecting user documents is to "Grant user exclusive rights" to their redirected folders. If your users store sensitive or private information that not even administrators should be able to see in their folders, it's a great way to automate the process of users creating their own folders with locked down permissions upon initial logon. What happens when you NEED access to all those folders for a specific group, or for the admins? The Microsoft recommendation is to go through each folder and seize ownership, then re-apply the correct permissions... a tedious process to say the least. Well, there is a scripted way to do it that works much better (taken from http://mypkb.wordpress.com/2008/12/29/how-to-restore-administrators-access-to-redirected-my-documents-folder/):
First, you'll want to make sure that the offending GPO is turned off. It's an option under the folder redirection GPO (User configuration / Policies / Windows Settings / Folder Redirection) that needs to be unchecked, otherwise any new users will be set up with exclusive rights still.
Next, you'll need PSExec from SysInternals (http://technet.microsoft.com/en-us/sysinternals/bb897553.aspx) and PowerShell installed on the server hosting the shares. Log on to your file server as the LOCAL (not domain!) Administrator. Now, create the following powershell script, and be sure to edit the $StartingDir attirbute to point to the parent folder that contains all the users' folders:
#ChangePermissions.ps1
# CACLS rights are usually
# F = FullControl
# C = Change
# R = Readonly
# W = Write
$StartingDir= "C:\Users"
$Principal="Administrators"
$Permission="F"
$Verify=Read-Host `n "You are about to change permissions on all" `
"files starting at"$StartingDir.ToUpper() `n "for security"`
"principal"$Principal.ToUpper() `
"with new right of"$Permission.ToUpper()"."`n `
"Do you want to continue? [Y,N]"
if ($Verify -eq "Y") {
foreach ($file in $(Get-ChildItem $StartingDir -recurse)) {
#display filename and old permissions
write-Host -foregroundcolor Yellow $file.FullName
#uncomment if you want to see old permissions
#CACLS $file.FullName
#ADD new permission with CACLS
CACLS $file.FullName /E /P "${Principal}:${Permission}" >$NULL
#display new permissions
Write-Host -foregroundcolor Green "New Permissions"
CACLS $file.FullName
}
}
Save that .ps1 file to the C: drive or somewhere convenient, then open the command prompt as an administrator (right click, run as), and execute the following command:
psexec -s -i powershell -noexit "& 'C:\ChangePermissions.ps1'"
This will execute the powershell script as the local 'SYSTEM' account, which still has access to the 'exclusive' user directories, thus allowing you to modify permissions without having to seize control! Now that the LOCAL 'Administrators' group has permission to the folders, you can browse to the folders and modify permissions as you see fit. My recommendation would be to set the permissions you want on the parent folder, then just check the box for 'allow permissions to be inherited from the parent folder' so you do not have to manually add domain admins to each user folder.
Thursday, July 28, 2011
Setting up PCNS 3.0.0 for ESXi 4.1 update 1
Note: An updated version of these instructions for ESXi 5.0 and PCNS 3.0.1 is available in a new post here: http://tsbraindump.blogspot.com/2012/08/installing-and-configuring-apc.html
I've done this setup 3 times now, and every time I end up having to resort to trolling the internet for instructions, so I figured it's about time I write down my own. Note these are from personal experience, contacting support, etc etc.
1) Download stuff. There's a bunch of it!
a) Download PCNS 3.0.0 from the APC website. It's free now, it just requires an apc.com login.
b) While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
c) Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 4.1 released in 2010.
d) Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap.
2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.
3) Create one vMA virtual machine FOR EACH HOST by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need a 5GB volume (I usually make the volume 10GB for snapshot space), 1 processor, and a gig or so of ram. Note that I said "for each host". I spoke to APC support, and they admitted (finally) that the vifp 'fastpass' stuff does not work. My personal experiences confirm this. For some reason it only shuts down the first host on it's list!
4) Open the console of the vMA and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local.
5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.
6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.
7) Back in the vMA console, browse to the tmp directory and run "gunzip pcns300ESXi.tar.gz", then run "tar -xf pcns300ESXi.tar" to extract the file. Browse into the ESXi folder that is created, and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts. Continue with all the defaults. When you get to the part about entering an IP, just press "q" to skip it.
8) Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful. Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver". You will be asked to enter the password for your host, so vMA will have it on file.
9) Now, we need to configure PCNS. Open a web browser and go to https://:6547 and follow the configuration wizard. The only step worth menitoning is that you should NOT check the box for "Turn off the UPS after shutdown finishes." If you check it, there's a good chance the UPS will turn off while your hosts are still shutting down. The caveat to leaving this unchecked is that your ESXi servers most likely will not turn back on if power is restored before the UPS battery loses all of it's charge... you'll have to use iLO or DRAC to get in and turn your hosts on instead.
10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)
11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.
12) Finally, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down!
13 Repeat these steps for each host in your cluster. Yes, you need one VMA per host! Don't believe me? Fine. Try it yourself. Or call APC. You'll see.
That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!
I've done this setup 3 times now, and every time I end up having to resort to trolling the internet for instructions, so I figured it's about time I write down my own. Note these are from personal experience, contacting support, etc etc.
1) Download stuff. There's a bunch of it!
a) Download PCNS 3.0.0 from the APC website. It's free now, it just requires an apc.com login.
b) While you are there, grab the latest firmware for the APC card you are using. Make sure you know the kind of network management card you have, as the older ones are not able to use the latest firmware.
c) Download the latest VMware vMA from the VMware.com website under 'tools'. As of this writing, it's 4.1 released in 2010.
d) Grab the free version of Veeam fastscp... it makes loading the APC installer on the vMA a snap.
2) Install the latest network management card firmware first. It may take several shots at it... for some reason it likes to fail, but doing it over and over usually gets it going. Don't ask why.
3) Create one vMA virtual machine FOR EACH HOST by extracting that zip file you downloaded and then launching the vSphere client and going to File -> Deploy OVF Template. It will guide you through what's required, but you basically need a 5GB volume (I usually make the volume 10GB for snapshot space), 1 processor, and a gig or so of ram. Note that I said "for each host". I spoke to APC support, and they admitted (finally) that the vifp 'fastpass' stuff does not work. My personal experiences confirm this. For some reason it only shuts down the first host on it's list!
4) Open the console of the vMA and follow the initial setup instructions. I recommend assigning a static IP as well as a real hostname to the vMA for use later. For the hostname, specify a full domain name such as VMA.domain.local.
5) Install Veeam FastSCP, then open it. Click "Add Server" and add the vMA server as a "Linux Server". Be sure to use the vi-admin username and password you specified while setting up the vMA. Uncheck the box to "Elevate account to root" as we do not have root access - it's been disabled.
6) In FastSCP, browse to the pcns300ESXi.tar.gz file you downloaded earler and right click / copy it. Note that you have to right click / copy in FastSCP, not in windows explorer... FastSCP does not have windows clipboard access built in. Once copied, expand your vMA in FastSCP, browse to the tmp folder, and paste it in the root of tmp. You can close FastSCP now.
7) Back in the vMA console, browse to the tmp directory and run "gunzip pcns300ESXi.tar.gz", then run "tar -xf pcns300ESXi.tar" to extract the file. Browse into the ESXi folder that is created, and run "sudo ./install_en.sh" to start the installation of pcns 3.0.0. You will likely get prompted with a warning (read it, it's funny) and need to enter your password. After that, installation starts. Continue with all the defaults. When you get to the part about entering an IP, just press "q" to skip it.
8) Make sure you get the message about Installation has completed, and the note about how to access it - this means installation was successful. Now, you need to add your ESXi servers to the 'fast pass' access via the following command: "sudo vifp addserver
9) Now, we need to configure PCNS. Open a web browser and go to https://
10) Next, you will want to configure shutdown events. Once the PCNS wizard finishes it forwards you to the PCNS configuration page. Click on "Configure Events" and configure some of the important events like "UPS: On Battery" and "Input Power: Restored". You'll most likely want to notify users for most of the events. Don't forget to check the box for "Shut Down System" next to "UPS: On Battery" and set it to go off after a reasonable amount of time on battery (this depends on how much runtime your battery has, but I usually set mine for 5 minutes. If the power is off for 5 minutes, it's most likely going to be off an awful lot longer.)
11) Also, you'll want to check the "Connected Servers" tab on the left under "UPS information" to make sure your vMA IP address is listed. If not, you'll want to add it as a client on your UPS's network management card's web page.
12) Finally, in the vSphere client, make sure all of your ESXi hosts have their "Virtual Machine Startup / Shutdown" options configured and they are set to "Enabled", otherwise your guests are likely to just be turned off rather than shut down!
13 Repeat these steps for each host in your cluster. Yes, you need one VMA per host! Don't believe me? Fine. Try it yourself. Or call APC. You'll see.
That should get the system going! You can test it by, well, pulling the plug! If that's too scary for you, you can also configure the shutdown option for "PowerChute cannot communicate with the NMC", then pull the network cable from the network management card. If your hosts and guests shut down correctly, all is well!
Friday, July 1, 2011
Setting LUN Policy and PSP in ESXi 4.1
For customers with many LUNs, it can be a serious pain to create, modify, and maintain all the multipathing settings in VMware ESXi... I have a client with four active iSCSI adapters on each host, and four active iSCSI frontend ports on their Compellent SAN, which means a delightful 16 paths per LUN. Couple that with 2-3 LUNs per guest (raw device mappings), and 15 or so guests... that's a lot of clicking.
This is where VMware's vCLI is a godsend... you can script all of those monotonous clicks. For example, here's a pair of scripts I used to convert all of their Path Selection Policies and default PSP's to Round Robin.
1) Download and install vCLI
2) type "connect-viserver hostname" (where hostname is the name of the ESX host you want to make the changes on)
3) type "get-scsilun -vmhost (get-vmhost hostname) -luntype disk | set-scsilun -multipathpolicy "roundrobin""
4) Open an SSH prompt to the ESX host in question (you may need to enable SSH access in your hosts Configuration > Security Profile in vSphere).
5) In the SSH prompt, type "esxcli nmp satp setdefaultpsp --psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA
NOTE: The --satp type may vary depending on what SAN you use... you can check what your SATP currently is by clicking "manage paths" of any one of your LUNs in vSphere... under the path selection type it displays it as "storage array type".
That's it... now all the LUNs on the host should be round robin, and all future volumes should default to round robin as well!
Update:
For ESXi 5, the SSH command syntax has changed slightly. You'll want to type the following from the console of your ESXi 5 server:
esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA
Also, the part about vCLI is no longer necessary! Once you switch the default path policy with he above command, you can just reboot the host and it should apply the policy to all existing volumes. Thanks, VMware!
This is where VMware's vCLI is a godsend... you can script all of those monotonous clicks. For example, here's a pair of scripts I used to convert all of their Path Selection Policies and default PSP's to Round Robin.
1) Download and install vCLI
2) type "connect-viserver hostname" (where hostname is the name of the ESX host you want to make the changes on)
3) type "get-scsilun -vmhost (get-vmhost hostname) -luntype disk | set-scsilun -multipathpolicy "roundrobin""
4) Open an SSH prompt to the ESX host in question (you may need to enable SSH access in your hosts Configuration > Security Profile in vSphere).
5) In the SSH prompt, type "esxcli nmp satp setdefaultpsp --psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA
NOTE: The --satp type may vary depending on what SAN you use... you can check what your SATP currently is by clicking "manage paths" of any one of your LUNs in vSphere... under the path selection type it displays it as "storage array type".
That's it... now all the LUNs on the host should be round robin, and all future volumes should default to round robin as well!
Update:
For ESXi 5, the SSH command syntax has changed slightly. You'll want to type the following from the console of your ESXi 5 server:
esxcli storage nmp satp set --default-psp VMW_PSP_RR --satp VMW_SATP_DEFAULT_AA
Also, the part about vCLI is no longer necessary! Once you switch the default path policy with he above command, you can just reboot the host and it should apply the policy to all existing volumes. Thanks, VMware!
Wednesday, June 22, 2011
Exchange 2010 Edge Role + 2 NIC cards + DNS issues
I am trying out a few technologies that are new to me for one of our clients, namely Exchange 2010 with the Edge role on a DMZ server, as well as installing Threat Management Gateway on the Edge server... good times! I am sure there will be several posts on TMG coming soon... I can feel it now...
Anyway, I ran into a DNS problem with my fresh Exchange 2010 install that took a little bit of doing to figure out what was going on. The basic layout of the environment is that they have a single Exchange 2010 server on their LAN with the CAS/HUB/MBX roles installed, and a single EDGE role on a DMZ server that is on the domain. This Edge server has two NIC cards... one for the inside network so it can talk to the domain and do windows authentication, and one for the outside network so that it can receive mail and publish the Exchange web interface.
Shortly after installation, mail started queuing up on my edge server. Taking immediate advantage of the tools in front of me, I used the Queue viewer and noticed a few DNS warnings that certain domains could not be resolved. Thinking I was smart, I proceeded to put in hosts entries on the Edge server... nope, didn't solve it. I then thought that perhaps it was using the external interface's DNS server to try and resolve internal server names... so I took the DNS servers away from the external NIC card... still no luck.
Then, I stumbled upon a setting that suddenly cleared it all up: on the edge server, if you open up the Exchange management console, right click on the Edge server in the middle pane, and go to properties, there is a tab for 'External DNS lookups.' I set this to the Inside NIC card, and bam, mail started flowing.
Anyway, I ran into a DNS problem with my fresh Exchange 2010 install that took a little bit of doing to figure out what was going on. The basic layout of the environment is that they have a single Exchange 2010 server on their LAN with the CAS/HUB/MBX roles installed, and a single EDGE role on a DMZ server that is on the domain. This Edge server has two NIC cards... one for the inside network so it can talk to the domain and do windows authentication, and one for the outside network so that it can receive mail and publish the Exchange web interface.
Shortly after installation, mail started queuing up on my edge server. Taking immediate advantage of the tools in front of me, I used the Queue viewer and noticed a few DNS warnings that certain domains could not be resolved. Thinking I was smart, I proceeded to put in hosts entries on the Edge server... nope, didn't solve it. I then thought that perhaps it was using the external interface's DNS server to try and resolve internal server names... so I took the DNS servers away from the external NIC card... still no luck.
Then, I stumbled upon a setting that suddenly cleared it all up: on the edge server, if you open up the Exchange management console, right click on the Edge server in the middle pane, and go to properties, there is a tab for 'External DNS lookups.' I set this to the Inside NIC card, and bam, mail started flowing.
Monday, April 25, 2011
Dell Server P2V stuck at loading screen
Almost every P2V I have done to date has gone flawlessly. I frankly expected more trouble, but have been very pleased with VMware's product thus far. Almost every virtualization that experienced trouble was a result of third party software, or in this case, drivers for third party hardware.
The specific server in question was a Dell PowerVault 700 series NAS... it was running Windows Server 2003 Appliance edition, an OEM only version of windows 2003. Not really supported for virtualization, but the intention was to upgrade it to Datacenter Edition anyway once all was said and done... it was just a matter of getting there. After the virtualization process finished, the server powered on, and would not get past that animated windows logo loading screen with the grey dots moving across the bottom. I let it sit for up to 30 minutes before I was fairly sure there was no hope. The processor spiked at 100% the entire time, so I knew it was some kind of driver or service misbehaving. The system booted into safe mode just fine, but wouldn't boot into normal mode, even in diagnostic startup mode.
I worked with Microsoft, they didn't have a clue either. Time for process of elimination! I opened up MSINFO32 and made an export of all the Software Environment > System Drivers section for safe keeping, then went to work disabling all of them, section by section, starting with the ones marked as 'Stopped' with 'Start Mode' Set to Boot. I knew it had to be a 'stopped' one as windows was starting fine in safe mode, but not when everything started up normally. You can change them by opening up regedit and going to HKLM\System\Services. Expand each service name, and under it is a key called 'Startup' with a value of 0-4. 0 is 'Boot', 1 is 'System', 2 is 'Automatic', 3 is 'Manual', and 4 is 'Disabled'. Make sure to write down the original value of each 'startup' item as you set it to 4, because you will need to revert the changes.
Either way, disabling all the 'Boot' startup items didn't fix it, so I moved on to disabling all of the ones with 'start mode' set to 'System'. With all Boot and System items set to disabled, bingo! It started up. Now I put them each back to their original startup mode, one by one, rebooting into normal mode each time to see if the server would start. It finally narrowed down to one service, called "Msdisp." The description? "DELL LED DRIVER". Go figure! It was a driver for the hardware chassis to control the LED lights on the front of it, and it apparently blew up when it couldn't find whatever LED controller it was looking for. I left the service disabled, returned all others to their original values, and I was off to the races!
Hopefully this process of elimination method helps someone else out there!
The specific server in question was a Dell PowerVault 700 series NAS... it was running Windows Server 2003 Appliance edition, an OEM only version of windows 2003. Not really supported for virtualization, but the intention was to upgrade it to Datacenter Edition anyway once all was said and done... it was just a matter of getting there. After the virtualization process finished, the server powered on, and would not get past that animated windows logo loading screen with the grey dots moving across the bottom. I let it sit for up to 30 minutes before I was fairly sure there was no hope. The processor spiked at 100% the entire time, so I knew it was some kind of driver or service misbehaving. The system booted into safe mode just fine, but wouldn't boot into normal mode, even in diagnostic startup mode.
I worked with Microsoft, they didn't have a clue either. Time for process of elimination! I opened up MSINFO32 and made an export of all the Software Environment > System Drivers section for safe keeping, then went to work disabling all of them, section by section, starting with the ones marked as 'Stopped' with 'Start Mode' Set to Boot. I knew it had to be a 'stopped' one as windows was starting fine in safe mode, but not when everything started up normally. You can change them by opening up regedit and going to HKLM\System\Services. Expand each service name, and under it is a key called 'Startup' with a value of 0-4. 0 is 'Boot', 1 is 'System', 2 is 'Automatic', 3 is 'Manual', and 4 is 'Disabled'. Make sure to write down the original value of each 'startup' item as you set it to 4, because you will need to revert the changes.
Either way, disabling all the 'Boot' startup items didn't fix it, so I moved on to disabling all of the ones with 'start mode' set to 'System'. With all Boot and System items set to disabled, bingo! It started up. Now I put them each back to their original startup mode, one by one, rebooting into normal mode each time to see if the server would start. It finally narrowed down to one service, called "Msdisp." The description? "DELL LED DRIVER". Go figure! It was a driver for the hardware chassis to control the LED lights on the front of it, and it apparently blew up when it couldn't find whatever LED controller it was looking for. I left the service disabled, returned all others to their original values, and I was off to the races!
Hopefully this process of elimination method helps someone else out there!
Thursday, April 21, 2011
Performing a P2V on a Domain Controller
Yes, I know, it's not recommended. Heck, it's not even supported. But what if you want to throw caution to the wind, and do it anyway? Here's the method I used that worked successfully. A few notes though:
- This only works in environments with two or more domain controllers
- There may still be problems with clients authenticating should you be ballsy enough to do this during business hours, as I was. Rebooting usually fixes them.
1) You'll first of all want to transfer all of your FSMO roles to another DC in the environment. You can see which servers are hosting the FSMO roles in your environment by opening up a command prompt on your domain controller and typing "netdom query fsmo". You'll get back a neat list of which services are where. Transfer them to other domain controllers (I'll leave the specifics out of this article, google has the answers.)
2) Open a command prompt on the DC you wish to P2V and type dcpromo, then hit enter. Proceed through the process to remove active directory from this server. You may get an error the first time... it's okay... just run it again. It's complaining that services didn't stop or start in a timely fashion, which is normal for a DC. Reboot once you are done.
3) Uninstall DNS and WINS using Add/Remove programs. Also, if you are running DHCP, you may want to start up a split scope on another server, as you should disable the dhcp scope on this server for the time being.
4) Change the DNS on your server you wish to P2V so that it's primary DNS points to a VALID server other than itself... otherwise, your P2V will fail immediately. This is because the server being virtualized needs to be able to resolve the DNS names of the destination ESX server and VirtualCenter server. Change the DNS for your VirtualCenter server and ESX server so that they are all pointing to the same DNS server, just to be safe.
5) Run the P2V, it should go off without a hitch now that DNS is right.
6) Now that the server is back up and virtual, run dcpromo again to make it a domain controller once more. This should reinstall DNS for you, if you uninstalled it earlier. Fix up DHCP and WINS, and you are back in business. Don't forget to transfer back your FSMO roles, if you want them to be on your virtual server!
7) Readjust your DNS settings on the re-promoted domain controller to point to itself, and fix your VirtualCenter/ESX server DNS settings too, if you had to change them.
- This only works in environments with two or more domain controllers
- There may still be problems with clients authenticating should you be ballsy enough to do this during business hours, as I was. Rebooting usually fixes them.
1) You'll first of all want to transfer all of your FSMO roles to another DC in the environment. You can see which servers are hosting the FSMO roles in your environment by opening up a command prompt on your domain controller and typing "netdom query fsmo". You'll get back a neat list of which services are where. Transfer them to other domain controllers (I'll leave the specifics out of this article, google has the answers.)
2) Open a command prompt on the DC you wish to P2V and type dcpromo, then hit enter. Proceed through the process to remove active directory from this server. You may get an error the first time... it's okay... just run it again. It's complaining that services didn't stop or start in a timely fashion, which is normal for a DC. Reboot once you are done.
3) Uninstall DNS and WINS using Add/Remove programs. Also, if you are running DHCP, you may want to start up a split scope on another server, as you should disable the dhcp scope on this server for the time being.
4) Change the DNS on your server you wish to P2V so that it's primary DNS points to a VALID server other than itself... otherwise, your P2V will fail immediately. This is because the server being virtualized needs to be able to resolve the DNS names of the destination ESX server and VirtualCenter server. Change the DNS for your VirtualCenter server and ESX server so that they are all pointing to the same DNS server, just to be safe.
5) Run the P2V, it should go off without a hitch now that DNS is right.
6) Now that the server is back up and virtual, run dcpromo again to make it a domain controller once more. This should reinstall DNS for you, if you uninstalled it earlier. Fix up DHCP and WINS, and you are back in business. Don't forget to transfer back your FSMO roles, if you want them to be on your virtual server!
7) Readjust your DNS settings on the re-promoted domain controller to point to itself, and fix your VirtualCenter/ESX server DNS settings too, if you had to change them.
Thursday, March 31, 2011
Configuring Dell PowerConnect 6224 Switches for iSCSI traffic to a Compellent
This is the second time I have set up a Compellent, and this time I figured I would go all the way down the rabbit's hole and delve into switch optimization for iSCSI traffic. My last project was with Cisco switches, this time, it's with Dell. A pair of PowerConnect 6224 switches to be precise... main reason being, Dell purchased Compellent not long ago, so like-products, yadda yadda. Alas, I digress.
In previous talks with Compellent, I was give the '10 commandments' of switch configuration from at least two different representatives upon having iSCSI issues. These commandments roughly consisted of:
Okay, so it was more than 10. Anyway, all that roughly distills down to the following needs:
1) Set spanning-tree mode rstp and enable portfast on all ports, or disable spanning-tree all together.
2) Enable jumbo frame support on all iSCSI ports
3) Disable unicast storm-control on all iSCSI ports
4) Enable multicast storm-control on all iSCSI ports
5) Enable broadcast storm-control on all iSCSI ports
6) Enable flow control
For each switch, the commands are different... but for Dell PowerConnect 6224 in particular, these needs translated into the following commands (in order, considering I used ports 1-12 for iSCSI)
enable
configure
spanning-tree mode rstp
interface 1/g1-1/g12
spanning-tree portfast
mtu 9216
no storm-control unicast
storm-control multicast
storm-control broadcast
exit
flowcontrol
That was it, switch optimized, on we go!
In previous talks with Compellent, I was give the '10 commandments' of switch configuration from at least two different representatives upon having iSCSI issues. These commandments roughly consisted of:
• Gigabit Full Duplex connectivity between Storage Center and all local Servers
• Auto-Negotiate for all switches that will correctly negotiate at Gigabit Full Duplex
• Gigabit Full Duplex hard set for all iSCSI ports, for both Storage Center and Servers for switches that do not correctly negotiate
• Bi-Directional Flow Control enabled for all Switch Ports that servers or controllers are using for iSCSI traffic.
• Bi-Directional Flow Control enabled for all Server Ports used for iSCSI (Storage Center and QLogic HBA's automatically enable it).
• Bi-Directional Flow Control enabled for all ports that handle iSCSI traffic. This includes all devices in between two sites that are used for replication.
• Separate VLAN for iSCSI.
• Two separate networks or VLANs for multipathed iSCSI.
• Two separate IP subnets for the seperate networks or VLANs in multipathed iSCSI.
• Unicast storm control disabled on every switch that handles iSCSI traffic.
• Multicast disabled at the switch level for any iSCSI VLANs.
o Multicast storm control enabled (if available) when multicast can not disabled.
• Broadcast disabled at the switch level for any iSCSI VLANs.
o Broadcast storm control enabled (if available) when broadcast can not disabled.
• Routing disabled between regular network and iSCSI VLANs.
• Do not use Spanning Tree (STP or RSTP) on ports which connect directly to end nodes (the server or Compellent controllers iSCSI ports.) If you must use it, enable the Cisco PortFast option on these ports so that they are configured as edge ports.
• Ensure that any switches used for iSCSI are of a non-blocking design.
• When deciding which switches to use, remember that you are running SCSI traffic over it. Be sure to use a quality managed enterprise class networking equipment. It is not recommended to use SBHO (small business/home office) class equipment outside of lab/test environments.
• Verify optimal MTU for replications. Default is 1500 but sometimes WAN circuits or VPNs can create additional overhead which can cause packet fragmentation. This fragmentation can suboptimal performance. The MTU is adjustable via the GUI in 5.x Storage Center Firmware.
For Jumbo Frame Support
• Some switches have limited buffer sizes and can only support Flow Control or Jumbo Frames, but not both at the same time. Compellent strongly recommends choosing Flow Control.
• All devices connected through iSCSI need to support 9k jumbo frames.
• All devices used to connect iSCSI devices need to support it.
o This means every switch, router, WAN Accelerator and any other network device that will handle iSCSI traffic needs to support 9k Jumbo Frames.
• If the customer is not 100% positive that every device in their iSCSI network supports 9k Jumbo Frames, then they should NOT turn on Jumbo Frames.
• QLogic 4010 series cards (Early Compellent iSCSI Cards) do not support Jumbo Frames.
o In the Storage Center GUI default screen, expand the tree in the following order Controllers->SN#(for the controller)->IO Cards->iSCSI->Highlight the port and the general tab should list the model number in the description.
• Because devices on both sides (server and SAN) need Jumbo Frames enabled, the change to enable to disable Jumbo Frames is recommended during a maintenance window. If servers have it enabled first, the Storage Center will not understand their packets. If Storage Center enables it first, servers will not understand its packets.
Okay, so it was more than 10. Anyway, all that roughly distills down to the following needs:
1) Set spanning-tree mode rstp and enable portfast on all ports, or disable spanning-tree all together.
2) Enable jumbo frame support on all iSCSI ports
3) Disable unicast storm-control on all iSCSI ports
4) Enable multicast storm-control on all iSCSI ports
5) Enable broadcast storm-control on all iSCSI ports
6) Enable flow control
For each switch, the commands are different... but for Dell PowerConnect 6224 in particular, these needs translated into the following commands (in order, considering I used ports 1-12 for iSCSI)
enable
configure
spanning-tree mode rstp
interface 1/g1-1/g12
spanning-tree portfast
mtu 9216
no storm-control unicast
storm-control multicast
storm-control broadcast
exit
flowcontrol
That was it, switch optimized, on we go!
Tuesday, February 8, 2011
How to fix a blackberry sync issue (the secret CNFG menu)
So the title is a bit cryptic, but I ran into an issue where a certain blackberry decided to no longer sync it's contacts with our newly migrated BES server. No idea why, tried rebooting the phone, toggling wireless sync of contacts, etc etc... no luck. The user had a bunch of stuff on the phone, so I didn't really want to wipe it or reactivate it particularly. Found this little tidbit of info on Expert's Exchange at:
http://www.experts-exchange.com/Hardware/Handhelds_-_PDAs/Blackberry/Q_23024284.html
All you need to do is go to Options > Advanced Options > Enterprise Activation and in the email field press and hold the ALT key and type CNFG. Once you enter this a hidden menu will appear and you need to change "Wireless Sync" to No, now exit this menu and wait 30 seconds and repeat the process but turn sync back to Yes. Once you've changed this setting you will see a slow sync will automatically start and it will repair all the wireless sync settings. In the rare case that this fails you just need to wipe the device and reactivate.
http://www.experts-exchange.com/Hardware/Handhelds_-_PDAs/Blackberry/Q_23024284.html
All you need to do is go to Options > Advanced Options > Enterprise Activation and in the email field press and hold the ALT key and type CNFG. Once you enter this a hidden menu will appear and you need to change "Wireless Sync" to No, now exit this menu and wait 30 seconds and repeat the process but turn sync back to Yes. Once you've changed this setting you will see a slow sync will automatically start and it will repair all the wireless sync settings. In the rare case that this fails you just need to wipe the device and reactivate.
Monday, February 7, 2011
Moving ESXi guests without vCenter
vCenter is a very cool application, enabling you to do many things that would otherwise take hours to figure out and execute, especially in a multi-host environment (which is pretty typical for VMware). One such feature is the ability to move guest VM's between hosts. Using vCenter, it's as simple as right clicking the guest and picking 'Migrate', then picking the new destination and waiting. But what if you don't HAVE vCenter, or in my case, the server you want to move IS your vCenter server. Not so easy then!
Veeam to the rescue! If you don't know what Veeam is, you haven't been working with VMware very long. It's a great company that produces lots of useful software for managing VMware environments... and some of their best tools are free! In this case, their free product FastSCP is what we are after. Register for it and install it on any other server than the one you want to move. Setup is simple, and adding hosts is even simpler. Once you add a host, you can expand it's storage just like the storage browser in vCenter or the vSphere client, and moving servers is just a simple copy/paste operation. Well, okay, there are a few more steps:
1) If the server you are moving is part of a vCenter environment, it is highly recommended that you unregister it first. Even if it's not, you should unregister it anyway. Either from vCenter, or using vSphere to connect directly to the host, shut down the guest machine then right click and select 'Remove from Inventory'. Don't click delete! It will warn you about some stuff, just click yes and proceed. You'll notice the VM is no longer listed under your host. It's now just a bunch of files in a folder on the datastore.
2) Start up Veeam, and add the source and destination hosts to the manager if you haven't already. Expand the source host, right click on the folder for the virtual machine you want to move, and click copy. Then, expand the destination host, and right click in the right hand side's empty area, and click paste.
NOTE: If you installed Veeam on an x64 server, you may get errors when trying to copy and paste folders, or create new folders. It's a bug, and Veeam disclaimers that x64 installations are purely experimental at this point. There are two workarounds... either create the folder first using vSphere by browsing to the datastore on the host and making a new folder, then copying only files from one folder to another, or follow this method to fix up your server so that you won't get that error:
http://www.virtualvcp.com/content/view/26/1/ details below
1. Download the Microsoft .NET Framework 2.0 SDK (THE 64-BIT VERSION!) from the Microsoft Website. This download is about 300MB if I remember correctly.
2. Install the SDK on the 64-bit machine that you would like to run Veeam FastSCP on.
3. Now, open a command prompt (Start -> Run -> Type "cmd" -> OK)
4. Change directory to: C:\Program Files\Microsoft.NET\SDK\v2.0 64bit\Bin
5. Now Run: corflags "C:\Program Files (x86)\Veeam\Veeam Backup and FastSCP\VeeamShell.exe" /32BIT+
6. Now when you try and run Veeam FastSCP again, it should work fine.
3) Anyway, now that the files are moved, all you need to do is register the virtual machine on the new host. Open up vSphere, connect directly to the destination host, then browse the datastore. Find the .vmx file inside of the folder you copied and right click on it, then select 'Import' to add it to your inventory. You may get asked something to the effect of "Where did this virtual machine come from?"... in our case, you will want to pick "I moved it".
That's it, start up your server, it should work fine!
Just wait for the copy to finish, then delete the source... migration complete!
Veeam to the rescue! If you don't know what Veeam is, you haven't been working with VMware very long. It's a great company that produces lots of useful software for managing VMware environments... and some of their best tools are free! In this case, their free product FastSCP is what we are after. Register for it and install it on any other server than the one you want to move. Setup is simple, and adding hosts is even simpler. Once you add a host, you can expand it's storage just like the storage browser in vCenter or the vSphere client, and moving servers is just a simple copy/paste operation. Well, okay, there are a few more steps:
1) If the server you are moving is part of a vCenter environment, it is highly recommended that you unregister it first. Even if it's not, you should unregister it anyway. Either from vCenter, or using vSphere to connect directly to the host, shut down the guest machine then right click and select 'Remove from Inventory'. Don't click delete! It will warn you about some stuff, just click yes and proceed. You'll notice the VM is no longer listed under your host. It's now just a bunch of files in a folder on the datastore.
2) Start up Veeam, and add the source and destination hosts to the manager if you haven't already. Expand the source host, right click on the folder for the virtual machine you want to move, and click copy. Then, expand the destination host, and right click in the right hand side's empty area, and click paste.
NOTE: If you installed Veeam on an x64 server, you may get errors when trying to copy and paste folders, or create new folders. It's a bug, and Veeam disclaimers that x64 installations are purely experimental at this point. There are two workarounds... either create the folder first using vSphere by browsing to the datastore on the host and making a new folder, then copying only files from one folder to another, or follow this method to fix up your server so that you won't get that error:
http://www.virtualvcp.com/content/view/26/1/ details below
1. Download the Microsoft .NET Framework 2.0 SDK (THE 64-BIT VERSION!) from the Microsoft Website. This download is about 300MB if I remember correctly.
2. Install the SDK on the 64-bit machine that you would like to run Veeam FastSCP on.
3. Now, open a command prompt (Start -> Run -> Type "cmd" -> OK)
4. Change directory to: C:\Program Files\Microsoft.NET\SDK\v2.0 64bit\Bin
5. Now Run: corflags "C:\Program Files (x86)\Veeam\Veeam Backup and FastSCP\VeeamShell.exe" /32BIT+
6. Now when you try and run Veeam FastSCP again, it should work fine.
3) Anyway, now that the files are moved, all you need to do is register the virtual machine on the new host. Open up vSphere, connect directly to the destination host, then browse the datastore. Find the .vmx file inside of the folder you copied and right click on it, then select 'Import' to add it to your inventory. You may get asked something to the effect of "Where did this virtual machine come from?"... in our case, you will want to pick "I moved it".
That's it, start up your server, it should work fine!
Just wait for the copy to finish, then delete the source... migration complete!
Wednesday, November 17, 2010
Upgrading VMware ESX 4.1 guests to ver. 7 hardware
This is pretty straight forward, but thought I would log my process just in case I totally forget down the road.
First, you can check the version of the virtual machine by going into the 'edit settings' dialogue and looking in the top right hand corner. Version 7 is the latest and greatest.
1) Install the upgraded vmware tools and reboot the server
2) Shut down the guest after it reboots
3) Right click on the guest in the VMware console and select 'Upgrade Virtual Hardware'
4) Remove the old NIC card and add a fancy new VMXNET3 connection. You may want to make a note of your network config if you don't know it.
5) Start up the VM, and once in windows, configure the network settings.
6) Wait for the hardware to finish installing, reboot once more, and done!
First, you can check the version of the virtual machine by going into the 'edit settings' dialogue and looking in the top right hand corner. Version 7 is the latest and greatest.
1) Install the upgraded vmware tools and reboot the server
2) Shut down the guest after it reboots
3) Right click on the guest in the VMware console and select 'Upgrade Virtual Hardware'
4) Remove the old NIC card and add a fancy new VMXNET3 connection. You may want to make a note of your network config if you don't know it.
5) Start up the VM, and once in windows, configure the network settings.
6) Wait for the hardware to finish installing, reboot once more, and done!
Subscribe to:
Posts (Atom)