W3SVC1 Log Files Growing with WebDAV-Miniredir entries

One of my clients had a server with a somewhat limited C: drive. The server would occasionally fill up with log files that were hundreds of MB in size under the W3SVC folder with the same entry over and over... some kind of web request from a particular Vista machine to my server hitting WebDAV. This one plagued me for a while, but I finally sat down and traced where the error was being generated from.

According to a few internet sources, apparently there is some kind of problem with Vista and WebDAV, or more importantly Web Folders. I won't go into the details as they do not matter really... what's important is that Vista machines attempting to access a server running IIS and WebDAV may generate endless amounts of these WebDAV-Miniredir entries in your IIS logs. Eliminating whatever application is making these web requests is the key to preventing the log bloat.

In my case, the machine had Quickbooks 2009 installed, and along with it some kind of unused quickbooks web connector. The connector was set to run on startup (even though it wasn't doing anything), and thus whenever the vista machine was on it was generating a few lines of logs per second. Removing the web connector from the startup folder fixed the problem.

Installing 32 Bit Printer Drivers on a 64 Bit Windows 2008 Server

So here is a really dumb situation I found a resolution to after an hour wasted scratching my head. When installing an older printer on a Windows 2008 64 bit server and attempting to add some x86 drivers so that 32 bit workstations can print to it, Windows displays an error and asks for the "Windows Media (x86)". No CD or DVD I had contained what it was looking for... it turns out the solution was to install the printer with just 64 bit drivers, connect to it from a 32 bit windows 2008 server, and install the 32 bit drivers when prompted. After that, go to the properties of the network printer, click additional drivers, and check off the x86 box. It will upload the drivers to the 64 bit server for you. Detailed instructions below:

I created a queue for a new printer, installed the 64bit drivers and sent a test page. All was fine, so I decided to add the 32bit drivers so that some XP workstations would be able to print to it.

Each time I ticked the box, it would come up saying “Please provide path to Windows Media (x86 processor)” asking me for the ntprint.inf files from the i386 folder of the Windows media CD. Being 64bit Windows 2008, it doesn’t have an i386 folder on the media.

If you are having this issue with Windows 2008 64bit and HP drivers, here is the fix that I used below.

1. Go to a 32bit Windows Vista or 32bit Windows 2008 system and log on a user with admin rights on the print server.

2. Download the 32bit drivers you require for your printer and extract them.

3. Click on Start – run then type \\printserver (replace “printserver” with the name of your print server) and then double click on the printer, and install it. When prompted for drivers, point them to folder where you downloaded the drivers.

4. After it installs on the 32bit system, go into the local printers, right click on the network printer you just installed and choose properties, sharing, additional drivers and tick the x86 box. Click OK and the x86 drivers will upload to the server.

Credit for this one goes to: http://www.shnake.com/blog/?p=228

Removing Permissions in Exchange 2007 = Adding DENY permissons

Sometimes the most intuitive GUI based actions have strong underlying consequences. In this case, I was at a site that had an IT user whose mailbox suddenly became inaccessible. No other admins could get into it either. I began down the path of assuming this mailbox was corrupted... I disabled the mailbox thus removing it from the users account, and just out of interest, tried reattaching it to a test domain users account... it worked! The user was able to see and use the mailbox, no problem. I reattached it to the original admin... it stopped working.

Long story short, come to find, another domain admin had decided it would be more secure to go in and remove the 'domain admins' full permission to each of the admins at the site. This would only allow each admin to see their own mailbox and not eachothers. The problem with this was that Exchange 2007 implicitly adds a DENY permission whenever you remove an ALLOW permission. In this case, they removed the domain admins allow permission, so exchange 2007 added a DENY permission to the admins mailbox... thus blocking out all admins including himself.

Here is the command I used to discover this erroneous permissions entry:

get-mailboxpermission "username" | where{$_.Deny} | where{!$_.IsInherited}

This retrieves all deny permissions that are not inherited from the parent (mailstore). Low and Behold, Domain admins was listed with deny full access.

You will also notice this behavior if you look at the output of the Exchange GUI when you remove permissions... note there will be two successful commands displayed in the confirmation dialog... one to remove the permission, and one to add the deny permission. Goofy, no?

This command will remove that deny permission:

Remove-MailboxPermission -identity 'CN=User Name,CN=Users,DC=domain,DC=local' -User 'DOMAIN\Domain Admins' -Deny -InheritanceType 'All' -AccessRights 'FullAccess'

Citrix and Adobe Updater

Tired of all your users getting that message that they are not able to run the Adobe Updater every time they log into Citrix or the Citrix Desktop? Or even worse... tired of users updating your Adobe all the time on your Citrix servers? Here is a handy registry entry that will press the murder button on that Adobe Updater:

REG ADD "HKLM\SOFTWARE\Policies\Adobe\Acrobat Reader\8.0\FeatureLockdown" /v bUpdater /d 0 /t REG_DWORD /f

Note the 8.0 in that string... change it to 9.0 or whatever version you are running (best bet is to browse the registry for the specific string you need first). Don't forget to do a "change user /install" before and "change user /execute" after just to make sure.

---Update---

It seems that in some cases, that may not even be enough to stop the Adobe Updater. The final nail in the coffin, so to speak, after performing the above is to go into the C:\documents and settings\all users\ directory and hunt down Adobe_updater.exe, then rename it to .old. That should put this problem in the ground for good.

Ghost Network Cards after a P2V migration

I have learned many invaluable things in the past few days about P2Ving live production systems... but none so important as the following little gem. It seems that with most systems, when you P2V them you end up with Ghost network cards for each of your original physical NICs... that is, they do not show up in the device manager, yet they still use the static IP addresses you had assigned, do funny things to network communication and name resolution, etc. Here is how to get rid of them:

1) Open a command prompt and type in "SET DEVMGR_SHOW_NONPRESENT_DEVICES=1". You will not get a confirmation back, it will just drop back to the prompt. Now, in the SAME CMD window, without closing it...

2) Type in "DEVMGMT.MSC" to open the device manager from the command prompt. This is critical, if you dont open it from the same CMD window you typed the first step in, it wont work.

3) In device manager, go to View > Show hidden devices

4) Expand nic cards and right click on each fadded network card to remove it.

I tried following similar instructions over and over, and it was not until I launched the DEVMGMT.MSC from the command prompt that I set that first variable in that it worked.

Once I removed the ghost nic cards, set my static IP back to what it was supposed to be (as VMware converter gives the new vNIC a DHCP address) then rebooted, all was well.

The Six Simple Steps to a working ASA

Working with a Cisco ASA series can be daunting up front, especially when trying to configure it from the command line... but when you distill out all the fancy frills and extra features, there are really only six core components that need to be configured for a working firewall. Here they are:

Step1: Configure the internal interface vlan
------------------------------------------------------
ASA5505(config)# interface Vlan 1
ASA5505(config-if)# nameif inside
ASA5505(config-if)# security-level 100
ASA5505(config-if)# ip address 192.168.1.1 255.255.255.0
ASA5505(config-if)# no shut

Step 2: Configure the external interface vlan (connected to Internet)
-------------------------------------------------------------------------------------
ASA5505(config)# interface Vlan 2
ASA5505(config-if)# nameif outside
ASA5505(config-if)# security-level 0
ASA5505(config-if)# ip address 200.200.200.1 255.255.255.0
ASA5505(config-if)# no shut

Step 3: Assign Ethernet 0/0 to Vlan 2
-------------------------------------------------
ASA5505(config)# interface Ethernet0/0
ASA5505(config-if)# switchport access vlan 2
ASA5505(config-if)# no shut

Step 4: Enable the rest interfaces with no shut
--------------------------------------------------
ASA5505(config)# interface Ethernet0/1
ASA5505(config-if)# no shut
Do the same for Ethernet0/1 to 0/7.

Step 5: Configure PAT on the outside interface
-----------------------------------------------------
ASA5505(config)# global (outside) 1 interface
ASA5505(config)# nat (inside) 1 0.0.0.0 0.0.0.0

Step 6: Configure default route towards the ISP (assume default gateway is 200.200.200.2)
-----------------------------------------------------------------------------
ASA5505(config)# route outside 0.0.0.0 0.0.0.0 200.200.200.2 1

Cisco Etherchannel Configuration

I never seriously thought I would be blogging on how to configure Cisco devices, much less the advanced portions of the Cisco devices... but work has a funny way of making you learn things you never thought you would.

The site I am configuring has a HP blade enclosure with two Cisco Switches serving up the blades' primary and secondary ethernet connections. Right now, those switches have a single physical ethernet connection back to the core switch of the network (that way if one switch in the enclosure fails, the other takes over). We want to increase the bandwidth available between that blade center and the core switch because we are planning on filling up that blade center.

In addition, we have added a new core cisco switch as a fail over switch for total hardware redundancy up to the firewall. These two switches need to talk to eachother to pass traffic in the event that one goes down.

Enter Etherchannel, which aggregates single network connections on switches into a single logical connection with the combined bandwidth of both.

This is the site I used for configuring my switches: http://jklogic.net/cisco-ios-to-catos-etherchannel-configuration/

It gives a good understanding of what the tasks are at hand, but there are a few additional steps required to really get it going, and after confirming with a Cisco representative, this is what I ended up with for configuration:

Of the Cisco IOS flavor (cisco 3560G specifically, running 12.2):

...
!
interface port-channel1
switchport trunk encapsulation dot1q
switchport mode trunk
!
...
!
interface GigabitEthernet0/47
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 1 mode on
!
...

And of the CatOS variety (specifically the Cisco 2948G-GE-TX) we get:

...
!
#port channel
set port channel 2/47-48 20 (note that the 20 is an ID and is optional)
!
...
!
set trunk 2/47 nonegotiate dot1q (extra stuff about vlans)
set trunk 2/48 nonegotiate dot1q (extra stuff about vlans)
set port channel 2/47-48 mode on
end

At this point, you can connect the physical cables and you should have a working etherchannel. It looks deceptively easy written down... but this took like 6 hours to work out on the phone with Cisco.

For connecting IOS to IOS, just do the first IOS part on both devices... for CatOS to CatOS... you get the point.

When opening Microsoft Excel 2003 files in Microsoft Office Excel 2007 without having Excel 2007 already running you may find that files that a long time to open (20-60 seconds) while Excel 2007 or even Microsoft Office Word 2007, PowerPoint 2007 files take only a portion of that time to load.

The solution to this problem is easy and does not require any additional software other than the standard Microsoft Windows Explorer.

Open Windows Explorer (explorer.exe)
Select Tools | Folder Options from the menu
Go to the File Types tab
Select list item XLS (Microsoft Office Excel 97-2003 Worksheet) and click on Advanced.
If the "Appl ication used to perform action:" field the default value should be "C:\Program Files\Microsoft Office\Office12\EXCEL.EXE" /e" . Change this to "C:\Program Files\Microsoft Office\Office12\EXCEL.EXE" /e "%1" . Remember to enclose the full excel.exe path and %1 in parenthesis. Since Windows 95 this is a way to escape spaces in file names; %1 will return the file name of that file that is being opened by double-clicking.You can uncheck "U se DDE" but Microsoft Office Excel 2007 will bring it back to what you see above when you next open it.

Just confirm (hit "OK" ) on all the Microsoft Windows Explorer dialogs and next time you open a Excel 2003 file in Excel 2007 without having it already running you won’t have to wait! Problem solved

Ripped directly from m3rlin.org: http://www.m3rlin.org/wordpress/slow-opening-excel-2003-files-in-excel-2007/

NTLM Authentication using Outlook Anywhere on Exchange 2007

Here's a little gem I found in a response post to an article I was reading which worked for me immediately in getting NTLM authentication to work on Exchange 2007 / Outlook Anywhere / Outlook 2007 / Windows Server 2008:

1. In Exchange Management Console, go to Server Configuration > Client Access, double-click the server name, go to Outlook Anywhere (last tab), switch from Basic to NTLM.
2. in IIS, expand your server, go to Sites > Default Web Site > RPC, then in the “Features” page double-click Authentication, right-click Windows Authentication and choose Enable.

Exchange 2007 on Windows 2008 Bug With IPv6

Ok, whoever let this out of the door at Microsoft without resolving it is fired. Apparently, as described in this article: http://blog.aaronmarks.com/?p=65 there is a problem with running exchange and outlook anywhere on windows server 2008 such that if IPv6 is enabled in any form, you cannot connect. The nitty-gritty details come down to the server not listening on port 6004 for some reason when IPv6 is enabled... you can try it yourself by running "Telnet localhost 6004" in the command propmpt... no response.

Long story short, enabling the registry entry detailed in that article, unchecking ipv6 on the adapter, and removing the ::1 localhost entry from the hosts file then rebooting cleared the issue right up!

(For those wondering, the issue I was having was that outlook web access was working fine, and outlook was connecting just fine inside the network, but when i tried to use outlook anywhere with outlook 2007 outside the network I would persistently get an error indicating that outlook had to be online in order to connect.)

Configuring Exchange 2007 for external access to the Autodiscover service

You would like to think that going out and buying that expensive UCC Certificate then installing it properly into Exchange (as described in my previous post) would be enough, wouldn't you? Far from it. There are several follow-up steps required to properly configure Exchange 2007 (especially its Autodiscover service). Microsoft outlines a few of the commands here:

To configure the Outlook Anywhere service, run this in the EMS:

Enable-OutlookAnywhere -Server CAS01 -ExternalHostname "mail.contoso.com" -ExternalAuthenticationMethod "Basic" -SSLOffloading:$False

To configure the Offline Address Book service, run this in the EMS:

Set-OABVirtualDirectory -identity "CAS01\OAB (Default Web Site)" -externalurl https://mail.contoso.com/OAB -RequireSSL:$true

To configure the Exchange web services, run this in the EMS:

Set-WebServicesVirtualDirectory -identity "CAS01\EWS (Default Web Site)" -externalurl https://mail.contoso.com/EWS/Exchange.asmx -BasicAuthentication:$True

If using Unified Messaging, run this in the EMS:

Set-UMVirtualDirectory -identity "CAS01\UnifiedMessaging (Default Web Site)" -externalurl https://mail.contoso.com/UnifiedMessaging/Service.asmx -BasicAuthentication:$True

Of course, don't forget to change the contoso stuff to your own domain. Many of these can be configured in the EMC if you are scared of command line, except for EWS... that one can only be found in the EMS. Finally, add an A record for Autodiscover and point it to the external IP of your exchange server (the same IP that mail.domain.com is pointing to) and Autodiscover should be set.

Exchange 2007 and SSL

Installing a SSL certificate on an Exchange 2007 server is about the most unnessecarily complicated task I have come across to date. Unlike 2003, you have several names that this certificate needs to respond to, called SANs or 'Subject Alternative Names' such as autodiscover, mail (or your external fqdn), your internal dns name, and your internal netbios name. This certificate is so conveluted that most SSL certificate companies have a special process JUST for creating Exchange2007 certificates (and a REALLY special price to go with it...)

This site makes the process a little less painful by explaining in clear english step by step what needs to be done to generate the request from Exchange which you then submit to your CA (or external CA such as Entrust) so that you can get your .cer and move on.

http://exchangeninjas.com/cascert

Microsoft even links to that site, as well as other useful sources (such as vendors that provide these special Exchange certificates) in this site:

http://support.microsoft.com/?id=929395

-----------UPDATE---------------

Just another site that has proved useful to me in configuring an Exchange 2007 server with a UCC (multi SAN certificate):

http://www.sembee.co.uk/archive/2008/05/30/78.aspx

-----------UPDATE---------------

After doing some reasearch and consulting with some of my fellow techs, I found an article that describes how you can actually implement an Exchange 07 server (in a supported manner) that needs only a single name SSL certificate. The site is here:

http://www.amset.info/exchange/singlenamessl.asp

The major limitations of going this way are that you cannot use UC in Exchange 07 if you use this method, and your external DNS host MUST support SRV records (not a lot of them do...).

Backup Exec agents fail to start

It is unlikely that anyone will run into the same issue I have been facing over the past few days, but I figured I should post my eventual solution to this nightmare.

The starting symptoms of the issue I had were that the backup exec agent would not start... the service would sit at starting for a minute or two, then fail saying that the service did not respond in a timely fashion... or some generic message like that. Pushing out the remote agents again, trying different versions, manually cleaning the registry... none of it worked.

The problem started with the Mariofev virus that was discovered on almost all the workstations and servers at this site. The behavior of the virus is to copy itself to any network shares that it can access, then create an autorun.inf file in the directory such that every time a user opens that directory the virus autoruns and copies itself into the unknowing users registry. The virus then appends itself to any executables it can reach in a manner of ways, including attaching itself to outgoing emails. With some virus updates and installations of Spybot, the worst of the active virus was caught and stopped from further spread.

In addition, a Trojan W32.AgentMon found its way onto several systems, particularly the ones having problems with the backup agent. The action of this Trojan is unknown, but it modifies the tcp stack with some sort of hook. This was overlooked by all the antivirus and antispyware systems installed. It was discovered that a single file named vapor.dll exists in the C:\windows\system32 directory of infected computers with a date/time stamp of the moment the Trojan activated. This dll deeply entwines itself with the winsock/tcp functions of windows causing havoc due to being poorly written (I am not sure that it actually functions, but it messes stuff up).

While working with Symantec support on the backup agent installation issues, I decided to grab a copy of process monitor from sysinternals and track all calls etc from beremote.exe (the backup exec remote agent)… and low-and-behold when beremote.exe (and soon come to find any network application) tried to establish a port it was directed to call vapor.dll first… which promptly crashed. I renamed vapor.dll to vapor.dll.old and suddenly no dns worked. Returning the file to its original name re-enabled dns. It was clear at this point that the tcp stack had been hijacked. I began thinking that it may be a good idea to try and rebuild tcp/ip on the servers to see if that fixes the issue.

Following MS article 811259, I rebuilt Winsock and reinstalled tcp/ip on the network connection, rebooting after each. After rebooting, beremote.exe started up just fine and the system could get out to the internet while vapor.dll was still renamed. Examining process monitor, no calls are made to vapor.dll anymore.

I was then able to push out the backup exec 12 agent via the normal method and start the backups running. Backup of the two systems is being performed as I write this.

Vista / Server 2008 Volume License Activation

Here's a strange one I ran across with a remarkably simple solution. When you initially install Vista or Server 2008 with a volume license key (or MAK/KMS key) then check on your activation, it will indicate that activation will occurr in a few days. When you click to activate now, it throws some kind of 'DNS name not found' error. The simple solution is to go back and click 'change product key' then re-enter the same exact key you used to install. Windows will activate fine and you are on your way!

Prerequisites for installing Exchange 2007 on Windows Server 2008

Installing Exchange 2007 and its prerequisites is a piece of cake on Windows Server 2003... install iis, add com+ and .net framework, and you are good to go. Not so much with Windows Server 2008... there are more requirements than I could count or care to remember anyway. Luckily those lovely people over at msexchangeteam.com have thrown together a zip of xml files which can be used to do all the work for you.

Here is where the strength of the new windows powershell comes in. The zip of files can be found at http://msexchangeteam.com/files/12/attachments/entry448276.aspx. Unzip all the xml files to the C: drive root, then open up the command prompt and run "ServerManagerCmd -ip xmlfilename.xml" for each xml file that matches a rule set you need. You will likely have to reboot after each one, but the powershell will tell you when its needed.

Credit goes to: http://www.petri.co.il/installing-exchange-2007-prerequisites-on-windows-server-2008.htm

How to reset the iLO password on an HP system

Typically new HP systems come with an iLO port that allows you to access the console output of the machine, even when it is booting, crashed etc in a RDP-style fashion through any browser by going to the IP address assigned to that iLO card (its DHCP by default). Nowadays, this iLO web management portal has a factory-assigned username and password which are printed on a nifty little plastic card tucked into the front of the server's bezel.

A good administrator will realize the immediate value of this tool, and the extreme threat it provides were an unscrupulous user to decide they wanted to gain access, and promptly change the password. Then they will write it down somewhere and not lose it. The former admin of a site I am administering failed it big time.

You can reset the password by cracking the case and flipping a dip switch, but who wants to down a server and pull it apart for that? Not me. Here is another way to change the password from the windows based operating system installed on that hardware.

HP made a tool called the HP Lights-Out Online Configuration utility (HPONCFG for short) that will run scripts against your host's ILO system to make changes without requiring a reboot. Finding the tool on HP's website is damn near impossible, but i found it by searching for 'cp008120.exe' on google. You need to have some prereqs installed, but if the HP installation CD's were used you may already have the prereqs (and this tool) installed. You can tell by looking under Add/Remove programs for 'HP Lights-Out Online Configuration Utility' or by checking C:\Program Files\HP\hponcfg to see if that folder exists.

Next you are going to need an admin script for resetting the password. I got it from this site: http://blog.netnerds.net/2006/04/ribcl-reset-administrator-password-on-ilo/ which also details the prereqs you need in the event that the HP installer cd was not used to setup the system. Either that or you can make your own XML file by opening up a notepad and typing in

<ribcl version="2.0">
<login password="boguspassword" user_login="Administrator">
<user_info mode="write">
<mod_user user_login="Administrator">
<password value="newpass">
</mod_user>
</user_info>
</login>
</ribcl>

Be sure to change "newpass" to your desired password in up above next to password value... then save it as something like resetpassword.xml in the same directory as your hponcfg.exe file. Now go to a command prompt and type in:

HPONCFG.exe /f resetpassword.xml /l log.txt > output.txt

and review the output files. Should work like a charm. All credit goes to that link i posted up above for this one... just logging it with the steps i had to follow for my own sake...

Windows Installer CleanUp Utility

Another tool that I have known about for some time but figured I should write about before I forget: The Windows Installer CleanUp Utility. Its a great little tool that will give you a list of all the programs installed on your machine and allow you to rip out their registry keys. Great for stingy antivirus or office installations that just don't want to uninstall. Run this tool, remove it, and reinstall on top of it... then uninstall it for reall. Voila!

http://support.microsoft.com/kb/290301

Uninstalling Symantec (or any program) remotely

Found a nifty tool (that I knew about all along) in PSTools from Sysinternals called PSExec. It lets you run processes on remote computers. I namely found this useful in uninstalling Symantec Antivirus Corporate from over 100 workstations that needed McAfee installed on them. You can get PSTools from a simple search on Microsoft's website.

The first step is to find the uninstall string. In the registry of a computer with the app you want to uninstall, browse to HKLM\Software\Microsoft\Windows\CurrentVersion\Uninstall and browse through the list of applications till you find the one you want. Then look for the UninstallString value (or something similar). It likely has a large registry key associated with it. For example, Here is the one for Symantec AV CE 10.x:

{5A633EDO-E5D7-4D65-AB8D-53ED43510284}

Now just run the following command to uninstall it:

psexec.exe \\remotecomputername -u Domain\administrator -p Password MSIExec.exe /norestart /q /x{5A633EDO-E5D7-4D65-AB8D-53ED43510284}

As a side note... it is also worth mentioning that if policies are in place to require a password to uninstall Symantec AV CE... the following string will remove the password requirements so that the above string will work:

psexec.exe \\remotecomputername -u Domain\administrator -p Password regedit /s \\server\pathtoregfile.reg

And the contents of that regfile are (verbatim):
-------------------------------------------------------------------------
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\INTEL\LANDesk\VirusProtect6\CurrentVersion\AdministratorOnly\Security]
"LockUnloadServices"=dword:00000000
"UseVPUninstallPassword"=dword:00000000
-------------------------------------------------------------------------

These changes essentially set the above keys to 0 instead of 1, so the password is no longer required.

You can even stitch the two psexec commands together into a bat... then supply psexec with a list of computers in a text file and let-er-rip.

Hidden Symantec Tools Site

Found a great hidden sub-site at Symantec.com that hosts all their tools that you usually have to call them to get a copy of.... like SymNRT, NoNAV, etc.

http://service1.symantec.com/SUPPORT/tsgeninfo.nsf/docid/2005103109480139

MSDTC 53258 error after dcpromo

Run the Component Services MMC snap-in: Start -> Administrative Tools ->
Component Services

Browse to: Console Root -> Component Services -> Computers -> My Computer

In Component Services, right click on My Computer, and click "Properties".

Select the "MSDTC" tab. On the bottom of the tab page, click the "Security
Configuration" button. A new dialog will open. Just click "OK" to close the
dialog (I believe this may set the MSDTC defaults). Then click "OK" to close
the first dialog and try restarting the MSDTC service and see if the warnings
still occurr.

Then as a test, restart MSDTC (net stop msdtc && net start msdtc) and check
the application event log. You should not see any warnings as before.