Tag Archives: Troubleshooting

GUID Chase – Group Policy troubleshooting

It started with an alert from System Center Operations Manager about a failed scheduled task. Of course, the alert references a task name that looks like a SID.  Running schtasks /query show a few jobs with a status that warranted inspection. Looking at the Microsoft-Windows-TaskScheduler/Operational log I found that the task “\Microsoft\Windows\CertificateServicesClient\UserTask” is the one the failed and triggered the alert.

I also noted that there were some Group Policy processing errors occurring at about the same time as the task failure, including a problem applying the Group Policy Scheduled Tasks settings. And the failing task starts at user login.

Next, I ran gpresult /h to create a report of the GPOs and settings that applied, and any errors that were generated. The report confirmed that there were failures in applying the Group Policy Files settings and the Group Policy Scheduled Tasks settings.

Some web searching turned up this thread, among others, which pointed me to the Group Policy History files in C:\Users\All Users\Microsoft\Group Policy\History. This directory contained four subdirectories named with the GUIDs for the corresponding GPOs. I was able to find three of the four GPOs by inspecting the details in the GPMC, but I couldn’t find the fourth.

I decided to search more programmatically, and started with an LDAP search with ADFind:

adfind -f "&(objectClass=groupPolicyContainer)(Name={DC257675-89C1-5AA6-5F65-B5D5CFC35E17})"
0 Objects returned

Then, just to be sure, I used the PowerShell GroupPolicy module:

PS Z:\> import-module GroupPolicy
PS Z:\> get-gpo -guid "{DC257675-89C1-5AA6-5F65-B5D5CFC35E17}"
Get-GPO : A GPO with ID {DC257675-89C1-5AA6-5F65-B5D5CFC35E17} was not found in the campus.ad.uvm.edu domain.

So I removed the subdirectory with that name from the GP History directory, and retried gpupdate /force. This time, it completed successfully.

String arrays and mandatory parameters

I have been working on a function to convert the output of NET SHARE <sharename> commands into usable PowerShell objects. In the course of my work, I was storing the output of the command in a variable, which I later pass into a parsing function. Curiously, the function I developed iteratively in the console worked fine, but when I dressed it up in my script, it failed:

test-array : Cannot bind argument to parameter 'foo' because it is an empty string.
At line:1 char:12
+ test-array $party
+            ~~~~~~
    + CategoryInfo          : InvalidData: (:) [test-array], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationErrorEmptyStringNotAllowed,test-array

I had verified that the variable was of type System.Array, and that it had string elements. After banging my head on it for a while, I decided to break out the parameter handling and test it separately. I wrote a quick function to accept and process the elements of a string array:

function test-array {
param( [string[]] $foo )
    $i = 0
    foreach ( $line in $foo ) {
        write "[$i] $line"
        $i++
    }
}

Continue reading

VSS diagnostics

For the past eight month, I’ve been working with EMC and Microsoft to diagnose a problem. Several time a month, during the backup of our primary Windows 2008 R2 file server, all the VSS shadow copies get deleted for the volume containing all our shared departmental directories.

This has two major effects. First, it means that our clients no longer can recover files using the Previous Versions feature of Windows. Second, it casts significant doubt on the validity of the backups performed at that time, which EMC NetWorker reports as having completed successfully.

We have been unable to find a technical solution to the shadow copy loss, so we will be reconfiguring our storage and shared directories to accommodate the limitations of NetWorker. In the meantime, I want to note a few of resources that have been helpful in diagnosing problems with VSS (it will be easier to find them here than in my pile o’ email):

Volume Shadow Copy Service (TechNet)

Volume Shadow Copy Service (MSDN)

Registry Keys and Values for Backup and Restore

How to enable the Volume Shadow Copy service’s debug tracing features in Microsoft Windows Server 2003 and Windows 2008

Using Tracing Tools with VSS

Edit a meeting you can’t see

The situation:

I’m working in Oracle Calendar as a person’s designate, managing his calendar on his behalf. We’ll call him Sam. I create a meeting for Sam with some other attendees. Later, I remove Sam from the meeting rather than deleting it. Perhaps the other still want to meet but didn’t want to create a new meeting. Later still, those folks decide they want to reschedule the meeting.

The problem:

If a person isn’t listed as an attendee, then that meeting doesn’t appear in their calendar. However, in Oracle Calendar, only the person who created the meeting can edit it or delete it. This person is listed in the details of the meeting as Proposed by.

So Sam owns the meeting, but it isn’t displayed on his agenda for me to manage it. How do I edit or delete a meeting I can’t see?

The solution:

I need the In-tray Window in Oracle Calendar. This window is something that most people ignore or disable, but it will display the calendar entries you’ve sent out, including once that you aren’t attending. In addition, if I’ve been granted rights to work as someone’s designate, there’s a folder for their entries in my In-tray as well.

In this screenshot, I’m looking at a meeting that I created as Sam’s designate and from which I then removed his as an attendee. If the meeting isn’t recent, you may need to adjust the display options (Tools – Options – In-tray – Sent out) to allow you see the particular event.

OracleCalendar-InTray-AsDesignate

Another work-around would be to have Sam open the calendar of one of the attendees, find the meeting and edit or delete it. But since I can get to it via the In-tray, I don’t need to bother Sam at all.

I hope this is helpful.

Microsoft Office Troubleshooting

Recently, I was asked to talk with our Help Line staff about strategies for troubleshooting problems with Microsoft Office. I spent some time addressing the activation issues relating specifically Office 2010, which I wrote up in a separate post.

The most important point I want to make about general Office troubleshooting is that reinstalling office will rarely fix a problem. Office will kick-off a repair operation automatically if it detects problems with core Office files. Application, heal thyself.

More importantly, a repair operation or uninstall/reinstall process will refresh Office program components, but it won’t touch templates, user and system specific registry information, and add-ins that are the most frequent cause of problems.

Safe mode

The first step in troubleshooting should be to start the application in safe mode. Most versions of Office applications support a safe mode, which doesn’t load templates, registry info, and add-ins. This step quickly determines whether the problem lies with Office itself or elsewhere.

Invoking Office safe mode is as easy as adding the command-line parameter /safe. Usually, I open the Run window (WindowsKey+R), and type the name of the office executable and add the /safe parameter. If you don’t know the executable name, you can find it with the browse button, and then add the parameter at the end:

office14-safemode-run

If the app doesn’t start, then you probably do need to perform a Repair installation. If the application starts successfully (sometimes without opening a document in safe mode), then you know that the core office files are fine, and a reinstall isn’t likely to help.

Continue reading

Troubleshooting Office 2010 & 2013 Activation

Microsoft Office 2010 and 2013 volume license editions use the Volume License 2.0 mechanism to manage activation. Office 2010 and 2013 will activate against our campus Key Management Service (KMS), without user intervention, in a manner similar to Windows Vista and Windows 7.

Occasionally, the activation process doesn’t work. Problems are usually related to network communication with the KMS. Below are some steps to identify and resolve problems that might occur during activation.

Gather Information.

Gathering data is essential to fixing problems. If you ask me (or other IT staff) for help with Office 2010 activation, the first thing I will ask from you is the output of the commands in the steps below.

There are a few steps that will make it easy to collect all the output of your troubleshooting steps.

  • Open an elevated Command Prompt (Run As Administrator)
  • Change the Properties of the command prompt window to increase the Screen Buffer
    height to, say, 3000 lines. This will prevent you losing earlier steps as the lines scroll off the screen.
  • Run cscript /h:cscript, which changes the default script host to cscript, so that output will go to the command prompt instead of a pop-up dialog box.

When you are ready to copy the text from the command prompt, right-click the title bar of the window, select Edit > Select All, and then Control-C to Copy the text to the clipboard. Then you can paste the text to any place you want; a webmail message, a footprint entry, or a text file in notepad.

Continue reading

Network Policy Service error – eventid 4402

I’ve been working on deploying a load-balanced Remote Desktop Gateway service. I deployed the first farm member, then cloned it to create a second member. The second member was throwing Error events, which has the description "There is no domain controller available for domain CAMPUS."

Now, I know that the domain controllers are up and available. I remembered having fixed this at some point with the Terminal Services Gateway box I set up originally.

Google pointed be to a technet blog entry describing the solution(s).

nps-error-fix

When I selected Register server in Active Directory, I received an error because the account I was using didn’t have rights to modify the the AD objects. And that explains why this system as having the problem: when I joined the cloned system to the domain, I was not using a domain admin account.

I logged back in as a domain admin and reran the registration step. Done, and blogged for my future reference.

Server 2008 R2 DNS client issues

We use BIND for our DNS, and allow certain systems to perform dynamic DNS registration. This arrangement has worked well for years. When I started deploying Server 2008 R2, I noticed that they weren’t registering PTR records.

At the same time, I noticed a bunch of errors that seemed to indicate that Dynamic DNS wasn’t working at all. It turns out this is a false error, due to the differently formatted, but still correct, success message returned by the BIND DNS. (see KB977158 for details)

After spending lots of time doing packet captures (thanks for your help, Sam!), I opened an issue with Microsoft. After collecting a few traces to analyze, they determined that the same differently formatted success message was responsible.

I installed the KB977158 hotfix, and now my Server 2008 R2 hosts are successfully registering their PTR records.

2008 R2 DCDIAG errors with NIC teaming

I’m in the process of deploying a couple new Server 2008 R2 domain controllers. I’m using two IBM blades, each having a pair of Broadcom NICs that I configured in fault-tolerance teams.

In trying to verify the configuration of one of the DCs, I used the command:

dcdiag /test:dns

The output surprised me:

Starting test: Connectivity 
    Message 0x621 not found. 
    Got error while checking LDAP and RPC connectivity. Please check your firewall settings. 
    ......................... CDC01 failed test Connectivity

I ran the command from a Server 2008 Sp2 (not R2) host:

dcdiag /s:cdc01 /test:dns

The test passed without error. Strange. I verified firewall and DNS. Then turned to the hivemind. This post shows similar behavior. This post on the TechNet forums identified the NIC Team as a probable source, and a contributor referenced a hotfix KB978387 for a bug in dcdiag on Server 2008 R2 on systems with NIC Teams.

Installed and now the test passes:

Starting test: Connectivity
   ......................... CDC01 passed test Connectivity

I spent much of my day working on this, and on tracking the connections to AD by clients using unsigned SASL binds or LDAP simple binds without an encrypted connection.

Capture Windows VM memory dump in ESX

I’m working with Microsoft to identify a problem I’m seeing with LSASS, possibly related to the VSS snapshot created by our backup software. At this point, I need to be able to capture the memory state on the system, even if I can’t log into the box.

There are several ways to trigger a crash in order to collect a memory dump, but this system is a guest running in VMWare VSphere (ESX4). I asked VMWare support, and they pointed me to KB article 1009187, Generating a Windows core dump from an ESX virtual machine.

I configured my test system guest to crash and collect a memory dump on an NMI event, then used the vmdumper command to send the NMI to the guest.

It worked like a champ:

vm-crash-nmi

I verified the integrity of the dump file with dumpchk. It looks good. I’m setting the same thing up on my production guest.