Author Archives: Geoff

About Geoff

Sr. System Administrator at the University of Vermont

One-liner: duplicate a folder collection, without files

New fiscal year; new set of empty folders, but with the same structure and permissions as the previous year? Robocopy to the rescue:

robocopy "FY 2015" "FY 2016" /e /xf * /COPY:DATSO /log:c:\temp\new-year-folders.log /tee

/e = copy all subdirectories, even empty ones.
/xf * = Exclude filenames matching *, i.e., all of them
/COPY:DATSO = what to copy: Data, Attributes, Timestamp, Security, and Owner.

I like to log things, so I include that, too. If you’re really cautious, you could do a dry run with the /L switch, which makes robocopy just log what it would do, but not actually perform any actions. Kind of like the PowerShell -whatif switch.

Robocopy file classes

This information comes from the Robocopy.exe documentation PDF file for Windows XP version, but it’s the best description I’ve been able to find. From page 15 of that document:

Using Robocopy File Classes

For each directory processed, Robocopy constructs a list of files in both the source
and destination directories. This list matches the files specified on the command line
for copying.

Robocopy then cross-references the lists, determining where files exist and comparing
file times and sizes. The program places each selected file in one of the following
classes.

File Class In source In destination Source/Dest file times Source/dest file sizes Source/dest attributes
Lonely Yes No n/a n/a n/a
Tweaked Yes Yes Equal Equal Different
Same Yes Yes Equal Equal Equal
Changed Yes Yes Equal Different n/a
Newer Yes Yes Source > Destination n/a n/a
Older Yes Yes Source < Destination n/a n/a
Extra No Yes n/a n/a n/a
Mismatched Yes (file) Yes (directory) n/a n/a n/a

By default, Changed, Newer, and Older files are candidates for copying (subject to
further filtering, as described later). Same files are not copied. Extra and Mismatched
files and directories are only reported in the output log.

Normally, Tweaked files are neither identified nor copied – they are usually identified
as Same files by default. Only when /IT is used will the distinction between Same and
Tweaked files be made, and only then will Tweaked files be copied.

Readable System Event logs

I think I’m not alone in finding the Service Control Manager logs so many informational events as to make it hard to read the important events in the System Event logs on modern Windows systems.

I’ve used custom XPath queries of Event logs before, and decided to define a Custom view of the System event log that suppresses the events generated by the Service Control Manager that are in the Informational or Verbose catergories. Here’s the XML that defines this custom view:


<QueryList>
 <Query Id="0" Path="System">
 <Select Path="System">*</Select>
 <Suppress Path="System">*[System[Provider[@Name='Service Control Manager']
 and (Level=4 or Level=0 or Level=5)]]</Suppress>
 </Query>
</QueryList>

References:

Renaming directories with invalid names

Somehow, a client managed to create several directories with names that ended with a period. However, File Explorer and other tools (i.e., backup) are unable to access the folder contents, getting an error that usually is interpreted as “The system cannot find the file specified.”

According to KB2829981, the Win32_API is supposed to remove trailing space and period characters. KB320081 has some helpful suggestions, and also indicates that some techniques allow programs to bypass the filename validation checks, and some POSIX tools are not subject to these checks.

I found that I was able to delete these problem folders by using rmdir /q /s “\\?\J:\path\to\bad\folder.” But I wanted to rename the folders in order to preserve any content. After flailing about for a while, including attempts to modify the folders using a MacOS Client and a third-party SSH service on the host, I was prodded by my colleague Greg to look at Robocopy.

In the end, my solution was this:

  1. I enabled 8dot3 file name creation on a separate recovery volume (I didn’t want to do so on the multi-terabyte source volume)
  2. Using robocopy, I duplicated the parent folder containing the invalid folder names to the recovery volume, resulting in the creation of 8dot3 names for all the folders
  3. I listed the 8dot3 names of the problem folders with dir /x
  4. The rename command with the short name as a source and a valid new name

This fixed the folders, and let me access their contents. I then deleted the invalid folders from the source and copied the renamed folders into place.

It seems like a simple process, but I managed to waste most of a morning figuring this out. Hopefully, this may save someone else some time.

Troubleshooting Offline Files

My previous post describes the normal operation of Offline Files. And most of the time, “it just works.” But there are times when it won’t, and getting it running again can be challenging.

Two Important concepts

First, it’s important to understand that the Offline Files facility is providing a virtual view of the network folder to which Documents has been redirected when Windows detects that the network folder is unavailable. This means that, when Offline Files is really borked, users can see different things in their Documents folder depending one whether their computers are online or offline.

Second, Windows treats different names for the same actual server as if they are different servers altogether. Specifically, Windows will only provide the Offline Files virtual view for the path to the target network folder. You can see the target folder path in the Properties of the Documents folder.

The Location tab shows the UNC path to the target network folder.

The Location tab shows the UNC path to the target network folder.

For example, these two UNC paths resolve to the same network folder:

\\files.uvm.edu\rallycat\MyDocs
\\winfiles1.campus.ad.uvm.edu\rallycat\MyDocs

If the second path is the one that is shown in the Location tab in the properties of the Documents folder, then you will be able to access that path while offline, but not the first path.

Show me the logs

There are event logs that can be examined. I’ll mention them, but I’ve rarely found them helpful in solving a persistent problem. If you want to get the client up and running again ASAP, skip ahead to the Fix it section.

There are some logging options available that can help in diagnosing problems with offline files. There are two logs that are normally visible in the Windows Event Viewer, under the Applications and Services logs heading:

  • Microsoft-Windows-Folder Redirection/Operational
  • Microsoft-Windows-OfflineFiles/Operational

Continue reading

Folder Redirection and Offline Files

The following information is not new. We are in the process of making changes to our Folder Redirection policy, though, and I thought it might be helpful to have this baseline information in a place that is handy for referral.

Background

Offline Files is a feature of Windows that was introduced in parallel with Folder Redirection in Windows 2000. Folder Redirection allows an administrator to relocate some of the user profile data folders to a network folder, which has the advantage of protecting that data from loss due to workstation issues like drive failure, malware infection, or theft. It also means you can access your data from multiple workstations.

The Offline Files facility provides a local cache of the redirected folder(s) so that mobile users can continue to work with the data in those folders when disconnected from the organization’s network. When the computer is connected to the network again, any changes to either the network folder or the local Offline Files cache are synchronized. Users are prompted to resolve any conflicting changes, e.g., the same file was modified in both places, or was deleted from one and modified in the other.

Continue reading

PowerShell Script: New-RandomString.ps1

I need to automate the setting of passwords on some Active Directory accounts. Since resetting passwords is also a task that I’m asked to perform with some routine, I decided to make a more generic tool script that could be used in a variety of tasks ( I listened to Don Jones‘ advice on building Tools and Controllers).

I also got a head start from Bill Stewart’s useful Windows IT Pro article Generating Random Passwords in PowerShell.  Among the changes I made are source character class handling, and a new SecureString output option. Please let me know if you find the script useful, or if you find any bugs.

<#
.SYNOPSIS
Generates one or more randomized strings containing specified
character classes.

.PARAMETER Length
The length of the string to be generated.

.PARAMETER CharacterClasses
An array of Character Classes from which to generate the string. The string
will contain at least one character from each specificied class. You may also use the alias 'Classes' for the parameter name

Valid Character classes are:

    Upper    - A..Z
    Lower    - a..z
    Digits   - 0..9
    AlphaNum - shorthand for Upper,Lower,Digits
    Symbols  - !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
    Safe     - #$%+-./:=\_~  (ODBC Safe, Shell Safe if quoted)

If no classes are specified, a string is generated with mixed-case letters,
digits, and symbol characters (i.e., ALL the classes).

.PARAMETER IncludeCharacters
A string of characters to include in the generated string:

.PARAMETER ExcludeCharacters
A string a characters to exclude in the generated string:

.PARAMETER Count
The number of strings to be generated.

.PARAMETER AsSecureString
Specifies that the new random string(s) will be returned as Secure String
objects, to make their use as passwords easier.

.EXAMPLE
> New-RandomString.ps1 -CharacterClasses Lower,Digits -Length 14 -Count 5
Generated five strings, each fourteen characters long, comprised of lowercase
letters and digits.

.EXAMPLE
> New-RandomString.ps1 -Classes AlphaNum,Symbols -length 21
> New-RandomString.ps1 -length 21

The previous two commands are equivalent, because the default character classes
used are upper and lowercase letters, digits, and symbol characters.

.EXAMPLE
> New-RandomString.ps1 -Class 'AlphaNum' -Include '#$%^'

The generated string will contain characters from the UpperCase, LowerCase
and Digits classes, as well as at least one character from among the four
specified.

.EXAMPLE
> New-RandomString.ps1 -Class 'AlphaNum' -Exclude 'O0l1'

The generated string will contain characters from the UpperCase, LowerCase
and Digits classes, but will not contain the "look-alike' characters.

.Notes
Author     : Geoff Duke <Geoffrey.Duke@uvm.edu>
Last Edit  : 2014-11-07 

Based on script "Get-RandomString.ps1" for Windows IT Pro:
http://windowsitpro.com/powershell/generating-random-passwords-powershell

#>

#Requires -version 3

[cmdletbinding()]
Param(
        [alias('Size')]
        [ValidateRange(7,256)]
        [int]
        $length = 21,

        [int]
        $count = 1,

        [Parameter()]
        [ValidateSet('Upper','Lower','Digits','AlphaNum','Symbols','Safe')]
        [alias('Classes')]
        [String[]]
        $CharacterClasses = @('Upper','Lower','Digits','Symbols'),

        [Parameter()]
        [string]
        $IncludeCharacters = '',

        [string]
        $ExcludeCharacters = '',

        [switch]
        $AsSecureString

     )

Set-StrictMode -version 'Latest'

# Additional parameter wrangling
# --------------------------------------------------------------------
[string[]] $Classes = $CharacterClasses.ToLower()

if ( $Classes.Contains('safe') -and $Classes.Contains('symbols') ) {
    write-warning 'You specified both "Symbols" and "Safe" character classes; this is the same as just specifying "Symbols".'
    $Classes = $Classes | where { $_ -ne 'safe' }
}

# Replace alphanum with the upper,lower, and digits classes
if ( $Classes.Contains('alphanum') ) {
    $Classes = $Classes | where { $_ -ne 'alphanum' }
    $Classes += 'upper','lower','digits'
}

# remove any duplicated classes
$Classes = $Classes | select -unique

# Setup source characters
# -------------------------------------------------------------------- 

# Character classes - functionally, a strongly-typed hash of string arrays
#       (addresses issue of singleton arrays turning into simple strings)
$chars = New-Object 'Collections.Generic.Dictionary[string,char[]]'

$chars['lower']    =  97..122 | foreach-object { [Char] $_ }
$chars['upper']    =  65..90  | foreach-object { [Char] $_ }
$chars['digits']   =  48..57  | foreach-object { [Char] $_ }
$chars['symbols']  = (33..47+58..64+91..96+123..126) | foreach-object { [Char] $_ }
$chars['safe']     = '#$%+-./:=\_~'.ToCharArray()

write-verbose $( 'String must include a character from each of ' +
              $( $Classes -join ',' ) +
              $( if ( $IncludeCharacters ) { " plus [$IncludeCharacters] " } ) +
              $( if ( $ExcludeCharacters ) {
                  "but must not include any of [$ExcludeCharacters]" } ) )

if ( $IncludeCharacters ) {
    $Classes += 'include'
    $chars['include'] = $IncludeCharacters.ToCharArray()
}

[char[]] $char_source  = $chars[ $Classes ] | % { $_ } | select -unique

if ( $ExcludeCharacters ) {
    $char_source = $char_source | Where { $_ -NotIn $ExcludeCharacters.ToCharArray() }
}

write-verbose "Source chars: $(-join $char_source)"

# Generating the random string(s)
# --------------------------------------------------------------------
$string_count = 0
:NewString while ( $string_count -lt $Count )  {

    $output = ''
    for ( $i=0; $i -lt $length; $i++) {
        $output += get-random @($char_source)
    }
    write-debug "NewString: generated string is -> $output"

    # Ensure that the requested character classes are present
    :CharClass foreach ($class in $Classes) {
        foreach ( $char in $output.ToCharArray() ) {
            if ( $chars[$class] -Ccontains $char ) {
                write-debug "CharClass: '$char' is in $class"
                continue CharClass # check the next character class
            }
        } # end foreach $char, didn't match the current character class
        write-debug "CharClass: No character from $class! Start again"
        continue NewString # Need to generate a new string
    } # end foreach #class

    # string matches required character classes"
    $string_count++

    if ( $AsSecureString ) {
        ConvertTo-SecureString $output -AsPlainText -Force
    }
    else {
        $output
    }
} # end while

It was while I was writing this script that I ran into the Loop Label documentation error. In PowerShell, as in Perl, Loop Labels do not include the colon when used with a break or continue statement.

PowerShell documentation error – loop labels

I’ve been banging my head on a problem with a script I’m writing. I want to stop executing an inner loop and resume with the next iteration of an outer loop. In Perl, I’d use a next statement with a loop label. In PowerShell, the analogous statement is continue, and loop labels are supported, as described in the about_Break help document.

I finally wrote simplified test code, following the documentation carefully. However, the documentation is wrong. It indicates that the break or continue statement should include the colon in the loop label. This doesn’t throw an error, but it executes as though the label isn’t present at all. The code below includes the colon.

$VerbosePreference = 'Continue'

write-warning 'There should be no output; the outer loop should be exited during first iteration'
:outer foreach ($a in ('red','green') ) {
    write-verbose "Outer loop"

    :inner foreach ($b in ('red','blue','green') ) {
        write-verbose "Inner loop"
 
        write-verbose "`$a is $a ; `$b is $b"
        if ( $a -eq $b ) {
            break :outer
        }
        "$a $b"
    }
}

Then cracked my copy of PowerShell in Action and saw that the loop label does not include the colon, just like Perl. Remove the colon and everything is good. Wish it hadn’t taken me hours to work it out.