J. Greg's Brain Corral

Ever have a disaster with one of your servers?  No?  You lucky bastard…

Recently we had corruption of a number of our Virtual Machines (caused by a fault in the firmware of our “enterprise” storage system from a Nameless Mainstream Vendor, which was triggered by unexpected filesystem behavior from an Evil Mainstream Company’s virtualization platform).  This event forced us to exercise our system disaster recovery tools, (also from a Nameless Evil Mainstream Company) and brought us to the subsequent discovery that some backup products just don’t do DR.  There is much that could be said about that, but I will leave it there.

Anyway, we thought we would have a look at Microsoft DPM 2007 SP1 to see if the DR story there is any better.

Here are some sticking points I have hit while evaluating the product:

• Server Recovery Tool (SRT) – This is the Bare Metal Recovery component of DPM.  As it turns out, it only can protect Server 2003 and XP systems.  Server 2000 is a no go (no tears here…), as are Server 2008 and Server 2008 R2 (aargh!).  SRT is pretty easy to setup and configure, but keep in mind that it must run on a Server 2003 OS (not Server 2008!).
• DPM Reporting Services – The DPM installer creates a IIS instance on your machine, and configures/installs SQL 2005 with Reporting Services.  Very nice!  Unfortunately, the installer misses one critical IIS setting when installed on Server 2008:
  http://scdpm.blogspot.com/2009/07/reporting-does-not-work-with-dpm-2007.html
  You need edit the feature permissions on the the “HTTP Handler Mappings” feature on the Reporting Services IIS site to allow “Script” access (not script execution, just script access).  After that, you should be able to run reports from the DPM console.
• Updates – In addition to SP1, there are numerous hotfix rollup packages available, of which you should take advantage. 

• Here the the most current DPM update pack, as of the time of this writing:
  http://www.microsoft.com/downloads/details.aspx?familyid=14E1A04B-2323-4344-B737-A3194B9AB3ED&displaylang=en
  In my case, I need this to address errors in the WSSCmdletsWrapper that crop up when attempting to create a protection group for Windows SharePoint Services.  Keep in mind that you need to update the server and your DPM Agents.
• You may also need VSS updates:
  http://support.microsoft.com/kb/940349
  It appears that VSS updates are available only for Server 2003.  I guess Server 2008 is perfect and does not need VSS update

• You must configure your WSS VSS Agent to run as an account that has both local Administrator rights on the SharePoint WFE, and Farm Administrator rights.  The writer must be configured after installing the DPM agent and before attempting backup.  Use “ConfigureSharePoint.exe” in the DPM bin directory to make these changes.
• You also can configure backup of MOSS Search, which is accomplished using a different switch to the “ConfigureSharePoint.exe” tool.
• The VSS updates mentioned above are required before backing up a WSS farm.
• Server 2008 DR – Documentation on this really bites.  I sent the following feedback to the DPM whitepaper team for their document on Server 2008 Bare Metal recovery (How to do Bare Metal Recovery of WS08 with DPM 2007 SP1):

There are a few points in the white paper that require some clarification as they will confuse most readers.  I also have a few questions about the reasoning behind some of the steps in the document.

1. In the section "Before you create the protection group" under "Configuring Backups for BMR" it is unclear on which system you should be performing these steps.  A seasoned DPM admin will be able to figure out that these steps need to be performed on every system which will be backed up, but a newbie will not know this.  The instructions should be more explicit. 
2. The instructions have us create a share on the local server rather than simply backing up the WSB image to a named volume.  Why?  It would seem that backing up to a local share adds unnecessary complexity to this operation.  Using a local volume will be simpler and more secure. 
3. Similarly, in the recovery instructions we are told to restore the system image to a local share.  Why?  In a bare metal recovery scenario, there is no local share to recover to!  The server referred to as "%computername%" in the recovery will likely be offline, and thus not available as a recovery target.
4. In step 3 of "Configuring Backups…" we instructed to add the PreBackupScript commands to the "PSDataSourceConfig.xml" file, but we are not told within which XML tags to insert the code.  I was unable to make Pre-backup scripts run when following these instructions.  Instead, I placed the code snippet into "ScriptConfig.xml" (where other documentation suggests that this code actually belongs), and my backup jobs then started to run.
5. There is no guidance here about the frequency with which BMR sets should be created.  Unfortunately, I can find very little in the way of best-practices on this subject (Server 2008 disaster recovery, as a general topic).  It seems that weekly (or perhaps even monthly) BME sets followed by daily "standard" DPM backups would be adequate to protect most operating systems, but it would be nice to have some verification of this.  Can you point me to any additional documentation on this subject.

• Server 2008 DR needs some improvement, to be blunt about it.  The team promises better BMR integration under DPM 2010, but details are not yet available.

After a semi-disaster with SharePoint earlier this week, I have been forced into the view that I really should have our SharePoint infrastructure hosted on more than one web server.  To that end, I am planning the deployment of a new, 2+ node Windows SharePoint Services farm. 

Initial architecture will be something like this:

• Host: SharePoint2
  • Roles:  Web front end, Search Server query and crawl, ECTS ADAM Instance
• Host: SharePoint3
  • Roles:  Web front end, Search Server query and index, ECTS ADAM Instance
• Hosts: WinDB1 and WinDB2
  • Roles:  Back-end SQL Database failover cluster
• F5 Big-IP Local Traffic Manager (hardware load balancer)

Once initial rollout is complete, we likely will want to add:

• Host: SharePoint3
  • Roles: Dedicated Search Server Index and crawl engine.
• Hosts: WinDB1 and WinDB2
  • Reconfigured in a SQL mirrored configuration

Here is an outline of the SharePoint2/3 build procedure:

1. Install Server 2008 x64 Standard OS
   1. Activate Roles:  IIS (with ASP.NET support), AD Lightweight Directory Services (AKA AD LDS, AKA ADAM).
   2. Activate Features:  .Net Framework 3.0, PowerShell
2. Install Search Server Express x64 bits:
   1. Perform “complete” install (Search Server will not install a SQL 2005 instance, as is the case with WSS installer).  Under “file location”, specify “E:\Office\12.0\Data” as the index storage location.
   2. Skip running of the Configuration Wizard after install.
3. Install SharePoint Administration Kit v2.0
   1. Exclude Profile replicator component as it will not work on WSS
4. Clone the server as many times as deemed necessary. (At present, make one clone!).  Any cloned systems must be sysprep-ed before joining the domain.  Once preped, join the computers, configure networking.
5. If planning to add this server to a load balanced cluster, install NLB feature:
   • from “administrator” cmd shell, run “ocsetup NetworkLoadBalancingFullServer”
   • Don’t join to a production NLB cluster until SharePoint configuration is complete!
6. Replicate AD LDS (ADAM) instance to new machine, if required.
   1. In Server Manager, Click on “AD Lightweight Directory Services” Role,
   2. Click “AD LDS Setup Wizard”
      1. Select “A replica of an existing instance”
      2. Name the instance “ECTSInstance”
      3. Accept standard LDAP ports
      4. specify a partnerpoint server to replicate from, use standard LDAP ports.
      5. Select the “OU=ects,…” partition set for replication (this should be the only partition!)
      6. Select secondary (non-system) volume as target for AD LDS data… generally this will be “E:\Microsoft ADAM\ECTSInstance\data”
      7. Specify domain service account to run the AD LDS instance.
      8. Add “domain admins” to the AD LDS Administrators list.  Finish the wizard.
   3. Run the campus…bat file located in e:\Microsoft ADAM\ECTSInstance\data\.  This will register the Kerberos Service Principal Names required for LDP replication mutual authentication.
   4. Open the “Local Security Policy” Admin tool.  Add the domain service account to the “generate security audits” User Rights Assignment branch.
   5. Open the AD Users and Computers tool, locate the computer object on which you installed the Instance.  Give the LDS service account “create all child objects” to the computer object.
   6. Add the cluster load balanced SSL cert into the Personal certificate store of the ECTSInstance service account.
      1. Request wildcard certificate using the procedure outlined here:
         http://erlend.oftedal.no/blog/?blogid=7
         (We use the web interface for requesting a certificate, make user we use the RSA SChannel crypto provider to generate the request, use the “SHA-1” hash, use PKCS10 format, and use the “UVM – Web Server” request template.  For load-balanced LDAP servers, we must request a wildcard certificate (*.uvm.edu)
         NOTE: This step will not have to be repeated again until the current cert expires.  To add another AD LDS server, export the cert from a current server, import into the new server
      2. Export the request cert to file selecting “export all extended attributes” and “export private key” options.
      3. Import the cert into the “Personal” branch of the service account’s certificate store on the target server.  Make sure that you import “all extended attributes”, and the private key.  Do not select the use of advanced encryption password.
      4. Restart AD LDS and test SSL connections.
      5. If all is not working (as is the case with one of my two servers), here is where we get into undocumented territory.  Here are some helpful resources for debugging:
         1. I set SChannel diag logging to verbose :
            • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\Schannel
              REG_DWORD EventLogging, value 0×7
            • Restart ECTSInstance, look for “SChannel” entries in the server “application” even logs.  These logs will tell you which certificate the system attempted to use, and why access failed.
         2. You may need to add the wildcard cert to the Local Computer Certificate Store as well… run MMC, add the “Certificates” snap-in for “Service Account”, using the “ECTS Instance” service.  Navigate to the “Personal” branch, run an import action, import the wildcard with all extended attributes and the private key.
         3. Now locate the physical copy of this cert in c:\programdata\microsoft\crypto\RSA\MachineKeys (it will be the file with the most recently modified time stamp).  Add “read/execute” permissions to this file for the AD LDS service account, then restart the LDS instance.
   7. Force mutual authentication for replication traffic:
      1. Run ADSI Edit
      2. “Connect to”, enter the AD LDS server name in the Computer field, select the “Configration” well-known naming context.  As documented in http://technet.microsoft.com/en-us/library/cc794841.aspx, get “properties” on the “CN=Configuration…” partition, and change the value of “msDSReplAuthenticationMode” to “2”.
   8. Set local password policy – this controls password policy of AD LDS accounts:
      1. Add the Sharepoint server computer account to the “ETS – SharePoint Password Policy” GP Object.  After running “gpupdate /target:computer /force”, verify the settings by doing the following:
         1. Open Local Security Policy control panel
         2. Expand “Account Policies”->”Password Policy”
         3. Settings applied should follow the 24/365/0/8/Disabled/Disabled format.  (we may want to revisit this policy later).
7. Run the SharePoint Products and Technologies Configuration Wizard:
   1. Connect to an existing Farm
   2. Enter “WINBD” as the database server.  The wizard will correctly select “SharePoint_FarmConfig” as the configuration database.  The correct service account username will be provided… you need to enter the password.
   3. Click “Advanced Settings”, specify that you which the server to host the Central Admnistration site.
      1. If setup fails with the error:
         "SharePoint Configuration Wizard failed with an exception "Error during encryption or decryption. System error code 997"
         A solution can be found here:
         http://blogs.msdn.com/priyo/archive/2007/08/11/add-new-sharepoint-server-to-existing-server-farm-an-unhandled-exception-occurred-in-the-user-interface-exception-information-unable-to-connect-to-the-remote-server.aspx
         Essentially we just run “stsadm –o updatefarmcredentials –userlogin “domain\service_acount” –password <thePassword>” on the first SharePoint server, then re-run the wizard.
   4. Update the “Central Admin” shortcut to point to the local Central Admin site by doing the following registry hack:
      http://blogs.technet.com/wbaer/archive/2007/08/30/sharepoint-3-0-central-administration-url-on-multiple-web-front-end-servers.aspx
      Essentially, edit the key:
      HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Shared Tools\Web Server Extensions\12.0\WSS
      Then locate CentralAdministrationURL and change it to point to the local server.
8. Configure Search Service:
   1. When Search is run in an environment where SharePoint services are accessed from a FQDN which is different from the physical host name (i.e. our environment, or any other environment with load balancers), you will need to work around the “loopback security check” feature of Windows.  Failing to do so will result in “access denied” errors in the crawl logs.  My thanks to Shawn Feldman for discovering this:
      http://blogs.msdn.com/fledman/archive/2008/09/18/access-denied-with-windows-server-2008-and-moss-when-crawling.aspx
      The relevant work-around is documented here (see “Method 2”):
      http://support.microsoft.com/kb/896861
      We simply need to add the public FQDN of our SharePoint server to:
      Key: HKLM\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
      Value: REG_MULTI_SZ, sharepoint.uvm.edu
      And then restart the IISAdmin service.
   2. Open the search admin page from SharePoint Central Administration:
      1. Access Crawling –> Content Sources
         1. Click the “Local Office SharePoint Server sites” default source.
         2. Define a crawling schedule for the SharePoint application
         3. Click “new content source” to add any additional content sources that are desired (i.e. our production file servers).
         4. Define additional crawl schedules for these new content sources.
   3. Add Search Center to our SharePoint landing page:
9. Install Infrastructure Update for WSS3 x64:
   1. Initiate the update on the first node in the cluster. 
   2. When prompted, start the install on the second cluster node.
   3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
10. Install Infrastructure Update for Search Server x64:
    1. Initiate the update on the first node in the cluster. 
    2. When prompted, start the install on the second cluster node.
    3. When the configuration wizard completes on the second node, go back to the first and allow configuration to complete.
11. Clean up IIS settings for the newly created Web Sites – configure binding, authentication and SSL:
    1. SSL Cert Installation: 
       Install SSL certs into “Personal” Store of the Computer account using the “Certificates” MMC snapin.
    2. Binding: 
       Open the IIS Manager MMC snapin.  On each site, right-click and select “edit bindings”:
       1. For site “SharePoint – 443” (which represent the traditional “sharepoint.uvm.edu” URL), bind https and http protocols to port 80 and port 443, using the IP address for “sharepoint.uvm.edu” (132.198.102.12).  When binding SSL, select the appropriate cert from the “SSL Certificate” drop down menu.
       2. For “SharePoint – Internet” (which represents SharePointLite), bind https and http, ports 443 and 80, to “sharepointlite.uvm.edu”, IP 132.198.102.36.  Again, select the correct SSL cert for this site.
       3. For “SharePoint – Extranet” (which represents PartnerPoint), bing https and http, ports 443 and 80, to “partnerpoint.uvm.edu”, IP 132.198.102.49, selecting the matching SSL cert once again.
    3. SSL Configuration:  (Note that these procedures are only accurate when using Windows-native load balancers… when we transition to f5 load balancing, it will not be necessary to return custom errors from IIS as the f5 will handle HTTP-to-HTTPS redirections.)
       1. In IIS Manager, open the “features view” for each site.
       2. Double-click “SSL Settings”
       3. Check “Require SSL”, leaving the default “ignore Client certificates” setting.
       4. Now double-click the “Error Pages” item for the server root.  Add a custom error for 403.4 (SSL required), pointing to our custom “redirect.html” javascript file.  We will need to have copied this file into “c:\inetpub\custerr\en-US\” before completing this step
       5. Now find the applicationHost.config file for the IIS server.  This should be located in “C:\Windows\system32\inetsrv\config”.  Locate the section for each site that serves SharePoint content (i.e. <location path=”SharePoint – 443”>), then locate the <httpErrors> tag under <system.webServer>.  In the httpErrors tag, change the value for “existingResponse” from “PassThrough” to “Replace” (response “Auto” also seems to work, but may produce inconsistent results).  This will prevent ASP.NET from replacing the 403.4 error response from the IIS server.  I am much indebted to this forum thread for this breakthrough:
          http://forums.iis.net/t/1113734.aspx
          Also helpful was the new “failed request tracing” module in IIS7:
          http://learn.iis.net/page.aspx/266/troubleshooting-failed-requests-using-tracing-in-iis7/
          More information on the meaning of the various existingResponse values can be found here:
          http://blogs.iis.net/ksingla/archive/2008/02/18/what-to-expect-from-iis7-custom-error-module.aspx
12. Install the MS FilterPack 1.0 (Search Server can already index most Office 2007 documents, but this adds ability to index inside of One Note files and ZIP archives):
    1. Follow instructions at:
       http://support.microsoft.com/?id=946336
13. Install Adobe iFilter, with 64-bit “thunking” DCOM service:
    • http://labs.adobe.com/wiki/index.php/PDF_iFilter_8_-_64-bit_Support
    • http://workerthread.wordpress.com/2008/07/18/adobe-reader-9-available-works-fine-with-sharepoint/
    • http://servergrrl.blogspot.com/2008/01/and-now-for-something-completely.html
    • TEST IT… Internet chatter suggests that this config is less than reliable.
14. Install MindManager extensions.
    1. DEPRECATED – We will discontinue this extension with the new upgrade as it does not work with MM v7 or v8
15. Install ECTS components on each web front end server.
    • Having problems with installation script… what if we try the ECTS update available though CodePlex???
      1. If using updated ECTS files, it will be necessary to update the PartnerAdmin and PartnerConfig pages, as the self-service Site Collection Manager.  The existing pages will not work because the GUIDs on the Web Parts have changed. 
      2. The ects_setup_sharepoint.vbs script still fails using the updated code… Since the codeplex team has not documented their changes, I think we will skip this option.
    • Troubleshooting issues:
      1. The ects_setup_sharepoint.vbs script succeeds in installing the ECTS solution, but fails when activating site features.  I suspect that “cscript” on Server 2008 is not processing return codes from stsadm.exe correctly, and this is reporting failure to install features (I am not positive about the reason for the script failure, although it certainly is not a result of stsadm.exe being broken. 
         I was able to work around this problem by opening the ects_setup_sharepoint.vbs file in a text editor, searching for the error string that was sent to the console when the script failed, then running all of the operations in the script manually from that point forward.  Fortunately, all of the stsadm commands in the script are successful when run from the command line.
      2. ECTS is not compatible with MS Load Balancing out of the box.  I switched to a F5 load balancer before working through the problem.  It is possible that the problem I was having could have been fixed with the same “loopback security check” that caused problems during our F5 configuration
         http://support.microsoft.com/kb/896861
         In fact, we may have had the problem even with the f5 in place, but I would not know because I applied the loopback fix before implementing the F5.
         The error codes suggest that a login failure is occurring between the IIS application and the AD LDS LDAP instance.  When I try to connect to the load-balanced LDAP DNS name using the “LDP.exe” LDAP client, I also get an authentication error.  However, when I connect to the local server address, authentication works. 
      3. As was the case when I first installed ECTS, the web.config files required a bit of hand-tuning to get services working correctly:
         http://www.uvm.edu/~jgm/wordpress/?p=112
         Once again, I had to modify the “ADAMConnectionString” in the web.config of each IIS site to reflect the actual DNS name of the load-balanced AD LDS servers.  I had installed ECTS using a different name initially, and the ECTS un-installation script did not clear out these values.
      4. I did find it necessary to deactivate all ECTS site collection features, re-activate them, then perform an IIS reset before my existing ECTS management pages would work again.  This seems pretty par for the course when removing and re-installing SharePoint solutions.
16. Install Globally-deployable solutions from the “fab 40” application template.  If you deploy a web front end into an existing farm, the files required by these features will get transferred automatically.  However, when building a new farm, we need to install them manually.  Currently required “server admin” templates are:
    • ApplicationTemplateCore
    • ChangeRequest
    • ContactsManagement
    • DocumentLibraryReview
    • EventPlanning
    • HelpDesk
    • InventoryTracking
    • ITTeamWorkspace
    • Knowledgebase
    • LendingLibrary
    • PhysicalAssetTracking
    • ProjectTrackingWorkspace
    • RoomEquipmentReservations
    • Procedure:
      • stsadm -o addsolution -filename <file_path>\<template_name>.wsp
      • stsadm -o deploysolution -name <template_name>.wsp –allowgacdeployment
17. Install radEditor:
    1. Install ASP.NET Ajax for .NET 2.0, version 1.0
    2. Follow the Ajax configuration for SharePoint configuration guide found here:
        http://sharepoint.microsoft.com/blogs/mike/Lists/Posts/Post.aspx?ID=3
    3. Install radEditor using the included instructions.
    4. Copy radEditor configuration files from an existing production server to the new server:
       1. In the directory:
          ”C:\Program Files\Common Files\Microsoft Shared\web server extensions\wpresources\RadEditorSharePoint\[versionString]\RadControls\Editor”
          Backup the existing ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml files.  Replace with versions customized for UVM.  Note that the MOSS LinkManager tool does not work in WSS.  Also note that when editing list content that does not support “Enhanced Content”, the first toolbar in the ListToolsFile.xml will be removed… in past versions, the toolbar named “enhancedTools” was removed.
       2. Copy the files ListConfigFile.xml, ConfigFile.xml, ListToolsFile.xml, and ToolsFile.xml to all other nodes in the cluster.
       3. perform an IISRESET.
       4. Update ONET.xml files in the “12” hive to activate the radEditor feature by default in all new sites (see ONET.xml template files on the prod web front end for examples).
          NOTE that the “RadEditor for non-IE browsers” and “RadEditor for IE” features have been collapsed into one unified feature.  Update the ONET.XML files accordingly!  (note that the feature ID for the main RadEditor List editor has not changed… only it’s name is different.  We did not have to insert a new default feature ID, but we did need to remove the “RadEditor for IE” feature because it is no longer present in RadEditor MOSS.)
       5. Run:
          stsadm –o uninstallfeature –name RadEditorFeatureRichHtml.
          This “Web Content Management” feature is not supported in WSS, so we may as well remove it to avoid confusion.
       6. Deactivate and then re-activate the radEditor features on at least one existing site, and test functionality.
18. Install “Smiling Goat” Feed Reader (RSS/ATOM subscriber web part)
    1. This will require Feed Reader users to update their web parts!
19. Install SharePoint Training Kit:
20. Tune web application settings to match production server:
    1. Set upload limits for files (also need to set IIS “maxAllowedContentLength” in each web.config to be longer than the SharePoint upload limit.  See http://support.microsoft.com/kb/944981/en-us for details.)
    2. Set time zone
    3. Set allowed/disallowed MIME types
    4. Set quota templates for new sites
    5. Config incoming/outgoing email settings
    6. Config site expiration/auto-deletion.
    7. Edit the footer of the “welcome” email message starting at line 5219 of “core.en-US.resx” in the 12-hive “resources” folder.  Replicate on all web front ends in the farm.
21. Configure f5 load balancers:
22. TEST TEST TEST:
    1. Test each feature on both web front ends by alternately disabling the nodes in the load balancer configuration.
    2. Test again with both nodes enabled… watch for authentication and session persistence issues.
    3. Test all features in each access mapping – SP, SPLite, and Partner… web.config file variations could cause problems!
23. Consider Deployment of “Group Board 2007” and “Sample Master Pages”:
    • http://www.microsoft.com/sharepoint/templates.mspx

I had a look at implementing MS IT Site Life Cycle Management as an alternative to AvePoint products, or the previously blogged-about MS IT Site Delete Capture utility:

http://www.codeplex.com/governance/Release/ProjectReleases.aspx?ReleaseId=4622

Unfortunately, this product just is not going to work for us.  It is possible that we could wrangle it into shape with enough time, and an ability/desire to check the code out of codeplex and patch it up.  However, I just can’t bring myself to deal with it.  Here are some problems that I encountered:

1. The utility has not been tested or developed to work on the Server 2008 platform.  The directions are written with Server 2003 as a reference platform, and tell you do do things like “create a virtual directory” when what they really want you to do is to create an application in your IIS App Pool.  I could live with this but…
2. The tool does not work on Server 2003 either, at least, not when using WSS 3.0 Service Pack 2.  The web.config file in the LCMWeb directory references an assembly in the GAC named “Microsoft.Internal.MIME”, with the version 8.0.0.0.  Guess what?  That assembly was upgraded with Service Pack to to version 8.0.681.0.  But even after updating the web.config file to the new version, I still get errors when attempting to load the LCMWebConfig.aspx page.  humph.

Essentially, this tool is not supported, and ,to make matters worse, it is not being maintained.  I really would hate to spend time beating it into shape only to have the damn thing break when the next service pack or release of WSS comes out.

“Site Delete Capture LE” from Microsoft IT… cool idea, tricky to implement.  Here is the problem:

Attempts to delete a site result in “Access Denied” error messages in the site delete log files.  No corresponding events found in the Security Event logs, nor are we able to detect any “ACCESS DENIED” messages using procmon.exe.  What’s up? 

Well, in this thread:
http://governance.codeplex.com/Thread/View.aspx?ThreadId=11781 
one of the project authors suggests that the utility requires additional rights beyond the “least-priviledge” baseline, specifically, the account running your SharePoint WFE applicaiton pool needs to be a “Farm Administrator”, and it needs “Full Control” over the web application.

Much as I did not like this suggestion, I decided to give it a try in the test environment, but it fails anyway.  Further investigation reveals that the sharepoint WFE service account is not actaully capable of performing site backups.  If you log in as the service account, you cannot run any “stsadm” commands at all… every command results in “ACCESS DENIED”.

It turns out that stsadm.exe will not run without local administrator privs.  It also would seem (although I cannot prove it) that the Site Delete Capture utility is using stsadm functions to generate its snapshots.  Since I will not be giving our SharePoint WFE app pool local admin rights, I guess I cannot use this utility.  On to testing the Site Lifecycle Manager instead…

Hey look… Microsoft IT has released some cool tools for SharePoint management:
http://governance.codeplex.com/

Possibly of most use would be:
http://www.codeplex.com/governance/Release/ProjectReleases.aspx?ReleaseId=14351
A utility to automatically backup sites upon deletion actions.

And:
http://www.codeplex.com/governance/Release/ProjectReleases.aspx?ReleaseId=4622
Site Lifecycle Management – a potential replacement for the hated “Site Expiration” process we have in place at present.

Previously I documented a rough outline of the AX 5.30 Infrastructure installation process:
http://www.uvm.edu/~jgm/wordpress/?p=71

With support for 5.30 expiring today, I think it high time we got our infrastructure up to date up to the most current version that is supported for use with SunGard Banner.

1. Uninstall all previously existing AX components.  Purge any residual files from the IIS  publishing directories, “Program Files”, “Application Data”, and the registry.
2. Set security for the global impersonation account according to the table on page 210 of the “concepts and planning guide”.
   1. Note that the account does not have to be a local administrator!
   2. However, the security accounts will have to have privileges to the resources accessed by the services (i.e. NTFS filesystems rights, shared folder access).
   3. Rendering Service -
      1. When granting rights to the DX data store, plan ahead.  Permissions could take a long time to apply.
      2. Requires Local Security Policy “Replace a Process Level Token” and “Adjust memory quotas for a process” rights.  Also, the “Allow service to interact with the desktop” box must be deselected in the “Log On” tab of the Rendering service properties.
   4. WebAccess.NET Services -
      1. Global Account needs only “Log on as a service” Local Security Policy assignment.  You can clear out all “legacy” security permissions as they are not needed for WebAccess!
3. Install AX Desktop, installing all administration tools:
   1. msiexec /i “ApplicationXtender Desktop.msi” /qb ADDLOCAL=DocumentManager,AppGen,ConfigurationTools,ManagementTools
4. Install the new License Server and install license file:
   1. Install the “ApplicationXtender License Server.msi” (FlexNet License Manager)
      1. Drop the .LIC license file into C:\Program Files\XtenderSolutions\Content Management\License Server
      2. Configure the Login identity of the “ApplicationXtender License Client Components” COM+ application to use the global impersonation account.  This component must be shut down to be reconfigured.  Details in EMC PowerLink solution esg92864.
      3. Restart the “ApplicationXtender License Service” Service.
   2. Install the “EMC License Server” (Proprietary License Server, to support DiskXtender)
      1. Install all current patches to the service
      2. Run the “License Server Administrator” GUI.
      3. Go to “Tools”, then “New License Wizard” to install the DiskXtender License.
5. Install DiskXtender
   1. Install DiskXtender patches, in sequence
      1. When prompted for the DX service account, you must provide an account that has local “administrator” rights, and the ability to “log on as a service”.
   2. Verify and/or re-establish RPC partition maps – See the “Core Components” guide for instructions.
   3. Consider switching to DCOM security model, which will require modifying the “AE_PATHS” table in each data source db.   See page 160 of the “Desktop Install Guide” for details.
      1. This is not actually practical to do since it will break AX Desktop on any system that is not joined to the CAMPUS domain (and why would they not be joined, I wonder?)
6. Launch AX Admin
   1. See “ApplicationXtender Desktop Installation Guide” for details
   2. Log in as SYSOP and perform the database upgrade, if prompted.
   3. Verify global settings:
      1. Add license server configuration: see “Core Components” guide for details.
      2. Web Access .NET must use Global credentials since we are using and Oracle database with Oracle security.
   4. Save the configuration and exit
7. Launch AppGen, and verify functionality.
   1. Connect to each defined data source, one at a time.
   2. Perform database upgrades if prompted (this should be safe, but can take several minutes to complete).
8. Set IIS web site root to use ASP.NET 2.0.
9. Install AX Web Services, making sure to install the required “Utility Services” component.  “AX Web Services” and “Workflow” components are optional.
   1. See “AppXtender Core Components Admin Guide” for installation and config details.
   2. Choose IIS installation option, and install into “Default Web Site” (which should be the only site present)
   3. Ensure that “Default.aspx” is listed as an accepted default page for the “AppXtender” IIS web application.
10. Install AX Web Access .NET
11. Install AX Rending Server
12. Run the Component Setup Wizard for all installed components
13. Outside of my control:
    1. BannerXtender updates need to be applied to production Banner systems.
    2. DocSend and ECopy stations need to be upgraded to 5.40 AX Desktop releases
    3. DocAccel server needs 5.40 AX Desktop upgrade
    4. All AX desktop clients need updates, too.
    5. Anyone using WX WebAccess.NET ActiveX controls will need to upgrade these components.
14. Test Test Test!

MDT looks to be gaining a lot of usage, which is good from my perspective as it means more brains with whom to share ideas.

Here are a few ideas I have been considering for addition to our MDT Workbench:

• Use the MDT Wizard Editor to add a page for whole-system backup.  This functionality has been requested by our College of Medicine IT Staff.
• Implement the “Roles Wizard” solution to allow Distributed IT staff the ability to select MSD database roles for their system.
  • This will require that I farm out data writer rights to the MSD database, too.
• Move all online versions of the MDT Distribution Points to our NetApp network storage and activate de-duplication features… this should keep the whole bloated thing much more trim.
• Create Model Aliases for Apple computers.  This will be helpful as we gear up for BootCamp support.
• Finish my own disk paritioning script to add support for BootCamp, and to preserve OEM utility partitions.
• Integrate all current Dell Driver CABs… figure out if there are notifications available for these.
• Move “sensitive” application installers into a single central distribution share and secure them.  Do not replicate these installers to secondary distribution points.
• Implement “media” based deployment points to allow high-speed deployment from low-speed networks.

Microsoft Enterprise Desktop Virtualization, or MED-V… an really cool new technology.  And as with any right-out-of-the-stable product, documentation is a bit sparse.  Worse, there are currently no Microsoft-sponsored forums for the product.  Those of us wising to deploy right away are going to be figuring things out on our own… again.

A few tidbits so far:

• There appears to be a missing manual, available at various places on the Internet, but not from Microsoft.  Perhaps it came with the MED-V beta?  Anyway, it has some good step-by-step configuration info for IIS that is not present in the official manual.  Get it here:
  http://www.mvug.co.uk/media/p/120.aspx
• When setting up an HTTP distribution server for MED-V, make sure the following role services get installed:
  • BITS Server Extensions (added as a general feature after IIS is installed)
  • Windows Authentication
  • Basic Authentication
  • Client Certificate Mapping Authentication
• You will need to grant read/write access to the image upload directory on the IIS server… kind of a “duh” note; however, you may want to add the permissions from the IIS console rather than from explorer, as doing so avoids issues with web.config and BITS transaction file permissions.
• Before you download a MED-V image to a MED-V client, you will need to add the two following MIME types to the IIS server:
  • .index – application/octet-stream
  • .ckm – application/octet-stream
• Reporting Database configuration:
  • Documentation suggests but does not specifically state that when configuring MED-V to use a remote SQL server for the reporting database, you should need to use SQL Authentication. If you attempt to use Windows Integrated security, you will see that the Med-V service attempts to connect to the new database as an “anonymous user” (which, of course, fails).
  • The documentation tells you to provide the SQL Server “SA” login and password to the MED-V database configuration tool.  This, if course, is not necessary.  You can and should create a separate SQL login for MED-V.  This login will need to have “dbcreator” rights at the time you create the database, but this role can be revoked as soon as the database has been created.
  • Your database connection string should resemble the following example:
    Data Source=DBSERVER1\Instance1;Initial Catalog=MEDVReports;uid=medvReportUser;password=G1bb3r1$hP@ssw0rd
• XP Image Configuration:
  • When configuring your XP image for sysprep and domain join, you must create an sysprep.inf configuration that will run mini-sup in fully unatteded mode.  You file must contain the following entries in addition to your preferred local settings:

  [Unattended]
  InstallFilesPath=C:\i386
  OemSkipEula=Yes

  [GuiUnattended]
  OEMSkipRegional=1
  OEMSkipWelcome=1
  AutoLogon=Yes
  AdminPassword=”insertYourPasswordHere”
  EncryptedAdminPassword=NO
  AutoLogonCount=5

  [Identification]
  JoinWorkgroup=UVM

  [Networking]
  InstallDefaultComponents=Yes

  [UserData]
  ProductKey=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
  FullName=”User Full Name”
  OrgName=”Your Organization Name”
  ComputerName=*

  • You also must configure the automatic administrator logon option for at least as many times as are required to perform your planned computer rename and domain join operations (whcihc would be two reboots, but you might want to pad it a bit to be safe).  Keep in mind that the MED-V client will not be able to hook into the guest GINA using your MED-V login creds until the workstation has been joined to the domain.

After two days of troubleshooting some vexing problems with ECTS, I have arrived at a new recommendation concerning this product:

Run Screaming

Okay, that may be a bit damning… here is a qualifier.  If you have no SLA with your customers, don’t mind lots of downtime, love C# programming, and otherwise find SharePoint troubleshooting highly amusing, then ECTS is the solution for you.  Otherwise, try something less painful.

The main discovery that pushes me to make this recommendation is that the ECTS solution is not supported by Microsoft.  The head of the MS Solutions Accelerator group once told me that all Solutinos Accelerators for currently supported MS products are supported by Microsoft.  This is not actually the case.  Always read the fine print for your Solutions Accelerators.  In the case of ECTS, we find the following in the ECTS FAQ, available in the “Informational Materials” download.  From http://www.microsoft.com/downloads/details.aspx?displaylang=en&FamilyID=d9af2c25-989c-45c4-8008-1f15722190ed:

Answer: Although every effort was made to ensure that the External Collaboration Toolkit for SharePoint provides trouble-free installation and reliable operation, it is not officially supported by Microsoft. We will provide best effort support on the SharePoint – Collaboration forum, but cannot guarantee timely resolution of any issues.

Other pertinent bits of info from the Information Materials:

• There is a suggestion that the product has been tested and is “supported” (whatever that means) only with Server 2003 (not Server 2008).
• There is a reference to the commercial product called the “External Collaboration Manager“, which I expect was developed by the same consultants that created ECTS in the first place (although I have no proof of that).

I am going to pursue aggressively a Forms-based authentication solution using ADFS as a replacement for this service.

FWIW, here is a quick breakdown on the current set of problems, and what I did to fix them:

1. When attempting to load the \sites\[sitename]\_layouts\ExternalCollaboration\aeu.aspx page (add new external user form), one user reported “Access Denied” error messages, even though he was in the site administrators list.
   • This happens because the aeu.aspx page checks to see is the currently logged in user is in the “Site Owners” group.  Of course, being a site administrator should be sufficient, but the page does not check for effective rights, but instead performs a simple ACL check.
2. Users would intermittently experience page load errors when attempting to load the same “aeu.aspx” page.  Experientation with our load balancer indicated that the problem occured on  only one of the two web front ends in our farm.  Attempts to troubleshoot by using verbose Diagnostic logging, SysInternals ProcMon, and various browser debuggers (ieHTTPHeaders, Fiddler) turned up nothing.
   • Finally, I discovered that there was a single “ExternalCollaboration.RESX” file missing in the inetpub\wwwroot\wss\VirtualServers\[IIS-site]\App_GlobalResources\ directory.  After copying the RESX file to all three production IIS sites on both web front ends, and performing an IIS Reset, the page load error went away.  I don’t believe that these files ever got replicated to the second web front end after installation, so add this step to your farm build procedure.
3. Our Account Services team reported that the “PartnerAdmin” page (or ECTS Administration web part) was reporting a “service unavailable” error.
   • This problem happened because all of the web.config files on our server reverted their “ADAMConnectionString” values to point to the pre-production host name for our ECTS LDAP service.  The event that triggered this service reversion is still unknown.  After I updated the string to point to “sharepoint.uvm.edu”, the problem was resolved.  Once again, I really need to re-install the whole solution using the correct LDAP connection string… the incorrect value is still cached somewhere in the SharePoint infrastructure that I cannot locate (most likely in configuration DB), waiting to bite me again.

I got a report from one of our power users that SharePoint search was not returning any results.  The default search service under WSS 3.0 is not very easy to troubleshoot.  Things will get better with Search Server 2008 (I hope).  Anyway, I did find the following in my Application Event Logs:

(more…)