Rebuild site servers without redistributing content over the WAN

, , , , , , , , , ,

Rebuild site servers without redistributing content over the WAN:

Outlining the Project


                In order to support the Windows 10 Migration project for this customer the hierarchy needed to be upgraded to a level to support deploying/managing of windows 10v1607. The current level of the environment is 2012 R2 SP1 and we will move to CB1606 and upgrade to CB1610. During this time I took the opportunity to simplify the architecture as there were dozens of unnecessary secondary sites. These locations would have the site replaced with a single distribution point. There was no getting around these secondary sites as the pre-req check would fail due to the unsupported version of SQL server express installed on the systems. SCCM does support in place upgrade of SQL on a site server but that is limited to full SQL and not the express version. The best practice would be to spin up the new servers, configure them as Pull Distribution Points, pull the content from the old servers, and then smoothly transition between the two. In this environment, however, we were not provided with new servers to use, and we were faced with the added difficulty of prohibitively slow WAN connections, requiring us to come up with an alternative solution.




Best Practice/Worst Practice


                What we came up with was a series of PowerShell scripts that eventually evolved into the Distribution Content Migration Tool-Kit module.  This module takes queries WMI to pull a list of all content assigned to a Distribution Point and runs on the Distribution Point to create prestage packages for all of that content.  Once the required roles have been removed, reconfigured, and added back to the server, the module then allows all of that content to be reassigned to the Distribution Point and subsequently extracted to complete the migration.



Prerequisite Components


                Because portions of this module had to run locally on our Secondary servers, we needed the Configuration Manager libraries loaded as well as the most recent version of the Windows Management Framework.  While it is possible to just copy over the required DLL files and import them into PowerShell, we did want to stick with something more reliable and consistent, so we installed .NET 4.5, WMF 5, which is required to install the Configuration Manager Console. These updates were all copied to the server and then installed to the clients with a quick PowerShell query to find all servers with the Secondary Site role installed piped into a Copy command.  A few reboots later, and the servers were primed to migrate.



Before Removing Roles


                Modify the site assignment and site server referenced on your boundary groups to talk to another site system server. This is set for your boundary by the boundary group applied to it. I changed the site assignment to my primary site server. I changed my site system servers to the MP on my primary and I left the DP blank as this was only an expected outage of less than 2 hours. Once the conversion is complete I will place my DP here for the boundary group. If you do not want to leave that blank you can use the closest DP that has open ports for communication. Remember do not remove any of the roles until we create the pre-staged content locally on the site server.



Prepare server for role removal


                The tool-kit is made up of four separate scripts written out as the functions Get-DPContent, Prestage-Content, Restage-Content, and Extract-Content.  



Script Step 1: Get-DPContent


                The Get-DPContent function requires you to specify a Distribution Point and will pull a list of every piece of content SCCM has assigned to it.  It returns an unformatted array of WMI SMS_PackageBaseClass objects which can look a little daunting, but can be easily formatted for reporting or further processing.


Title: A single DP Content Info object - Description: This is the raw dump of the information returned when you get the SMS DPContentInfo class


Figure 1 – Raw data produced by the SMS_DPContentInfo Class




Figure 2 – Table-formatted values for just PackageID and Name


                I’ve seen some guides online use Get-CMDeploymentPackage to get package info, but I’ve found that WMI works up to 3x faster when querying large data sets, and it runs without needing a connection to a CMSite drive, so it’s become my preferred method.



Script Step 2: Prestage-Content


                This function does the actual work of creating a prestage PKGX file based on the package ID you provide it, the Distribution Point that holds the content, and the location of the folder that will store the package for later use.  The ConfigurationManager PowerShell module actually comes with a cmdlet called Publish-CMPrestageContent, but because that cmdlet requires you to specify the type of item you’re prestaging, we wrote this function to make the WMI call, examine the package type, and issue the correct command for you.  For one off package prestaging, this is still far and away superior to going through right click menus, but where this function shines is when it’s used in conjunction with the Get-DPContent function. 



Figure 3 – Prestaging a single package





Figure 4 – Prestaging multiple packages via For loop


Remove roles from SCCM console.


After you successfully create the pre-staged content locally on the server we can move forward.   In our case, we needed to remove all roles assigned to the server and only add the DP role back. This action required us to remove the DP, MP, and SUP. After these roles are removed we can go ahead and remove the site server.


Note: when you are decommissioning the secondary site this will also uninstall the DP role naturally. Out of habit I recommend to remove all soles prior to uninstallation of the Site.


  • Validated via the distmgr.log on the primary site server
  • Validated by no longer being seen in distribution point configuration status in the console





Removing the site server


                From the console Administration > Expand Overview > Expand Site Configuratoin > Sites > Select the site and “Delete”. This will create a new dialog box and it is important that you read the differences btwn uninstall and remove. We will choose to uninstall.





From here you can see the state of the secondary site server has moved to “Deleting”



How to monitor and the site server uninstallation process


On your secondary site server you can monitor this from C:ConfigMgrSetup.log. The site server uninstallation process is roughly as follows.


1. ConfigMgr2012 Setup is started by system with command line options /deinstall / msg2parent /nouserinput

2. Information is checked, this will be things such as the following. FQDN, OS is verified, Checks for existing setup information, existing SQL information, existing configmgr installation and version number, etc.



3. removes SQL alias for sccm

4. Starts uninstallation of secondary site by first cleaning up SQl server replication data, start uninstallation of local dp (if applicable) Remove content SCCMContentLib, SMSPKG, SMSPKGF$, SMSSIG$ directories from the server. The process will also move through list of all SCCM Services and stop/uninstall them if present and then stop WMI



After services/connections are removed you will see a number of redlines in the log file. This is only b/c connection can not be established which is expected right after stopping WMI




5. Connect to database, drop schema SMS_SiteSystemToSQLConnection, drop database, and uninstall SQL (if applicable)



NOTE: ONLY If your admin installed SQL instead of letting SCCM perform the uninstall action during site install you will see this message



6. Attempt unregister list of Binaries



7. Attempt delete remaining folders/files from within the configmgr installation directory



8. remove registry keys, restart WMI, and other services then complete uninstallation of Configuration Manager Site.



NOTE: After site is uninstalled you might run into issues where the secondary server is still showing “Deleting” this can be resolved by my other blogpost HERE where I had to use the hierarchy maintenance tool.



Remove unnecessary items


Start by uninstalling SQL (if applicable) the only time you will have to uninstall this is if the admin installed / configured SQL on the secondary site instead of letting SCCM do this action. Remove any other roles/features that are no longer needed. For this environment I also removed WSUS as it is no longer needed nor will be able to patch win 10 when the server is on server 2008R2 w/ WSUS 3.0
since we uninstalled SQL this freed up two extra drives on the machine that stored the database and the log files. These were then reclaimed by the storage team. For the entire project this allowed 1200 GB to be reclaimed. Uninstall the sccm console as it is no longer needed.



Before reinstalling DP


I have performed a number of conversions in the past where there were problems reinstalling the DP role. Typically this process goes just fine but in rare instances I ran into issues and have to completely remove the client / delete from database / rediscover / reinstall client / reinstall role, so I recommend doing the following.  


  1. Completely uninstall SCCM Client
  2. Remove the following registry HIVE “HKEY_Local_MachineSoftwareMicrosoftSMS”
  3. Reinstall SCCM client



Reinstall DP Role


                There should not be any additional configuration needed as this server previously had the DP role. Make sure this is not configured for pull dp and you enable this for pre-staged content. You can track the installation process in 2 logs: DISTMGR.LOG on the primary site server installation path and and SMSDPPROV.LOG located SMS_DP$smslogs. You can also Track through monitoring on the console


Track though the distmgr.log on your primary server and smsdpprov.log on the DP.





Script Step 3: Restage-Content


                The Restage-Content function crawls through the list of packages we saved and tells SCCM to re-assign the content.  While the Prestage-Content and Extract-Content functions need to be run on the DP you’re migrating, this command, along with Get-DPContent, can be run from any computer as it is only interacting with meta-data on the SCCM server. All you need to specify here is the location of the prestage files and the name of the distribution point they’ll be assigned to.  



Figure 5 – Content restaging in progress



Figure 6 – Existing content will not waste time trying to reassign



Script Step 4: Extract-Content


This function takes input in the form of the prestaged content location and uses Microsoft’s ExtractContent.exe program to manually add them to the content library.  While we messed with the idea of having it prompt you for the location of the ExtractContent.exe utility, we eventually decided that it was simpler to just require the exe file to be in the same directory as the prestage packages.  This takes a while to run depending on the quantity and size of your PKGX files, and in the event that some do not sync properly when you check your Distribution Point Configuration Status messages, you can run this function again, and it will only try to extract content that isn’t flagged as State=0 (successful).



Figure 7 – Content extracting one package at a time




Final Product


In conclusion, while there are some tools and packages out there that are more “double click and go” automation, we’ve found that every environment is too different for one solution to work for everybody. With that in mind, we focused on developing a toolkit that could be adjusted and tweaked for any environment and then used that to simplify our infrastructure to make life easier for the local admins.  Our next step in this project is to begin the upgrade from SCCM 2012 R2 SP1 to Current Branch 1606, and eventually to Current Branch 1610.    When all is said and done, we’ll have converted nearly three dozen secondary sites, all with their own Distribution Point, Software Update Point, and Management Point roles over to just Distribution Points.  In addition to saving several hundred GB of content distribution traffic this conversion will have eliminated much of the unnecessary SQL and WSUS traffic we saw. The storage team was also thrilled to realize 1.2TB of storage can now be reclaimed.
ALSO SEE : Lockstate Object


Lockstate Object

, , ,

This comes from something that one of my PFE buddies showed me during a visit to Kuwait. From time to time you will have an object locked out by one of your team members whom left for the day. Another scenario that is common would be the SCCM Console crashed and when you to to modify the object SCCM will think the object is still being edited.

In my screenshot below we see that my zero touch task sequence has been locked out by “BuckC”

Launch SQL MGNT Studio > Connect to your database > New Query

Find the LockID for the object you are trying to delete by running the following

SELECT * FROM SEDO_LockState WHERE LockStateID <> 0

This will display ID’s of all items locked, locate the relevant user that has locked out the object.

In my case I will run

DELECT from SEDO_LockState where LockStateID <> 0 and assigneduser like ‘%buckc’

We can now open and modify my Zero Touch task sequence.

note: if anybody is interested I can provide task sequence in a future blogpost. The task sequence moves a machine from windows 7 to widows 10, converting from Legacy BIOS to UEFI partitioned w/ GPT. This also enables secure boot,device guard, credential guard.


ALSO SEE : Slow WinPE Boot Speed

Things to check after site recovery

, ,

Unfortunately I have found myself in a disaster scenario that required recovering one of our sites.
This is my list of identified problems from that experience. Please let me know if you have ran into more. Please first check everything listed from Microsoft here. It is important to reset your passwords in the console and apply all hotfixes.

1. Compliance items not showing up on SCCM Clients
– Resolved by redeploying the compliance items

2. Hardware Inventory/Asset Intelligence not being collected
– Resolved by recreating client settings and deploying

3. OSD Broken due to DP Certificate Store error
Status: 0xc0000098
Info: The Windows Boot Configuration data(BCD) file from the PXE server does not contain a valid operating system.
When you check the SMSPXE log, it shows the following message
PXE:: MP_LookupDevice failed 0x80092002
Failed to create certificate store from encoded certificate
– Resolved by Right click on DP under General and change the date for the self sign certificate

4. OSD Broken due to all unknown computers collection no longer contains members
– Resolved by recreating the unknown computers objects and adding to collection.

To recreate the collections, change the registry value of CreatedUnknownDDR under SOFTWAREMicrosoftSMSCOMPONENTSSMS_DISCOVERY_DATA_MANAGER on your primary site server (not the CAS). Set the value to 0.

Then, re-start the SMS_EXECUTIVE service. When the Discovery Data Manager (DDM) component starts, it will think that the Unknown System records have not been created, and will recreate them.

The above restored my unknown computers however it created duplicates. I went ahead and created a collection with those for the site code and then deployed my OSD task sequences as available. I tested against my test VM and successful deployment…or so I thought.

5. Reports that OSD was STILL Broken for 15% of systems due to duplicate unknown computers
– Resolved by removing the older unknown computer object directly from the sccm database

There are known issues after a site restore for duplicate unknown computer objects. When this occurs systems will not be able to identify which unknown object to identify with, and will not fall under the proper unknown collection. We have tried to create a collection with the duplicated/original computers but this will not work. Machines will attempt to pxe and return with the following.

PXE-E53: No boot filename received PXE-M0f: Exiting Intel PXE ROM. Operating system not found.

Typically in the past the solution is to go into the configmgr console and delete the duplicate object so it will now be able to view the advertisement to unknown computers collection.

Example of problem. This will show the duplicate objects for KW1.

I am in a CAS environment and I had to perform the restore on the CAS from my primary site servers database. I Launched SQL Mgnt studio on my primary and and ran the following query

select * from UnknownSystem_DISC

Pay attention to the far right column named Creation_Date0. These dates represent when the object was originally created (initial hierarchy implementation) and the date of site restore. We are looking for the dates on the duplicate unknown objects. The item keys need to be deleted for the older object.
Run this sql command

delete from UnknownSystem_DISC where ItemKey in (2046820352, 2046820353)

After this I decided now that I Have the two proper newly created unknown computers objects (x86 and x64)

my next action would to make a new unknown computers collection and redeploy my available task sequences.

Side note: Number 4/5 When I first tested OSD I must have been a test VM that was a known computer. When I tested OSD again I must have used another test VM that was an unknown computer. This was something that was overlooked at the time which lead me to believe OSD was 100% functional before I went on weekend vacation in Dubai. When I returned from partying it was reported that some systems were still having problems with OSD.

Yes… I know it is against Microsoft best practice to deploy to all systems however this is what the customer required. Roughly 90%+ of all systems imaged at the site are existing known objects.


ALSO SEE : Start Windows Update Service Compliance item